2024-04-27T01:48:41.787 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-27T01:48:41.788 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675525 --name teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675525/orig.config.yaml 2024-04-27T01:48:41.839 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-27T01:48:41.927 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675525 branch: squid description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7675525' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/v18.2.0 ' name: teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 4854 sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a sleep_before_teardown: 0 subset: 40/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC6Y1/iGgURJdjmMr99e4FyQN5Mawd9wp/wUJoS+Iq40L2ITjqAhz7XMQG8w6sgwgq/Zf7TfRoOcysBUwzMZryM= smithi052.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBG0HqsVnfhUZ9c4lxgbA0XF2R+ybv4CnxHRrldGfDt63LZaA33o6tWhAYl4Y8vLdCGG5gl7Y/pssZIwlNzx7TM= tasks: - install: exclude_packages: - ceph-volume tag: v18.2.0 - print: '**** done install task...' - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.io/ceph/ceph:v18.2.0 roleless: true - print: '**** done end installing v18.2.0 cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: - /home/teuthology/src/git.ceph.com_ceph_squid/qa/suites/orch/cephadm/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml meta: {} postmerge: [] variables: fail_fs: false teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_21:08:03 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs false || true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-04-27T01:48:41.927 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa; will attempt to use it 2024-04-27T01:48:41.928 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks 2024-04-27T01:48:41.928 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-27T01:48:41.930 INFO:teuthology.task.internal:Checking packages... 2024-04-27T01:48:41.953 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a' 2024-04-27T01:48:41.953 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-27T01:48:41.953 INFO:teuthology.packaging:ref: None 2024-04-27T01:48:41.954 INFO:teuthology.packaging:tag: None 2024-04-27T01:48:41.954 INFO:teuthology.packaging:branch: squid 2024-04-27T01:48:41.954 INFO:teuthology.packaging:sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T01:48:41.954 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-27T01:48:42.245 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2447.gb22e2ebd 2024-04-27T01:48:42.247 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-27T01:48:42.254 INFO:teuthology.task.internal:no buildpackages task found 2024-04-27T01:48:42.254 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-27T01:48:42.275 INFO:teuthology.task.internal:Saving configuration 2024-04-27T01:48:42.295 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-27T01:48:42.301 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-27T01:48:42.326 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675525', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-27 01:43:27.748629', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC6Y1/iGgURJdjmMr99e4FyQN5Mawd9wp/wUJoS+Iq40L2ITjqAhz7XMQG8w6sgwgq/Zf7TfRoOcysBUwzMZryM='} 2024-04-27T01:48:42.348 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675525', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-27 01:43:27.750281', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBG0HqsVnfhUZ9c4lxgbA0XF2R+ybv4CnxHRrldGfDt63LZaA33o6tWhAYl4Y8vLdCGG5gl7Y/pssZIwlNzx7TM='} 2024-04-27T01:48:42.348 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-27T01:48:42.354 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-04-27T01:48:42.354 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-04-27T01:48:42.354 INFO:teuthology.run_tasks:Running task console_log... 2024-04-27T01:48:42.428 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fadf727a0d0>, signals=[15]) 2024-04-27T01:48:42.428 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-27T01:48:42.434 INFO:teuthology.task.internal:Opening connections... 2024-04-27T01:48:42.435 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2024-04-27T01:48:42.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T01:48:42.517 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi052.front.sepia.ceph.com 2024-04-27T01:48:42.518 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T01:48:42.594 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-27T01:48:42.600 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2024-04-27T01:48:42.619 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2024-04-27T01:48:42.619 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2024-04-27T01:48:42.677 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2024-04-27T01:48:42.677 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="9" 2024-04-27T01:48:42.677 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2024-04-27T01:48:42.677 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2024-04-27T01:48:42.677 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="9" 2024-04-27T01:48:42.678 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el9" 2024-04-27T01:48:42.678 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-27T01:48:42.678 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2024-04-27T01:48:42.678 INFO:teuthology.orchestra.run.smithi029.stdout:LOGO="fedora-logo-icon" 2024-04-27T01:48:42.678 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-27T01:48:42.678 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2024-04-27T01:48:42.678 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-27T01:48:42.678 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-27T01:48:42.678 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-27T01:48:42.679 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2024-04-27T01:48:42.704 DEBUG:teuthology.orchestra.run.smithi052:> uname -m 2024-04-27T01:48:42.722 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2024-04-27T01:48:42.723 DEBUG:teuthology.orchestra.run.smithi052:> cat /etc/os-release 2024-04-27T01:48:42.781 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="CentOS Stream" 2024-04-27T01:48:42.781 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="9" 2024-04-27T01:48:42.781 INFO:teuthology.orchestra.run.smithi052.stdout:ID="centos" 2024-04-27T01:48:42.781 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE="rhel fedora" 2024-04-27T01:48:42.781 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="9" 2024-04-27T01:48:42.782 INFO:teuthology.orchestra.run.smithi052.stdout:PLATFORM_ID="platform:el9" 2024-04-27T01:48:42.782 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-27T01:48:42.782 INFO:teuthology.orchestra.run.smithi052.stdout:ANSI_COLOR="0;31" 2024-04-27T01:48:42.782 INFO:teuthology.orchestra.run.smithi052.stdout:LOGO="fedora-logo-icon" 2024-04-27T01:48:42.782 INFO:teuthology.orchestra.run.smithi052.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-27T01:48:42.782 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://centos.org/" 2024-04-27T01:48:42.782 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-27T01:48:42.782 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-27T01:48:42.782 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-27T01:48:42.782 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2024-04-27T01:48:42.803 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-27T01:48:42.813 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-27T01:48:42.819 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-27T01:48:42.819 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2024-04-27T01:48:42.822 DEBUG:teuthology.orchestra.run.smithi052:> test '!' -e /home/ubuntu/cephtest 2024-04-27T01:48:42.838 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-27T01:48:42.844 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-27T01:48:42.844 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2024-04-27T01:48:42.880 DEBUG:teuthology.orchestra.run.smithi052:> test -z $(ls -A /var/lib/ceph) 2024-04-27T01:48:42.905 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-27T01:48:42.963 INFO:teuthology.run_tasks:Running task kernel... 2024-04-27T01:48:43.001 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-27T01:48:43.001 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-27T01:48:43.001 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-27T01:48:43.002 DEBUG:teuthology.orchestra.run.smithi052:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-27T01:48:43.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T01:48:43.019 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-04-27T01:48:43.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T01:48:43.020 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-04-27T01:48:43.079 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-437.el9.x86_64 2024-04-27T01:48:43.079 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-437.el9.x86_64 2024-04-27T01:48:43.079 INFO:teuthology.task.kernel:Running kernel on smithi029: 5.14.0-437.el9.x86_64 2024-04-27T01:48:43.079 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum install -y kernel 2024-04-27T01:48:43.080 INFO:teuthology.task.kernel:Running kernel on smithi052: 5.14.0-437.el9.x86_64 2024-04-27T01:48:43.080 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum install -y kernel 2024-04-27T01:48:45.537 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 8.6 MB/s | 8.0 MB 00:00 2024-04-27T01:48:47.563 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-04-27T01:48:52.214 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 6.5 MB/s | 6.2 MB 00:00 2024-04-27T01:48:53.898 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-04-27T01:48:54.364 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 937 kB/s | 8.0 MB 00:08 2024-04-27T01:48:55.133 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-04-27T01:49:00.366 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-04-27T01:49:02.281 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-27T01:49:02.281 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-27T01:49:02.383 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:49:02.384 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-04-27T01:49:02.384 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:49:02.483 DEBUG:teuthology.orchestra.run.smithi052:> echo no | sudo yum reinstall kernel || true 2024-04-27T01:49:03.007 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:03 ago on Sat 27 Apr 2024 01:49:00 AM UTC. 2024-04-27T01:49:03.131 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:49:03.148 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:49:03.148 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-04-27T01:49:03.148 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:49:03.148 INFO:teuthology.orchestra.run.smithi052.stdout:Reinstalling: 2024-04-27T01:49:03.148 INFO:teuthology.orchestra.run.smithi052.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-27T01:49:03.148 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:49:03.149 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-04-27T01:49:03.149 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:49:03.149 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:49:03.149 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 255 k 2024-04-27T01:49:03.149 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 0 2024-04-27T01:49:03.149 INFO:teuthology.orchestra.run.smithi052.stderr:Operation aborted. 2024-04-27T01:49:03.200 INFO:teuthology.orchestra.run.smithi052.stdout:Is this ok [y/N]: 2024-04-27T01:49:03.200 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum reinstall -y kernel || true 2024-04-27T01:49:03.717 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:03 ago on Sat 27 Apr 2024 01:49:00 AM UTC. 2024-04-27T01:49:03.836 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:49:03.837 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:49:03.837 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-04-27T01:49:03.837 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:49:03.837 INFO:teuthology.orchestra.run.smithi052.stdout:Reinstalling: 2024-04-27T01:49:03.837 INFO:teuthology.orchestra.run.smithi052.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-27T01:49:03.838 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:49:03.838 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-04-27T01:49:03.838 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:49:03.838 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:49:03.838 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 255 k 2024-04-27T01:49:03.838 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 0 2024-04-27T01:49:03.838 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-04-27T01:49:04.051 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-437.el9.x86_64.rpm 2.6 MB/s | 255 kB 00:00 2024-04-27T01:49:04.052 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:49:04.052 INFO:teuthology.orchestra.run.smithi052.stdout:Total 1.2 MB/s | 255 kB 00:00 2024-04-27T01:49:04.071 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-04-27T01:49:04.270 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-04-27T01:49:04.270 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-04-27T01:49:04.275 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-04-27T01:49:04.275 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-04-27T01:49:04.438 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-04-27T01:49:04.511 INFO:teuthology.orchestra.run.smithi052.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-27T01:49:04.511 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T01:49:05.206 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T01:49:05.206 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-27T01:49:05.920 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T01:49:05.920 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:49:05.920 INFO:teuthology.orchestra.run.smithi052.stdout:Reinstalled: 2024-04-27T01:49:05.921 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-27T01:49:05.921 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:49:05.921 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:49:06.099 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2024-04-27T01:49:06.147 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-27T01:49:06.147 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-27T01:49:06.147 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-27T01:49:06.147 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-27T01:49:06.147 DEBUG:teuthology.orchestra.run.smithi052:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-27T01:49:06.176 INFO:teuthology.orchestra.run.smithi052.stdout:ttyS1 2024-04-27T01:49:06.197 DEBUG:teuthology.parallel:result is None 2024-04-27T01:49:17.056 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 903 kB/s | 19 MB 00:21 2024-04-27T01:49:28.599 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 810 kB/s | 6.2 MB 00:07 2024-04-27T01:49:30.027 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 16 kB 00:00 2024-04-27T01:49:31.949 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-27T01:49:37.164 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-27T01:49:39.138 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-27T01:49:39.138 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-27T01:49:39.257 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:49:39.258 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-27T01:49:39.258 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:49:39.393 DEBUG:teuthology.orchestra.run.smithi029:> echo no | sudo yum reinstall kernel || true 2024-04-27T01:49:39.918 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:02 ago on Sat 27 Apr 2024 01:49:37 AM UTC. 2024-04-27T01:49:40.057 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:49:40.083 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:49:40.083 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-04-27T01:49:40.083 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:49:40.083 INFO:teuthology.orchestra.run.smithi029.stdout:Reinstalling: 2024-04-27T01:49:40.084 INFO:teuthology.orchestra.run.smithi029.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-27T01:49:40.084 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:49:40.084 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-27T01:49:40.084 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:49:40.084 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:49:40.084 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 255 k 2024-04-27T01:49:40.084 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 0 2024-04-27T01:49:40.084 INFO:teuthology.orchestra.run.smithi029.stderr:Operation aborted. 2024-04-27T01:49:40.156 INFO:teuthology.orchestra.run.smithi029.stdout:Is this ok [y/N]: 2024-04-27T01:49:40.156 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum reinstall -y kernel || true 2024-04-27T01:49:40.670 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:03 ago on Sat 27 Apr 2024 01:49:37 AM UTC. 2024-04-27T01:49:40.792 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:49:40.793 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:49:40.793 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-04-27T01:49:40.793 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:49:40.793 INFO:teuthology.orchestra.run.smithi029.stdout:Reinstalling: 2024-04-27T01:49:40.793 INFO:teuthology.orchestra.run.smithi029.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-27T01:49:40.793 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:49:40.793 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-27T01:49:40.793 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:49:40.793 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:49:40.793 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 255 k 2024-04-27T01:49:40.794 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 0 2024-04-27T01:49:40.794 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-27T01:49:41.287 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.1 MB/s | 255 kB 00:00 2024-04-27T01:49:41.287 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:49:41.288 INFO:teuthology.orchestra.run.smithi029.stdout:Total 515 kB/s | 255 kB 00:00 2024-04-27T01:49:41.310 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-27T01:49:41.522 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-27T01:49:41.522 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-27T01:49:41.527 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-27T01:49:41.528 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-27T01:49:41.699 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-27T01:49:41.789 INFO:teuthology.orchestra.run.smithi029.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-27T01:49:41.789 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T01:49:42.840 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T01:49:42.840 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-27T01:49:43.703 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T01:49:43.703 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:49:43.703 INFO:teuthology.orchestra.run.smithi029.stdout:Reinstalled: 2024-04-27T01:49:43.703 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-27T01:49:43.703 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:49:43.703 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:49:43.931 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-04-27T01:49:43.981 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-27T01:49:43.981 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-27T01:49:43.981 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-27T01:49:43.982 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-27T01:49:43.982 DEBUG:teuthology.orchestra.run.smithi029:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-27T01:49:44.018 INFO:teuthology.orchestra.run.smithi029.stdout:ttyS1 2024-04-27T01:49:44.038 DEBUG:teuthology.parallel:result is None 2024-04-27T01:49:44.038 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-27T01:49:44.046 INFO:teuthology.task.internal:Creating test directory... 2024-04-27T01:49:44.046 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-27T01:49:44.081 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-27T01:49:44.100 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-27T01:49:44.128 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-27T01:49:44.157 INFO:teuthology.task.internal:Creating archive directory... 2024-04-27T01:49:44.157 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-27T01:49:44.160 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-27T01:49:44.194 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-27T01:49:44.200 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-27T01:49:44.201 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-27T01:49:44.220 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-27T01:49:44.251 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T01:49:44.261 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T01:49:44.265 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T01:49:44.276 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T01:49:44.278 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-27T01:49:44.285 INFO:teuthology.task.internal:Configuring sudo... 2024-04-27T01:49:44.285 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-27T01:49:44.305 DEBUG:teuthology.orchestra.run.smithi052:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-27T01:49:44.345 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-27T01:49:44.354 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-27T01:49:44.354 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-27T01:49:44.371 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-27T01:49:44.401 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T01:49:44.469 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T01:49:44.513 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T01:49:44.594 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T01:49:44.636 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-27T01:49:44.636 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-27T01:49:44.701 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T01:49:44.748 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T01:49:44.797 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T01:49:44.881 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T01:49:44.926 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-27T01:49:44.926 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-27T01:49:44.994 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2024-04-27T01:49:44.996 DEBUG:teuthology.orchestra.run.smithi052:> sudo service rsyslog restart 2024-04-27T01:49:45.050 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T01:49:45.077 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T01:49:45.544 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-27T01:49:45.552 INFO:teuthology.task.internal:Starting timer... 2024-04-27T01:49:45.552 INFO:teuthology.run_tasks:Running task pcp... 2024-04-27T01:49:45.565 INFO:teuthology.run_tasks:Running task selinux... 2024-04-27T01:49:45.650 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-27T01:49:45.651 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2024-04-27T01:49:45.751 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2024-04-27T01:49:45.753 DEBUG:teuthology.orchestra.run.smithi052:> sudo service auditd rotate 2024-04-27T01:49:45.820 INFO:teuthology.orchestra.run.smithi052.stdout:Rotating logs: 2024-04-27T01:49:45.822 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-27T01:49:45.822 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2024-04-27T01:49:45.850 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2024-04-27T01:49:45.850 DEBUG:teuthology.orchestra.run.smithi052:> /usr/sbin/getenforce 2024-04-27T01:49:45.871 INFO:teuthology.orchestra.run.smithi052.stdout:Permissive 2024-04-27T01:49:45.872 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi029.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi052.front.sepia.ceph.com': 'permissive'} 2024-04-27T01:49:45.872 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-27T01:49:45.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T01:49:45.917 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-27T01:49:45.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T01:49:45.943 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-27T01:49:45.943 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2024-04-27T01:49:45.986 DEBUG:teuthology.orchestra.run.smithi052:> sudo /usr/sbin/setenforce permissive 2024-04-27T01:49:46.011 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-27T01:49:46.021 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-27T01:49:46.026 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-27T01:49:46.137 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-27T01:49:46.153 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-27T01:49:46.154 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi052.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-27T01:53:20.115 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi052.front.sepia.ceph.com')] 2024-04-27T01:53:20.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-04-27T01:53:20.118 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T01:53:20.195 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-04-27T01:53:20.278 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-04-27T01:53:20.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T01:53:20.279 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T01:53:20.352 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-04-27T01:53:20.435 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T01:53:20.435 INFO:teuthology.run_tasks:Running task clock... 2024-04-27T01:53:20.445 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-27T01:53:20.446 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-27T01:53:20.446 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T01:53:20.448 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-27T01:53:20.449 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T01:53:20.477 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-27T01:53:20.491 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-27T01:53:20.507 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-27T01:53:20.516 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2024-04-27T01:53:20.521 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-27T01:53:20.526 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2024-04-27T01:53:20.540 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-27T01:53:20.546 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: ntpd: command not found 2024-04-27T01:53:20.554 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-27T01:53:20.557 INFO:teuthology.orchestra.run.smithi052.stdout:506 Cannot talk to daemon 2024-04-27T01:53:20.572 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-27T01:53:20.585 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-27T01:53:20.612 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-04-27T01:53:20.616 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T01:53:20.616 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-04-27T01:53:20.616 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T01:53:20.617 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T01:53:20.617 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T01:53:20.617 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T01:53:20.646 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2024-04-27T01:53:20.649 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T01:53:20.650 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-04-27T01:53:20.650 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T01:53:20.650 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T01:53:20.650 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T01:53:20.650 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T01:53:20.651 INFO:teuthology.run_tasks:Running task install... 2024-04-27T01:53:20.659 DEBUG:teuthology.task.install:project ceph 2024-04-27T01:53:20.659 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a'}} 2024-04-27T01:53:20.659 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v18.2.0', 'flavor': 'default', 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a'} 2024-04-27T01:53:20.660 INFO:teuthology.task.install:Using flavor: default 2024-04-27T01:53:20.666 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-27T01:53:20.666 INFO:teuthology.task.install:extra packages: [] 2024-04-27T01:53:20.666 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a', 'tag': 'v18.2.0', 'wait_for_package': False} 2024-04-27T01:53:20.667 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-27T01:53:20.667 INFO:teuthology.packaging:ref: None 2024-04-27T01:53:20.667 INFO:teuthology.packaging:tag: v18.2.0 2024-04-27T01:53:20.667 INFO:teuthology.packaging:branch: None 2024-04-27T01:53:20.667 INFO:teuthology.packaging:sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T01:53:22.835 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph v18.2.0^{} -> 5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-04-27T01:53:22.836 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-04-27T01:53:22.838 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a', 'tag': 'v18.2.0', 'wait_for_package': False} 2024-04-27T01:53:22.839 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-27T01:53:22.839 INFO:teuthology.packaging:ref: None 2024-04-27T01:53:22.839 INFO:teuthology.packaging:tag: v18.2.0 2024-04-27T01:53:22.839 INFO:teuthology.packaging:branch: None 2024-04-27T01:53:22.839 INFO:teuthology.packaging:sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T01:53:22.839 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-04-27T01:53:22.981 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/ 2024-04-27T01:53:22.981 INFO:teuthology.task.install.rpm:Package version is 18.2.0-0 2024-04-27T01:53:23.013 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-27T01:53:23.013 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-27T01:53:23.014 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-27T01:53:23.044 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-27T01:53:23.044 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-27T01:53:23.044 INFO:teuthology.packaging:ref: None 2024-04-27T01:53:23.044 INFO:teuthology.packaging:tag: v18.2.0 2024-04-27T01:53:23.044 INFO:teuthology.packaging:branch: None 2024-04-27T01:53:23.044 INFO:teuthology.packaging:sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T01:53:23.045 DEBUG:teuthology.orchestra.run.smithi052:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-27T01:53:23.061 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/ 2024-04-27T01:53:23.061 INFO:teuthology.task.install.rpm:Package version is 18.2.0-0 2024-04-27T01:53:23.086 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-27T01:53:23.086 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-27T01:53:23.086 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-27T01:53:23.115 DEBUG:teuthology.orchestra.run.smithi052:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-27T01:53:23.116 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-27T01:53:23.117 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-27T01:53:23.117 INFO:teuthology.packaging:ref: None 2024-04-27T01:53:23.117 INFO:teuthology.packaging:tag: v18.2.0 2024-04-27T01:53:23.117 INFO:teuthology.packaging:branch: None 2024-04-27T01:53:23.117 INFO:teuthology.packaging:sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T01:53:23.117 DEBUG:teuthology.orchestra.run.smithi029:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-27T01:53:23.187 DEBUG:teuthology.orchestra.run.smithi029:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-27T01:53:23.195 DEBUG:teuthology.orchestra.run.smithi052:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-27T01:53:23.263 INFO:teuthology.orchestra.run.smithi052.stdout:check_obsoletes = 1 2024-04-27T01:53:23.264 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum clean all 2024-04-27T01:53:23.267 DEBUG:teuthology.orchestra.run.smithi029:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-27T01:53:23.334 INFO:teuthology.orchestra.run.smithi029.stdout:check_obsoletes = 1 2024-04-27T01:53:23.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum clean all 2024-04-27T01:53:23.506 INFO:teuthology.orchestra.run.smithi052.stdout:48 files removed 2024-04-27T01:53:23.533 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-radosgw 2024-04-27T01:53:23.576 INFO:teuthology.orchestra.run.smithi029.stdout:48 files removed 2024-04-27T01:53:23.603 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw 2024-04-27T01:53:23.956 INFO:teuthology.orchestra.run.smithi052.stdout:ceph packages for x86_64 507 kB/s | 76 kB 00:00 2024-04-27T01:53:24.024 INFO:teuthology.orchestra.run.smithi029.stdout:ceph packages for x86_64 514 kB/s | 76 kB 00:00 2024-04-27T01:53:24.157 INFO:teuthology.orchestra.run.smithi052.stdout:ceph noarch packages 60 kB/s | 9.3 kB 00:00 2024-04-27T01:53:24.200 INFO:teuthology.orchestra.run.smithi029.stdout:ceph noarch packages 71 kB/s | 9.3 kB 00:00 2024-04-27T01:53:24.316 INFO:teuthology.orchestra.run.smithi052.stdout:ceph source packages 17 kB/s | 2.2 kB 00:00 2024-04-27T01:53:24.360 INFO:teuthology.orchestra.run.smithi029.stdout:ceph source packages 17 kB/s | 2.2 kB 00:00 2024-04-27T01:53:24.958 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-27T01:53:25.247 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 9.4 MB/s | 8.0 MB 00:00 2024-04-27T01:53:27.061 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-04-27T01:53:29.673 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 5.7 MB/s | 19 MB 00:03 2024-04-27T01:53:31.510 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 8.2 MB/s | 6.2 MB 00:00 2024-04-27T01:53:32.936 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 16 kB 00:00 2024-04-27T01:53:34.134 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 8.5 MB/s | 6.2 MB 00:00 2024-04-27T01:53:34.262 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-04-27T01:53:35.586 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 16 kB 00:00 2024-04-27T01:53:37.426 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-27T01:53:39.546 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-04-27T01:53:41.772 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:53:41.774 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:53:41.775 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-04-27T01:53:41.775 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:53:41.775 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-04-27T01:53:41.775 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw x86_64 2:18.2.0-0.el9 ceph 7.6 M 2024-04-27T01:53:41.775 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrading: 2024-04-27T01:53:41.775 INFO:teuthology.orchestra.run.smithi052.stdout: librados2 x86_64 2:18.2.0-0.el9 ceph 3.3 M 2024-04-27T01:53:41.775 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1 x86_64 2:18.2.0-0.el9 ceph 3.0 M 2024-04-27T01:53:41.775 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-04-27T01:53:41.775 INFO:teuthology.orchestra.run.smithi052.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-27T01:53:41.775 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base x86_64 2:18.2.0-0.el9 ceph 5.2 M 2024-04-27T01:53:41.775 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common x86_64 2:18.2.0-0.el9 ceph 18 M 2024-04-27T01:53:41.775 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux x86_64 2:18.2.0-0.el9 ceph 24 k 2024-04-27T01:53:41.775 INFO:teuthology.orchestra.run.smithi052.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-27T01:53:41.775 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-27T01:53:41.776 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-27T01:53:41.776 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-27T01:53:41.776 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2 x86_64 2:18.2.0-0.el9 ceph 653 k 2024-04-27T01:53:41.776 INFO:teuthology.orchestra.run.smithi052.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-27T01:53:41.776 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-27T01:53:41.776 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-27T01:53:41.776 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1 x86_64 2:18.2.0-0.el9 ceph 474 k 2024-04-27T01:53:41.776 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-27T01:53:41.776 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2 x86_64 2:18.2.0-0.el9 ceph 4.4 M 2024-04-27T01:53:41.776 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-27T01:53:41.776 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-27T01:53:41.776 INFO:teuthology.orchestra.run.smithi052.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-27T01:53:41.776 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse x86_64 2:18.2.0-0.el9 ceph 45 k 2024-04-27T01:53:41.776 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common x86_64 2:18.2.0-0.el9 ceph 119 k 2024-04-27T01:53:41.776 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs x86_64 2:18.2.0-0.el9 ceph 155 k 2024-04-27T01:53:41.777 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-27T01:53:41.777 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados x86_64 2:18.2.0-0.el9 ceph 321 k 2024-04-27T01:53:41.777 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd x86_64 2:18.2.0-0.el9 ceph 297 k 2024-04-27T01:53:41.777 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw x86_64 2:18.2.0-0.el9 ceph 99 k 2024-04-27T01:53:41.777 INFO:teuthology.orchestra.run.smithi052.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-27T01:53:41.777 INFO:teuthology.orchestra.run.smithi052.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-27T01:53:41.777 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:53:41.777 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-04-27T01:53:41.777 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:53:41.777 INFO:teuthology.orchestra.run.smithi052.stdout:Install 28 Packages 2024-04-27T01:53:41.777 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrade 2 Packages 2024-04-27T01:53:41.777 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:53:41.778 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 53 M 2024-04-27T01:53:41.778 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-04-27T01:53:42.397 INFO:teuthology.orchestra.run.smithi052.stdout:(1/30): ceph-base-18.2.0-0.el9.x86_64.rpm 28 MB/s | 5.2 MB 00:00 2024-04-27T01:53:42.422 INFO:teuthology.orchestra.run.smithi052.stdout:(2/30): ceph-selinux-18.2.0-0.el9.x86_64.rpm 971 kB/s | 24 kB 00:00 2024-04-27T01:53:42.456 INFO:teuthology.orchestra.run.smithi052.stdout:(3/30): libcephfs2-18.2.0-0.el9.x86_64.rpm 19 MB/s | 653 kB 00:00 2024-04-27T01:53:42.489 INFO:teuthology.orchestra.run.smithi052.stdout:(4/30): libradosstriper1-18.2.0-0.el9.x86_64.rp 14 MB/s | 474 kB 00:00 2024-04-27T01:53:42.590 INFO:teuthology.orchestra.run.smithi052.stdout:(5/30): librgw2-18.2.0-0.el9.x86_64.rpm 44 MB/s | 4.4 MB 00:00 2024-04-27T01:53:42.615 INFO:teuthology.orchestra.run.smithi052.stdout:(6/30): python3-ceph-argparse-18.2.0-0.el9.x86_ 1.8 MB/s | 45 kB 00:00 2024-04-27T01:53:42.640 INFO:teuthology.orchestra.run.smithi052.stdout:(7/30): python3-ceph-common-18.2.0-0.el9.x86_64 4.7 MB/s | 119 kB 00:00 2024-04-27T01:53:42.665 INFO:teuthology.orchestra.run.smithi052.stdout:(8/30): python3-cephfs-18.2.0-0.el9.x86_64.rpm 6.1 MB/s | 155 kB 00:00 2024-04-27T01:53:42.690 INFO:teuthology.orchestra.run.smithi052.stdout:(9/30): python3-rados-18.2.0-0.el9.x86_64.rpm 13 MB/s | 321 kB 00:00 2024-04-27T01:53:42.724 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-04-27T01:53:42.782 INFO:teuthology.orchestra.run.smithi052.stdout:(10/30): ceph-radosgw-18.2.0-0.el9.x86_64.rpm 13 MB/s | 7.6 MB 00:00 2024-04-27T01:53:42.807 INFO:teuthology.orchestra.run.smithi052.stdout:(11/30): python3-rbd-18.2.0-0.el9.x86_64.rpm 2.5 MB/s | 297 kB 00:00 2024-04-27T01:53:42.841 INFO:teuthology.orchestra.run.smithi052.stdout:(12/30): python3-rgw-18.2.0-0.el9.x86_64.rpm 1.7 MB/s | 99 kB 00:00 2024-04-27T01:53:43.050 INFO:teuthology.orchestra.run.smithi052.stdout:(13/30): ceph-common-18.2.0-0.el9.x86_64.rpm 22 MB/s | 18 MB 00:00 2024-04-27T01:53:43.092 INFO:teuthology.orchestra.run.smithi052.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 376 kB/s | 107 kB 00:00 2024-04-27T01:53:43.134 INFO:teuthology.orchestra.run.smithi052.stdout:(15/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 548 kB/s | 160 kB 00:00 2024-04-27T01:53:43.176 INFO:teuthology.orchestra.run.smithi052.stdout:(16/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 361 kB/s | 45 kB 00:00 2024-04-27T01:53:43.209 INFO:teuthology.orchestra.run.smithi052.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-04-27T01:53:43.235 INFO:teuthology.orchestra.run.smithi052.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-04-27T01:53:43.268 INFO:teuthology.orchestra.run.smithi052.stdout:(19/30): python3-prettytable-0.7.2-27.el9.noarc 458 kB/s | 42 kB 00:00 2024-04-27T01:53:43.585 INFO:teuthology.orchestra.run.smithi052.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 295 kB/s | 111 kB 00:00 2024-04-27T01:53:43.669 INFO:teuthology.orchestra.run.smithi052.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 311 kB/s | 26 kB 00:00 2024-04-27T01:53:43.695 INFO:teuthology.orchestra.run.smithi052.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 672 kB/s | 309 kB 00:00 2024-04-27T01:53:43.753 INFO:teuthology.orchestra.run.smithi052.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 581 kB/s | 49 kB 00:00 2024-04-27T01:53:43.779 INFO:teuthology.orchestra.run.smithi052.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 806 kB/s | 67 kB 00:00 2024-04-27T01:53:43.870 INFO:teuthology.orchestra.run.smithi052.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-04-27T01:53:43.937 INFO:teuthology.orchestra.run.smithi052.stdout:(26/30): libarrow-9.0.0-11.el9.x86_64.rpm 6.6 MB/s | 4.4 MB 00:00 2024-04-27T01:53:44.023 INFO:teuthology.orchestra.run.smithi052.stdout:(27/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.0 MB/s | 839 kB 00:00 2024-04-27T01:53:44.182 INFO:teuthology.orchestra.run.smithi052.stdout:(28/30): librados2-18.2.0-0.el9.x86_64.rpm 13 MB/s | 3.3 MB 00:00 2024-04-27T01:53:44.230 INFO:teuthology.orchestra.run.smithi052.stdout:(29/30): thrift-0.15.0-2.el9.x86_64.rpm 4.4 MB/s | 1.6 MB 00:00 2024-04-27T01:53:44.280 INFO:teuthology.orchestra.run.smithi052.stdout:(30/30): librbd1-18.2.0-0.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-04-27T01:53:44.286 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:53:44.286 INFO:teuthology.orchestra.run.smithi052.stdout:Total 21 MB/s | 53 MB 00:02 2024-04-27T01:53:44.361 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-04-27T01:53:44.401 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-04-27T01:53:44.402 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-04-27T01:53:44.846 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-04-27T01:53:44.847 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-04-27T01:53:45.030 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:53:45.032 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:53:45.032 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-27T01:53:45.032 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:53:45.033 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-27T01:53:45.033 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw x86_64 2:18.2.0-0.el9 ceph 7.6 M 2024-04-27T01:53:45.033 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2024-04-27T01:53:45.033 INFO:teuthology.orchestra.run.smithi029.stdout: librados2 x86_64 2:18.2.0-0.el9 ceph 3.3 M 2024-04-27T01:53:45.033 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1 x86_64 2:18.2.0-0.el9 ceph 3.0 M 2024-04-27T01:53:45.033 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-27T01:53:45.033 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-27T01:53:45.033 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base x86_64 2:18.2.0-0.el9 ceph 5.2 M 2024-04-27T01:53:45.033 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common x86_64 2:18.2.0-0.el9 ceph 18 M 2024-04-27T01:53:45.033 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux x86_64 2:18.2.0-0.el9 ceph 24 k 2024-04-27T01:53:45.033 INFO:teuthology.orchestra.run.smithi029.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-27T01:53:45.033 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-27T01:53:45.033 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-27T01:53:45.034 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-27T01:53:45.034 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2 x86_64 2:18.2.0-0.el9 ceph 653 k 2024-04-27T01:53:45.034 INFO:teuthology.orchestra.run.smithi029.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-27T01:53:45.034 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-27T01:53:45.034 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-27T01:53:45.034 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 x86_64 2:18.2.0-0.el9 ceph 474 k 2024-04-27T01:53:45.034 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-27T01:53:45.034 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2 x86_64 2:18.2.0-0.el9 ceph 4.4 M 2024-04-27T01:53:45.034 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-27T01:53:45.034 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-27T01:53:45.034 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-27T01:53:45.034 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse x86_64 2:18.2.0-0.el9 ceph 45 k 2024-04-27T01:53:45.035 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common x86_64 2:18.2.0-0.el9 ceph 119 k 2024-04-27T01:53:45.035 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs x86_64 2:18.2.0-0.el9 ceph 155 k 2024-04-27T01:53:45.035 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-27T01:53:45.035 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados x86_64 2:18.2.0-0.el9 ceph 321 k 2024-04-27T01:53:45.035 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd x86_64 2:18.2.0-0.el9 ceph 297 k 2024-04-27T01:53:45.035 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw x86_64 2:18.2.0-0.el9 ceph 99 k 2024-04-27T01:53:45.035 INFO:teuthology.orchestra.run.smithi029.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-27T01:53:45.035 INFO:teuthology.orchestra.run.smithi029.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-27T01:53:45.035 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:53:45.035 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-27T01:53:45.035 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:53:45.035 INFO:teuthology.orchestra.run.smithi029.stdout:Install 28 Packages 2024-04-27T01:53:45.035 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 2 Packages 2024-04-27T01:53:45.035 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:53:45.036 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 53 M 2024-04-27T01:53:45.036 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-27T01:53:45.685 INFO:teuthology.orchestra.run.smithi029.stdout:(1/30): ceph-base-18.2.0-0.el9.x86_64.rpm 21 MB/s | 5.2 MB 00:00 2024-04-27T01:53:45.711 INFO:teuthology.orchestra.run.smithi029.stdout:(2/30): ceph-selinux-18.2.0-0.el9.x86_64.rpm 956 kB/s | 24 kB 00:00 2024-04-27T01:53:45.753 INFO:teuthology.orchestra.run.smithi029.stdout:(3/30): libcephfs2-18.2.0-0.el9.x86_64.rpm 15 MB/s | 653 kB 00:00 2024-04-27T01:53:45.786 INFO:teuthology.orchestra.run.smithi029.stdout:(4/30): libradosstriper1-18.2.0-0.el9.x86_64.rp 14 MB/s | 474 kB 00:00 2024-04-27T01:53:45.954 INFO:teuthology.orchestra.run.smithi029.stdout:(5/30): ceph-radosgw-18.2.0-0.el9.x86_64.rpm 15 MB/s | 7.6 MB 00:00 2024-04-27T01:53:45.979 INFO:teuthology.orchestra.run.smithi029.stdout:(6/30): python3-ceph-argparse-18.2.0-0.el9.x86_ 1.7 MB/s | 45 kB 00:00 2024-04-27T01:53:46.005 INFO:teuthology.orchestra.run.smithi029.stdout:(7/30): python3-ceph-common-18.2.0-0.el9.x86_64 4.6 MB/s | 119 kB 00:00 2024-04-27T01:53:46.032 INFO:teuthology.orchestra.run.smithi029.stdout:(8/30): python3-cephfs-18.2.0-0.el9.x86_64.rpm 5.7 MB/s | 155 kB 00:00 2024-04-27T01:53:46.066 INFO:teuthology.orchestra.run.smithi029.stdout:(9/30): python3-rados-18.2.0-0.el9.x86_64.rpm 9.3 MB/s | 321 kB 00:00 2024-04-27T01:53:46.099 INFO:teuthology.orchestra.run.smithi029.stdout:(10/30): python3-rbd-18.2.0-0.el9.x86_64.rpm 8.8 MB/s | 297 kB 00:00 2024-04-27T01:53:46.133 INFO:teuthology.orchestra.run.smithi029.stdout:(11/30): python3-rgw-18.2.0-0.el9.x86_64.rpm 2.9 MB/s | 99 kB 00:00 2024-04-27T01:53:46.234 INFO:teuthology.orchestra.run.smithi029.stdout:(12/30): librgw2-18.2.0-0.el9.x86_64.rpm 9.9 MB/s | 4.4 MB 00:00 2024-04-27T01:53:46.249 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-04-27T01:53:46.331 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-27T01:53:46.360 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-27T01:53:46.410 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-27T01:53:46.576 INFO:teuthology.orchestra.run.smithi029.stdout:(13/30): ceph-common-18.2.0-0.el9.x86_64.rpm 16 MB/s | 18 MB 00:01 2024-04-27T01:53:46.789 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-27T01:53:46.844 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librados2-2:18.2.0-0.el9.x86_64 5/32 2024-04-27T01:53:46.937 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:18.2.0-0.el9.x86_64 5/32 2024-04-27T01:53:47.027 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rados-2:18.2.0-0.el9.x86_64 6/32 2024-04-27T01:53:47.055 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs2-2:18.2.0-0.el9.x86_64 7/32 2024-04-27T01:53:47.242 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs2-2:18.2.0-0.el9.x86_64 7/32 2024-04-27T01:53:47.288 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-27T01:53:47.331 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-argparse-2:18.2.0-0.el9.x86_64 9/32 2024-04-27T01:53:47.399 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cephfs-2:18.2.0-0.el9.x86_64 10/32 2024-04-27T01:53:47.423 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libradosstriper1-2:18.2.0-0.el9.x86_64 11/32 2024-04-27T01:53:47.489 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libradosstriper1-2:18.2.0-0.el9.x86_64 11/32 2024-04-27T01:53:47.600 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-27T01:53:47.630 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-04-27T01:53:47.670 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-27T01:53:47.704 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-27T01:53:47.731 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-04-27T01:53:47.767 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-27T01:53:47.800 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-27T01:53:48.129 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-27T01:53:48.171 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librbd1-2:18.2.0-0.el9.x86_64 20/32 2024-04-27T01:53:48.262 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:18.2.0-0.el9.x86_64 20/32 2024-04-27T01:53:48.292 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rbd-2:18.2.0-0.el9.x86_64 21/32 2024-04-27T01:53:48.329 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-27T01:53:48.744 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-04-27T01:53:48.774 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librgw2-2:18.2.0-0.el9.x86_64 24/32 2024-04-27T01:53:48.845 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librgw2-2:18.2.0-0.el9.x86_64 24/32 2024-04-27T01:53:48.923 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rgw-2:18.2.0-0.el9.x86_64 25/32 2024-04-27T01:53:48.951 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-common-2:18.2.0-0.el9.x86_64 26/32 2024-04-27T01:53:50.802 INFO:teuthology.orchestra.run.smithi029.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 11 kB/s | 45 kB 00:04 2024-04-27T01:53:50.835 INFO:teuthology.orchestra.run.smithi029.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 23 kB/s | 107 kB 00:04 2024-04-27T01:53:50.861 INFO:teuthology.orchestra.run.smithi029.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 35 kB/s | 160 kB 00:04 2024-04-27T01:53:50.920 INFO:teuthology.orchestra.run.smithi029.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 723 kB/s | 42 kB 00:00 2024-04-27T01:53:50.953 INFO:teuthology.orchestra.run.smithi029.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-04-27T01:53:50.996 INFO:teuthology.orchestra.run.smithi029.stdout:(19/30): librdkafka-1.6.1-102.el9.x86_64.rpm 3.3 MB/s | 662 kB 00:00 2024-04-27T01:53:51.112 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-04-27T01:53:51.156 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-04-27T01:53:51.513 INFO:teuthology.orchestra.run.smithi029.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 187 kB/s | 111 kB 00:00 2024-04-27T01:53:51.630 INFO:teuthology.orchestra.run.smithi029.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 223 kB/s | 26 kB 00:00 2024-04-27T01:53:51.655 INFO:teuthology.orchestra.run.smithi029.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 440 kB/s | 309 kB 00:00 2024-04-27T01:53:51.721 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-04-27T01:53:51.756 INFO:teuthology.orchestra.run.smithi029.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 386 kB/s | 49 kB 00:00 2024-04-27T01:53:51.759 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-base-2:18.2.0-0.el9.x86_64 28/32 2024-04-27T01:53:51.784 INFO:teuthology.orchestra.run.smithi029.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 525 kB/s | 67 kB 00:00 2024-04-27T01:53:51.851 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-base-2:18.2.0-0.el9.x86_64 28/32 2024-04-27T01:53:51.852 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-27T01:53:51.852 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-27T01:53:51.852 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:53:51.878 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-selinux-2:18.2.0-0.el9.x86_64 29/32 2024-04-27T01:53:51.935 INFO:teuthology.orchestra.run.smithi029.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-04-27T01:53:52.068 INFO:teuthology.orchestra.run.smithi029.stdout:(26/30): libarrow-9.0.0-11.el9.x86_64.rpm 4.1 MB/s | 4.4 MB 00:01 2024-04-27T01:53:52.178 INFO:teuthology.orchestra.run.smithi029.stdout:(27/30): librados2-18.2.0-0.el9.x86_64.rpm 30 MB/s | 3.3 MB 00:00 2024-04-27T01:53:52.211 INFO:teuthology.orchestra.run.smithi029.stdout:(28/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 1.8 MB/s | 839 kB 00:00 2024-04-27T01:53:52.303 INFO:teuthology.orchestra.run.smithi029.stdout:(29/30): librbd1-18.2.0-0.el9.x86_64.rpm 24 MB/s | 3.0 MB 00:00 2024-04-27T01:53:52.353 INFO:teuthology.orchestra.run.smithi029.stdout:(30/30): thrift-0.15.0-2.el9.x86_64.rpm 3.8 MB/s | 1.6 MB 00:00 2024-04-27T01:53:52.359 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:53:52.359 INFO:teuthology.orchestra.run.smithi029.stdout:Total 7.2 MB/s | 53 MB 00:07 2024-04-27T01:53:52.434 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-27T01:53:52.475 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-27T01:53:52.475 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-27T01:53:52.919 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-27T01:53:52.920 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-27T01:53:54.298 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-27T01:53:54.398 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-27T01:53:54.438 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-27T01:53:54.497 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-27T01:53:54.890 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-27T01:53:55.013 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librados2-2:18.2.0-0.el9.x86_64 5/32 2024-04-27T01:53:55.107 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:18.2.0-0.el9.x86_64 5/32 2024-04-27T01:53:55.197 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-2:18.2.0-0.el9.x86_64 6/32 2024-04-27T01:53:55.224 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-2:18.2.0-0.el9.x86_64 7/32 2024-04-27T01:53:55.288 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-2:18.2.0-0.el9.x86_64 7/32 2024-04-27T01:53:55.319 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-27T01:53:55.363 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-argparse-2:18.2.0-0.el9.x86_64 9/32 2024-04-27T01:53:55.439 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-2:18.2.0-0.el9.x86_64 10/32 2024-04-27T01:53:55.472 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradosstriper1-2:18.2.0-0.el9.x86_64 11/32 2024-04-27T01:53:55.538 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libradosstriper1-2:18.2.0-0.el9.x86_64 11/32 2024-04-27T01:53:55.649 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-27T01:53:55.680 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-04-27T01:53:55.721 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-27T01:53:55.756 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-27T01:53:55.792 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-04-27T01:53:55.829 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-27T01:53:55.862 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-27T01:53:56.177 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-27T01:53:56.234 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librbd1-2:18.2.0-0.el9.x86_64 20/32 2024-04-27T01:53:56.326 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:18.2.0-0.el9.x86_64 20/32 2024-04-27T01:53:56.357 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-2:18.2.0-0.el9.x86_64 21/32 2024-04-27T01:53:56.403 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-27T01:53:56.819 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-04-27T01:53:56.857 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librgw2-2:18.2.0-0.el9.x86_64 24/32 2024-04-27T01:53:56.929 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librgw2-2:18.2.0-0.el9.x86_64 24/32 2024-04-27T01:53:57.008 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-2:18.2.0-0.el9.x86_64 25/32 2024-04-27T01:53:57.036 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-common-2:18.2.0-0.el9.x86_64 26/32 2024-04-27T01:53:59.221 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-04-27T01:53:59.273 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-04-27T01:53:59.838 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-04-27T01:53:59.881 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-2:18.2.0-0.el9.x86_64 28/32 2024-04-27T01:53:59.990 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-2:18.2.0-0.el9.x86_64 28/32 2024-04-27T01:53:59.990 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-27T01:53:59.990 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-27T01:53:59.990 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:00.026 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-selinux-2:18.2.0-0.el9.x86_64 29/32 2024-04-27T01:54:01.809 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-selinux-2:18.2.0-0.el9.x86_64 29/32 2024-04-27T01:54:01.809 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /sys 2024-04-27T01:54:01.809 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /proc 2024-04-27T01:54:01.809 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /mnt 2024-04-27T01:54:01.809 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /var/tmp 2024-04-27T01:54:01.809 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /home 2024-04-27T01:54:01.809 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /root 2024-04-27T01:54:01.810 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /tmp 2024-04-27T01:54:01.810 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:02.569 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-radosgw-2:18.2.0-0.el9.x86_64 30/32 2024-04-27T01:54:02.594 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-0.el9.x86_64 30/32 2024-04-27T01:54:02.594 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:54:02.594 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-27T01:54:02.594 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-27T01:54:02.594 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-27T01:54:02.594 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:02.596 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-27T01:54:02.663 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-27T01:54:02.663 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-27T01:54:03.892 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-27T01:54:03.892 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-base-2:18.2.0-0.el9.x86_64 1/32 2024-04-27T01:54:03.892 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-common-2:18.2.0-0.el9.x86_64 2/32 2024-04-27T01:54:03.892 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-radosgw-2:18.2.0-0.el9.x86_64 3/32 2024-04-27T01:54:03.892 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-selinux-2:18.2.0-0.el9.x86_64 4/32 2024-04-27T01:54:03.892 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs2-2:18.2.0-0.el9.x86_64 5/32 2024-04-27T01:54:03.892 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libradosstriper1-2:18.2.0-0.el9.x86_64 6/32 2024-04-27T01:54:03.892 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librgw2-2:18.2.0-0.el9.x86_64 7/32 2024-04-27T01:54:03.892 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-argparse-2:18.2.0-0.el9.x86_64 8/32 2024-04-27T01:54:03.893 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-common-2:18.2.0-0.el9.x86_64 9/32 2024-04-27T01:54:03.893 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cephfs-2:18.2.0-0.el9.x86_64 10/32 2024-04-27T01:54:03.893 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rados-2:18.2.0-0.el9.x86_64 11/32 2024-04-27T01:54:03.893 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rbd-2:18.2.0-0.el9.x86_64 12/32 2024-04-27T01:54:03.893 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rgw-2:18.2.0-0.el9.x86_64 13/32 2024-04-27T01:54:03.893 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-27T01:54:03.893 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-27T01:54:03.893 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-27T01:54:03.893 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-27T01:54:03.893 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-27T01:54:03.893 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-27T01:54:03.893 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-27T01:54:03.893 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-27T01:54:03.893 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-04-27T01:54:03.893 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-04-27T01:54:03.894 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-27T01:54:03.894 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-27T01:54:03.894 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-04-27T01:54:03.894 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-27T01:54:03.894 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-27T01:54:03.894 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:18.2.0-0.el9.x86_64 29/32 2024-04-27T01:54:03.894 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-27T01:54:03.894 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:18.2.0-0.el9.x86_64 31/32 2024-04-27T01:54:04.679 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-27T01:54:04.679 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:04.680 INFO:teuthology.orchestra.run.smithi052.stdout:Upgraded: 2024-04-27T01:54:04.680 INFO:teuthology.orchestra.run.smithi052.stdout: librados2-2:18.2.0-0.el9.x86_64 librbd1-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:04.680 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-04-27T01:54:04.680 INFO:teuthology.orchestra.run.smithi052.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-27T01:54:04.680 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:04.680 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:04.680 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:04.680 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:04.680 INFO:teuthology.orchestra.run.smithi052.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-27T01:54:04.680 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-27T01:54:04.680 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-27T01:54:04.680 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-27T01:54:04.680 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:04.680 INFO:teuthology.orchestra.run.smithi052.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-27T01:54:04.681 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-27T01:54:04.681 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-27T01:54:04.681 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:04.681 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-27T01:54:04.681 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:04.681 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-27T01:54:04.681 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-27T01:54:04.681 INFO:teuthology.orchestra.run.smithi052.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-27T01:54:04.681 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:04.681 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:04.681 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:04.681 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-27T01:54:04.681 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:04.681 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:04.682 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:04.682 INFO:teuthology.orchestra.run.smithi052.stdout: re2-1:20211101-3.el9.x86_64 2024-04-27T01:54:04.682 INFO:teuthology.orchestra.run.smithi052.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-27T01:54:04.682 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:04.682 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:54:04.948 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-test 2024-04-27T01:54:05.486 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:26 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:54:05.610 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:54:05.610 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:05.610 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-04-27T01:54:05.611 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:05.611 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-04-27T01:54:05.611 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test x86_64 2:18.2.0-0.el9 ceph 40 M 2024-04-27T01:54:05.611 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-04-27T01:54:05.611 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite x86_64 2:18.2.0-0.el9 ceph 161 k 2024-04-27T01:54:05.611 INFO:teuthology.orchestra.run.smithi052.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-27T01:54:05.611 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-27T01:54:05.611 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:05.611 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-04-27T01:54:05.611 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:05.612 INFO:teuthology.orchestra.run.smithi052.stdout:Install 4 Packages 2024-04-27T01:54:05.612 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:05.613 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 40 M 2024-04-27T01:54:05.613 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 165 M 2024-04-27T01:54:05.613 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-04-27T01:54:05.803 INFO:teuthology.orchestra.run.smithi052.stdout:(1/4): libcephsqlite-18.2.0-0.el9.x86_64.rpm 1.3 MB/s | 161 kB 00:00 2024-04-27T01:54:05.904 INFO:teuthology.orchestra.run.smithi052.stdout:(2/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 305 kB 00:00 2024-04-27T01:54:05.991 INFO:teuthology.orchestra.run.smithi052.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 340 kB/s | 64 kB 00:00 2024-04-27T01:54:06.427 INFO:teuthology.orchestra.run.smithi052.stdout:(4/4): ceph-test-18.2.0-0.el9.x86_64.rpm 53 MB/s | 40 MB 00:00 2024-04-27T01:54:06.429 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:54:06.429 INFO:teuthology.orchestra.run.smithi052.stdout:Total 49 MB/s | 40 MB 00:00 2024-04-27T01:54:06.456 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-04-27T01:54:06.479 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-04-27T01:54:06.479 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-04-27T01:54:06.812 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-04-27T01:54:06.812 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-04-27T01:54:07.343 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-04-27T01:54:07.443 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-27T01:54:07.507 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-27T01:54:07.551 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephsqlite-2:18.2.0-0.el9.x86_64 3/4 2024-04-27T01:54:11.490 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephsqlite-2:18.2.0-0.el9.x86_64 3/4 2024-04-27T01:54:11.604 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-test-2:18.2.0-0.el9.x86_64 4/4 2024-04-27T01:54:12.103 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-selinux-2:18.2.0-0.el9.x86_64 29/32 2024-04-27T01:54:12.103 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /sys 2024-04-27T01:54:12.104 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /proc 2024-04-27T01:54:12.104 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /mnt 2024-04-27T01:54:12.104 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /var/tmp 2024-04-27T01:54:12.104 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /home 2024-04-27T01:54:12.104 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /root 2024-04-27T01:54:12.104 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /tmp 2024-04-27T01:54:12.104 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:12.873 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-2:18.2.0-0.el9.x86_64 30/32 2024-04-27T01:54:12.899 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-0.el9.x86_64 30/32 2024-04-27T01:54:12.899 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:54:12.900 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-27T01:54:12.900 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-27T01:54:12.900 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-27T01:54:12.900 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:12.902 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-27T01:54:12.996 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-27T01:54:12.996 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-27T01:54:13.523 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-test-2:18.2.0-0.el9.x86_64 4/4 2024-04-27T01:54:13.523 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-test-2:18.2.0-0.el9.x86_64 1/4 2024-04-27T01:54:13.523 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephsqlite-2:18.2.0-0.el9.x86_64 2/4 2024-04-27T01:54:13.523 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-27T01:54:13.799 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-27T01:54:13.799 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:13.799 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-04-27T01:54:13.799 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test-2:18.2.0-0.el9.x86_64 libcephsqlite-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:13.799 INFO:teuthology.orchestra.run.smithi052.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-04-27T01:54:13.800 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:13.800 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:54:14.087 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph 2024-04-27T01:54:14.529 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-27T01:54:14.529 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-2:18.2.0-0.el9.x86_64 1/32 2024-04-27T01:54:14.529 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-2:18.2.0-0.el9.x86_64 2/32 2024-04-27T01:54:14.529 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-2:18.2.0-0.el9.x86_64 3/32 2024-04-27T01:54:14.529 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-selinux-2:18.2.0-0.el9.x86_64 4/32 2024-04-27T01:54:14.529 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-2:18.2.0-0.el9.x86_64 5/32 2024-04-27T01:54:14.530 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradosstriper1-2:18.2.0-0.el9.x86_64 6/32 2024-04-27T01:54:14.530 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librgw2-2:18.2.0-0.el9.x86_64 7/32 2024-04-27T01:54:14.530 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-argparse-2:18.2.0-0.el9.x86_64 8/32 2024-04-27T01:54:14.530 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-common-2:18.2.0-0.el9.x86_64 9/32 2024-04-27T01:54:14.530 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-2:18.2.0-0.el9.x86_64 10/32 2024-04-27T01:54:14.530 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-2:18.2.0-0.el9.x86_64 11/32 2024-04-27T01:54:14.530 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-2:18.2.0-0.el9.x86_64 12/32 2024-04-27T01:54:14.530 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-2:18.2.0-0.el9.x86_64 13/32 2024-04-27T01:54:14.530 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-27T01:54:14.530 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-27T01:54:14.530 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-27T01:54:14.530 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-27T01:54:14.530 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-27T01:54:14.530 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-27T01:54:14.531 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-27T01:54:14.531 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-27T01:54:14.531 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-04-27T01:54:14.531 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-04-27T01:54:14.531 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-27T01:54:14.531 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-27T01:54:14.531 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-04-27T01:54:14.531 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-27T01:54:14.531 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-27T01:54:14.531 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:18.2.0-0.el9.x86_64 29/32 2024-04-27T01:54:14.531 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-27T01:54:14.531 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:18.2.0-0.el9.x86_64 31/32 2024-04-27T01:54:14.611 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:35 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:54:14.739 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:54:14.741 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:14.741 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-04-27T01:54:14.741 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:14.742 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-04-27T01:54:14.742 INFO:teuthology.orchestra.run.smithi052.stdout: ceph x86_64 2:18.2.0-0.el9 ceph 6.4 k 2024-04-27T01:54:14.742 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-04-27T01:54:14.742 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds x86_64 2:18.2.0-0.el9 ceph 2.1 M 2024-04-27T01:54:14.742 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr x86_64 2:18.2.0-0.el9 ceph 1.4 M 2024-04-27T01:54:14.742 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core noarch 2:18.2.0-0.el9 ceph-noarch 240 k 2024-04-27T01:54:14.742 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon x86_64 2:18.2.0-0.el9 ceph 4.4 M 2024-04-27T01:54:14.743 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd x86_64 2:18.2.0-0.el9 ceph 18 M 2024-04-27T01:54:14.743 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-27T01:54:14.743 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-27T01:54:14.743 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-27T01:54:14.743 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-27T01:54:14.743 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-27T01:54:14.743 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-27T01:54:14.743 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-27T01:54:14.743 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-27T01:54:14.743 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-27T01:54:14.743 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-27T01:54:14.743 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-27T01:54:14.743 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-27T01:54:14.744 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-27T01:54:14.744 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-27T01:54:14.744 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-27T01:54:14.744 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-27T01:54:14.744 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-27T01:54:14.744 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-27T01:54:14.744 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-27T01:54:14.744 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-27T01:54:14.744 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-27T01:54:14.744 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-27T01:54:14.744 INFO:teuthology.orchestra.run.smithi052.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-27T01:54:14.744 INFO:teuthology.orchestra.run.smithi052.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-27T01:54:14.744 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-27T01:54:14.744 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-27T01:54:14.745 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-27T01:54:14.745 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:14.745 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-04-27T01:54:14.745 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:14.745 INFO:teuthology.orchestra.run.smithi052.stdout:Install 33 Packages 2024-04-27T01:54:14.745 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:14.745 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 30 M 2024-04-27T01:54:14.746 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 109 M 2024-04-27T01:54:14.746 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-04-27T01:54:15.278 INFO:teuthology.orchestra.run.smithi052.stdout:(1/33): ceph-18.2.0-0.el9.x86_64.rpm 55 kB/s | 6.4 kB 00:00 2024-04-27T01:54:15.337 INFO:teuthology.orchestra.run.smithi052.stdout:(2/33): ceph-mgr-18.2.0-0.el9.x86_64.rpm 8.2 MB/s | 1.4 MB 00:00 2024-04-27T01:54:15.348 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-27T01:54:15.348 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:15.348 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2024-04-27T01:54:15.348 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-2:18.2.0-0.el9.x86_64 librbd1-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:15.348 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-27T01:54:15.348 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-27T01:54:15.348 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:15.348 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:15.348 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:15.348 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:15.349 INFO:teuthology.orchestra.run.smithi029.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-27T01:54:15.349 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-27T01:54:15.349 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-27T01:54:15.349 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-27T01:54:15.349 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:15.349 INFO:teuthology.orchestra.run.smithi029.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-27T01:54:15.349 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-27T01:54:15.349 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-27T01:54:15.349 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:15.349 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-27T01:54:15.349 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:15.349 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-27T01:54:15.349 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-27T01:54:15.349 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-27T01:54:15.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:15.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:15.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:15.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-27T01:54:15.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:15.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:15.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:15.350 INFO:teuthology.orchestra.run.smithi029.stdout: re2-1:20211101-3.el9.x86_64 2024-04-27T01:54:15.350 INFO:teuthology.orchestra.run.smithi029.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-27T01:54:15.350 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:15.350 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:54:15.379 INFO:teuthology.orchestra.run.smithi052.stdout:(3/33): ceph-mds-18.2.0-0.el9.x86_64.rpm 9.6 MB/s | 2.1 MB 00:00 2024-04-27T01:54:15.404 INFO:teuthology.orchestra.run.smithi052.stdout:(4/33): ceph-mgr-modules-core-18.2.0-0.el9.noar 9.3 MB/s | 240 kB 00:00 2024-04-27T01:54:15.488 INFO:teuthology.orchestra.run.smithi052.stdout:(5/33): ceph-mon-18.2.0-0.el9.x86_64.rpm 21 MB/s | 4.4 MB 00:00 2024-04-27T01:54:15.678 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-test 2024-04-27T01:54:15.730 INFO:teuthology.orchestra.run.smithi052.stdout:(6/33): ceph-osd-18.2.0-0.el9.x86_64.rpm 45 MB/s | 18 MB 00:00 2024-04-27T01:54:16.072 INFO:teuthology.orchestra.run.smithi052.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 380 kB/s | 253 kB 00:00 2024-04-27T01:54:16.147 INFO:teuthology.orchestra.run.smithi052.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 256 kB/s | 106 kB 00:00 2024-04-27T01:54:16.214 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:34 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:54:16.336 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:54:16.337 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:54:16.337 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-04-27T01:54:16.337 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:54:16.337 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-27T01:54:16.337 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test x86_64 2:18.2.0-0.el9 ceph 40 M 2024-04-27T01:54:16.337 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-27T01:54:16.337 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite x86_64 2:18.2.0-0.el9 ceph 161 k 2024-04-27T01:54:16.337 INFO:teuthology.orchestra.run.smithi029.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-27T01:54:16.337 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-27T01:54:16.338 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:16.338 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-27T01:54:16.338 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:54:16.338 INFO:teuthology.orchestra.run.smithi029.stdout:Install 4 Packages 2024-04-27T01:54:16.338 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:16.338 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 40 M 2024-04-27T01:54:16.339 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 165 M 2024-04-27T01:54:16.339 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-27T01:54:16.572 INFO:teuthology.orchestra.run.smithi052.stdout:(9/33): python3-requests-2.25.1-8.el9.noarch.rp 294 kB/s | 125 kB 00:00 2024-04-27T01:54:16.680 INFO:teuthology.orchestra.run.smithi052.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 222 kB/s | 135 kB 00:00 2024-04-27T01:54:16.701 INFO:teuthology.orchestra.run.smithi029.stdout:(1/4): libcephsqlite-18.2.0-0.el9.x86_64.rpm 1.6 MB/s | 161 kB 00:00 2024-04-27T01:54:16.747 INFO:teuthology.orchestra.run.smithi052.stdout:(11/33): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-04-27T01:54:16.760 INFO:teuthology.orchestra.run.smithi029.stdout:(2/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.9 MB/s | 305 kB 00:00 2024-04-27T01:54:16.794 INFO:teuthology.orchestra.run.smithi029.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 685 kB/s | 64 kB 00:00 2024-04-27T01:54:16.798 INFO:teuthology.orchestra.run.smithi052.stdout:(12/33): python3-markupsafe-1.1.1-12.el9.x86_64 695 kB/s | 35 kB 00:00 2024-04-27T01:54:16.856 INFO:teuthology.orchestra.run.smithi052.stdout:(13/33): python3-pytz-2021.1-5.el9.noarch.rpm 875 kB/s | 51 kB 00:00 2024-04-27T01:54:16.915 INFO:teuthology.orchestra.run.smithi052.stdout:(14/33): python3-toml-0.10.2-6.el9.noarch.rpm 714 kB/s | 42 kB 00:00 2024-04-27T01:54:16.973 INFO:teuthology.orchestra.run.smithi052.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 747 kB/s | 43 kB 00:00 2024-04-27T01:54:17.040 INFO:teuthology.orchestra.run.smithi052.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-04-27T01:54:17.091 INFO:teuthology.orchestra.run.smithi052.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 7.0 MB/s | 358 kB 00:00 2024-04-27T01:54:17.124 INFO:teuthology.orchestra.run.smithi052.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 323 kB/s | 11 kB 00:00 2024-04-27T01:54:17.158 INFO:teuthology.orchestra.run.smithi052.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 530 kB/s | 18 kB 00:00 2024-04-27T01:54:17.183 INFO:teuthology.orchestra.run.smithi052.stdout:(20/33): python3-urllib3-1.26.5-5.el9.noarch.rp 351 kB/s | 215 kB 00:00 2024-04-27T01:54:17.208 INFO:teuthology.orchestra.run.smithi052.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-04-27T01:54:17.242 INFO:teuthology.orchestra.run.smithi052.stdout:(22/33): python3-jaraco-text-3.2.0-6.el9.noarch 589 kB/s | 20 kB 00:00 2024-04-27T01:54:17.268 INFO:teuthology.orchestra.run.smithi052.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 231 kB/s | 19 kB 00:00 2024-04-27T01:54:17.293 INFO:teuthology.orchestra.run.smithi052.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 919 kB/s | 46 kB 00:00 2024-04-27T01:54:17.318 INFO:teuthology.orchestra.run.smithi052.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-04-27T01:54:17.344 INFO:teuthology.orchestra.run.smithi052.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-27T01:54:17.360 INFO:teuthology.orchestra.run.smithi052.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 391 kB/s | 16 kB 00:00 2024-04-27T01:54:17.386 INFO:teuthology.orchestra.run.smithi052.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-04-27T01:54:17.402 INFO:teuthology.orchestra.run.smithi052.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 857 kB/s | 36 kB 00:00 2024-04-27T01:54:17.428 INFO:teuthology.orchestra.run.smithi052.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 5.4 MB/s | 230 kB 00:00 2024-04-27T01:54:17.461 INFO:teuthology.orchestra.run.smithi052.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.1 MB/s | 427 kB 00:00 2024-04-27T01:54:17.487 INFO:teuthology.orchestra.run.smithi052.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2024-04-27T01:54:17.520 INFO:teuthology.orchestra.run.smithi052.stdout:(33/33): python3-cryptography-36.0.1-4.el9.x86_ 618 kB/s | 1.2 MB 00:02 2024-04-27T01:54:17.524 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:54:17.524 INFO:teuthology.orchestra.run.smithi052.stdout:Total 11 MB/s | 30 MB 00:02 2024-04-27T01:54:17.566 INFO:teuthology.orchestra.run.smithi029.stdout:(4/4): ceph-test-18.2.0-0.el9.x86_64.rpm 41 MB/s | 40 MB 00:00 2024-04-27T01:54:17.569 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:54:17.569 INFO:teuthology.orchestra.run.smithi029.stdout:Total 33 MB/s | 40 MB 00:01 2024-04-27T01:54:17.596 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-27T01:54:17.618 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-27T01:54:17.618 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-27T01:54:17.644 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-04-27T01:54:17.685 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-04-27T01:54:17.685 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-04-27T01:54:17.951 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-27T01:54:17.952 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-27T01:54:17.985 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-04-27T01:54:17.986 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-04-27T01:54:18.463 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-27T01:54:18.595 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-27T01:54:18.670 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-27T01:54:18.722 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephsqlite-2:18.2.0-0.el9.x86_64 3/4 2024-04-27T01:54:19.285 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-04-27T01:54:19.348 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-27T01:54:19.381 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-27T01:54:19.414 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-27T01:54:19.513 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-27T01:54:19.613 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-27T01:54:19.668 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-27T01:54:19.703 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-27T01:54:19.733 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-04-27T01:54:19.768 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-04-27T01:54:19.800 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-04-27T01:54:19.841 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-04-27T01:54:19.885 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-04-27T01:54:19.918 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-04-27T01:54:19.957 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-04-27T01:54:20.034 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-04-27T01:54:20.185 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-04-27T01:54:20.267 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-04-27T01:54:20.336 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-04-27T01:54:20.374 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-04-27T01:54:20.432 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-04-27T01:54:20.496 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-04-27T01:54:20.702 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-04-27T01:54:20.752 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-04-27T01:54:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-04-27T01:54:20.924 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-04-27T01:54:20.960 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-04-27T01:54:21.090 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-04-27T01:54:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 28/33 2024-04-27T01:54:21.278 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-2:18.2.0-0.el9.x86_64 29/33 2024-04-27T01:54:21.305 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-2:18.2.0-0.el9.x86_64 29/33 2024-04-27T01:54:21.305 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:54:21.305 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-27T01:54:21.305 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-27T01:54:21.305 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-27T01:54:21.305 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:22.650 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephsqlite-2:18.2.0-0.el9.x86_64 3/4 2024-04-27T01:54:22.904 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-osd-2:18.2.0-0.el9.x86_64 30/33 2024-04-27T01:54:22.932 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-osd-2:18.2.0-0.el9.x86_64 30/33 2024-04-27T01:54:22.932 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:54:22.932 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-27T01:54:22.932 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-27T01:54:22.932 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-27T01:54:22.932 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:23.015 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-2:18.2.0-0.el9.x86_64 4/4 2024-04-27T01:54:23.397 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mon-2:18.2.0-0.el9.x86_64 31/33 2024-04-27T01:54:23.421 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mon-2:18.2.0-0.el9.x86_64 31/33 2024-04-27T01:54:23.421 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:54:23.421 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-27T01:54:23.421 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-27T01:54:23.421 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-27T01:54:23.421 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:23.639 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mds-2:18.2.0-0.el9.x86_64 32/33 2024-04-27T01:54:23.663 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mds-2:18.2.0-0.el9.x86_64 32/33 2024-04-27T01:54:23.664 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:54:23.664 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-27T01:54:23.664 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-27T01:54:23.664 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-27T01:54:23.664 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:23.728 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-2:18.2.0-0.el9.x86_64 33/33 2024-04-27T01:54:25.484 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-2:18.2.0-0.el9.x86_64 33/33 2024-04-27T01:54:25.484 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-2:18.2.0-0.el9.x86_64 1/33 2024-04-27T01:54:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mds-2:18.2.0-0.el9.x86_64 2/33 2024-04-27T01:54:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-2:18.2.0-0.el9.x86_64 3/33 2024-04-27T01:54:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mon-2:18.2.0-0.el9.x86_64 4/33 2024-04-27T01:54:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-osd-2:18.2.0-0.el9.x86_64 5/33 2024-04-27T01:54:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 6/33 2024-04-27T01:54:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-04-27T01:54:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-04-27T01:54:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-04-27T01:54:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-04-27T01:54:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-04-27T01:54:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-04-27T01:54:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-04-27T01:54:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-04-27T01:54:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-04-27T01:54:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-27T01:54:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-27T01:54:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-27T01:54:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-27T01:54:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-27T01:54:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-27T01:54:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-27T01:54:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-27T01:54:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-27T01:54:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-27T01:54:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-27T01:54:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-27T01:54:25.487 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-27T01:54:25.487 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-27T01:54:25.487 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-27T01:54:25.487 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-27T01:54:25.487 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-27T01:54:26.165 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-test-2:18.2.0-0.el9.x86_64 4/4 2024-04-27T01:54:26.165 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-2:18.2.0-0.el9.x86_64 1/4 2024-04-27T01:54:26.165 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephsqlite-2:18.2.0-0.el9.x86_64 2/4 2024-04-27T01:54:26.165 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-27T01:54:26.409 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-27T01:54:26.409 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:26.409 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-27T01:54:26.409 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-2:18.2.0-0.el9.x86_64 libcephsqlite-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:26.409 INFO:teuthology.orchestra.run.smithi029.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-04-27T01:54:26.409 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:26.409 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:54:26.449 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-27T01:54:26.449 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:26.449 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-04-27T01:54:26.449 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:26.449 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:26.449 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:26.450 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 2024-04-27T01:54:26.450 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:26.450 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:26.450 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-27T01:54:26.450 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-27T01:54:26.450 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-27T01:54:26.450 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-27T01:54:26.450 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-27T01:54:26.450 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-27T01:54:26.450 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-27T01:54:26.450 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-27T01:54:26.450 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-27T01:54:26.450 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-27T01:54:26.450 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-27T01:54:26.451 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-27T01:54:26.451 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-27T01:54:26.451 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-27T01:54:26.451 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-27T01:54:26.451 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ply-3.11-14.el9.noarch 2024-04-27T01:54:26.451 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-27T01:54:26.451 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-27T01:54:26.451 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-27T01:54:26.451 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-27T01:54:26.451 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-27T01:54:26.451 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-27T01:54:26.451 INFO:teuthology.orchestra.run.smithi052.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-27T01:54:26.451 INFO:teuthology.orchestra.run.smithi052.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-27T01:54:26.451 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-27T01:54:26.451 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-27T01:54:26.451 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-27T01:54:26.452 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:26.452 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:54:26.730 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph 2024-04-27T01:54:26.877 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-base 2024-04-27T01:54:27.260 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:45 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:54:27.388 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:54:27.390 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:54:27.391 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-27T01:54:27.391 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:54:27.391 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-27T01:54:27.391 INFO:teuthology.orchestra.run.smithi029.stdout: ceph x86_64 2:18.2.0-0.el9 ceph 6.4 k 2024-04-27T01:54:27.391 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-27T01:54:27.391 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds x86_64 2:18.2.0-0.el9 ceph 2.1 M 2024-04-27T01:54:27.391 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr x86_64 2:18.2.0-0.el9 ceph 1.4 M 2024-04-27T01:54:27.391 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core noarch 2:18.2.0-0.el9 ceph-noarch 240 k 2024-04-27T01:54:27.391 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon x86_64 2:18.2.0-0.el9 ceph 4.4 M 2024-04-27T01:54:27.391 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd x86_64 2:18.2.0-0.el9 ceph 18 M 2024-04-27T01:54:27.391 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-27T01:54:27.391 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-27T01:54:27.391 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-27T01:54:27.392 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-27T01:54:27.392 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-27T01:54:27.392 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-27T01:54:27.392 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-27T01:54:27.392 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-27T01:54:27.392 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-27T01:54:27.392 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-27T01:54:27.392 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-27T01:54:27.392 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-27T01:54:27.392 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-27T01:54:27.392 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-27T01:54:27.392 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-27T01:54:27.392 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-27T01:54:27.392 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-27T01:54:27.393 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-27T01:54:27.393 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-27T01:54:27.393 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-27T01:54:27.393 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-27T01:54:27.393 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-27T01:54:27.393 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-27T01:54:27.393 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-27T01:54:27.393 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-27T01:54:27.393 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-27T01:54:27.393 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-27T01:54:27.393 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:27.393 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-27T01:54:27.394 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:54:27.394 INFO:teuthology.orchestra.run.smithi029.stdout:Install 33 Packages 2024-04-27T01:54:27.394 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:27.394 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 30 M 2024-04-27T01:54:27.394 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 109 M 2024-04-27T01:54:27.395 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-27T01:54:27.415 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:48 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:54:27.496 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-base-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-27T01:54:27.539 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:54:27.540 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-04-27T01:54:27.540 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:54:27.604 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install cephadm 2024-04-27T01:54:28.128 INFO:teuthology.orchestra.run.smithi029.stdout:(1/33): ceph-mds-18.2.0-0.el9.x86_64.rpm 14 MB/s | 2.1 MB 00:00 2024-04-27T01:54:28.136 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:49 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:54:28.153 INFO:teuthology.orchestra.run.smithi029.stdout:(2/33): ceph-18.2.0-0.el9.x86_64.rpm 36 kB/s | 6.4 kB 00:00 2024-04-27T01:54:28.204 INFO:teuthology.orchestra.run.smithi029.stdout:(3/33): ceph-mgr-18.2.0-0.el9.x86_64.rpm 6.3 MB/s | 1.4 MB 00:00 2024-04-27T01:54:28.238 INFO:teuthology.orchestra.run.smithi029.stdout:(4/33): ceph-mgr-modules-core-18.2.0-0.el9.noar 7.0 MB/s | 240 kB 00:00 2024-04-27T01:54:28.261 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:54:28.262 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:28.262 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-04-27T01:54:28.262 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:28.262 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-04-27T01:54:28.262 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm noarch 2:18.2.0-0.el9 ceph-noarch 209 k 2024-04-27T01:54:28.262 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:28.262 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-04-27T01:54:28.262 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:28.262 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-04-27T01:54:28.263 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:28.263 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 209 k 2024-04-27T01:54:28.263 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 200 k 2024-04-27T01:54:28.263 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-04-27T01:54:28.338 INFO:teuthology.orchestra.run.smithi052.stdout:cephadm-18.2.0-0.el9.noarch.rpm 2.7 MB/s | 209 kB 00:00 2024-04-27T01:54:28.339 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:54:28.339 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.7 MB/s | 209 kB 00:00 2024-04-27T01:54:28.339 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-04-27T01:54:28.344 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-04-27T01:54:28.344 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-04-27T01:54:28.347 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-04-27T01:54:28.347 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-04-27T01:54:28.372 INFO:teuthology.orchestra.run.smithi029.stdout:(5/33): ceph-mon-18.2.0-0.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-04-27T01:54:28.417 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-04-27T01:54:28.723 INFO:teuthology.orchestra.run.smithi029.stdout:(6/33): ceph-osd-18.2.0-0.el9.x86_64.rpm 31 MB/s | 18 MB 00:00 2024-04-27T01:54:28.790 INFO:teuthology.orchestra.run.smithi029.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 459 kB/s | 253 kB 00:00 2024-04-27T01:54:28.840 INFO:teuthology.orchestra.run.smithi029.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-04-27T01:54:28.882 INFO:teuthology.orchestra.run.smithi029.stdout:(9/33): python3-cryptography-36.0.1-4.el9.x86_6 2.4 MB/s | 1.2 MB 00:00 2024-04-27T01:54:28.918 INFO:teuthology.orchestra.run.smithi029.stdout:(10/33): python3-ply-3.11-14.el9.noarch.rpm 547 kB/s | 106 kB 00:00 2024-04-27T01:54:28.951 INFO:teuthology.orchestra.run.smithi029.stdout:(11/33): python3-requests-2.25.1-8.el9.noarch.r 1.1 MB/s | 125 kB 00:00 2024-04-27T01:54:28.985 INFO:teuthology.orchestra.run.smithi029.stdout:(12/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.0 MB/s | 215 kB 00:00 2024-04-27T01:54:28.991 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-27T01:54:29.084 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-27T01:54:29.186 INFO:teuthology.orchestra.run.smithi029.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 148 kB/s | 35 kB 00:00 2024-04-27T01:54:29.211 INFO:teuthology.orchestra.run.smithi029.stdout:(14/33): python3-mako-1.1.4-6.el9.noarch.rpm 587 kB/s | 172 kB 00:00 2024-04-27T01:54:29.237 INFO:teuthology.orchestra.run.smithi029.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 203 kB/s | 51 kB 00:00 2024-04-27T01:54:29.262 INFO:teuthology.orchestra.run.smithi029.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 548 kB/s | 42 kB 00:00 2024-04-27T01:54:29.447 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-27T01:54:29.646 INFO:teuthology.orchestra.run.smithi029.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 100 kB/s | 43 kB 00:00 2024-04-27T01:54:29.706 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-27T01:54:29.706 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:29.707 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-04-27T01:54:29.707 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm-2:18.2.0-0.el9.noarch 2024-04-27T01:54:29.707 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:29.707 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:54:29.797 INFO:teuthology.orchestra.run.smithi029.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 71 kB/s | 11 kB 00:00 2024-04-27T01:54:29.831 INFO:teuthology.orchestra.run.smithi029.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 289 kB/s | 172 kB 00:00 2024-04-27T01:54:29.876 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-immutable-object-cache 2024-04-27T01:54:29.923 INFO:teuthology.orchestra.run.smithi029.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 140 kB/s | 18 kB 00:00 2024-04-27T01:54:29.948 INFO:teuthology.orchestra.run.smithi029.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 197 kB/s | 23 kB 00:00 2024-04-27T01:54:29.982 INFO:teuthology.orchestra.run.smithi029.stdout:(22/33): python3-cherrypy-18.6.1-2.el9.noarch.r 498 kB/s | 358 kB 00:00 2024-04-27T01:54:30.058 INFO:teuthology.orchestra.run.smithi029.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 145 kB/s | 19 kB 00:00 2024-04-27T01:54:30.091 INFO:teuthology.orchestra.run.smithi029.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 138 kB/s | 20 kB 00:00 2024-04-27T01:54:30.125 INFO:teuthology.orchestra.run.smithi029.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 324 kB/s | 46 kB 00:00 2024-04-27T01:54:30.236 INFO:teuthology.orchestra.run.smithi029.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 149 kB/s | 16 kB 00:00 2024-04-27T01:54:30.286 INFO:teuthology.orchestra.run.smithi029.stdout:(27/33): python3-more-itertools-8.12.0-2.el9.no 345 kB/s | 79 kB 00:00 2024-04-27T01:54:30.329 INFO:teuthology.orchestra.run.smithi029.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.1 MB/s | 272 kB 00:00 2024-04-27T01:54:30.354 INFO:teuthology.orchestra.run.smithi029.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 764 kB/s | 90 kB 00:00 2024-04-27T01:54:30.416 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:51 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:54:30.421 INFO:teuthology.orchestra.run.smithi029.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 266 kB/s | 36 kB 00:00 2024-04-27T01:54:30.488 INFO:teuthology.orchestra.run.smithi029.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.4 MB/s | 230 kB 00:00 2024-04-27T01:54:30.539 INFO:teuthology.orchestra.run.smithi029.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 170 kB/s | 20 kB 00:00 2024-04-27T01:54:30.540 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:54:30.541 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:30.541 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-04-27T01:54:30.541 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:30.541 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-04-27T01:54:30.541 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-0.el9 ceph 142 k 2024-04-27T01:54:30.541 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:30.541 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-04-27T01:54:30.541 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:30.541 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-04-27T01:54:30.541 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:30.542 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 142 k 2024-04-27T01:54:30.542 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 399 k 2024-04-27T01:54:30.542 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-04-27T01:54:30.572 INFO:teuthology.orchestra.run.smithi029.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.9 MB/s | 427 kB 00:00 2024-04-27T01:54:30.579 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:54:30.579 INFO:teuthology.orchestra.run.smithi029.stdout:Total 9.4 MB/s | 30 MB 00:03 2024-04-27T01:54:30.610 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-immutable-object-cache-18.2.0-0.el9.x86_64 2.0 MB/s | 142 kB 00:00 2024-04-27T01:54:30.611 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:54:30.611 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.0 MB/s | 142 kB 00:00 2024-04-27T01:54:30.611 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-04-27T01:54:30.619 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-04-27T01:54:30.619 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-04-27T01:54:30.679 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-04-27T01:54:30.680 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-04-27T01:54:30.700 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-27T01:54:30.742 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-27T01:54:30.743 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-27T01:54:30.828 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-04-27T01:54:31.031 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:54:31.045 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-27T01:54:31.045 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-27T01:54:31.057 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:54:31.057 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:54:31.057 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-27T01:54:31.057 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:32.033 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:54:32.033 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:32.034 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-04-27T01:54:32.034 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:32.034 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:32.034 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:54:32.230 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr 2024-04-27T01:54:32.260 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-27T01:54:32.326 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-27T01:54:32.359 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-27T01:54:32.401 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-27T01:54:32.507 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-27T01:54:32.595 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-27T01:54:32.641 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-27T01:54:32.685 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-27T01:54:32.723 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-04-27T01:54:32.758 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-04-27T01:54:32.766 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:53 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:54:32.792 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-04-27T01:54:32.843 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-04-27T01:54:32.849 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-mgr-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-27T01:54:32.888 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-04-27T01:54:32.892 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:54:32.893 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-04-27T01:54:32.894 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:54:32.922 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-04-27T01:54:32.960 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-dashboard 2024-04-27T01:54:32.961 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-04-27T01:54:33.048 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-04-27T01:54:33.208 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-04-27T01:54:33.294 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-04-27T01:54:33.354 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-04-27T01:54:33.401 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-04-27T01:54:33.460 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-04-27T01:54:33.487 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:54 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:54:33.517 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-04-27T01:54:33.611 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:54:33.612 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:33.612 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-04-27T01:54:33.612 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:33.612 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-04-27T01:54:33.612 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard noarch 2:18.2.0-0.el9 ceph-noarch 1.7 M 2024-04-27T01:54:33.612 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-04-27T01:54:33.612 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards noarch 2:18.2.0-0.el9 ceph-noarch 23 k 2024-04-27T01:54:33.612 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts noarch 2:18.2.0-0.el9 ceph-noarch 15 k 2024-04-27T01:54:33.613 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-27T01:54:33.613 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-27T01:54:33.613 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-27T01:54:33.613 INFO:teuthology.orchestra.run.smithi052.stdout:Installing weak dependencies: 2024-04-27T01:54:33.613 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-27T01:54:33.613 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:33.613 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-04-27T01:54:33.613 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:33.613 INFO:teuthology.orchestra.run.smithi052.stdout:Install 7 Packages 2024-04-27T01:54:33.613 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:33.614 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 2.0 M 2024-04-27T01:54:33.614 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 8.1 M 2024-04-27T01:54:33.614 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-04-27T01:54:33.722 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-04-27T01:54:33.785 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-04-27T01:54:33.867 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-04-27T01:54:33.890 INFO:teuthology.orchestra.run.smithi052.stdout:(1/7): ceph-grafana-dashboards-18.2.0-0.el9.noa 202 kB/s | 23 kB 00:00 2024-04-27T01:54:33.916 INFO:teuthology.orchestra.run.smithi052.stdout:(2/7): ceph-prometheus-alerts-18.2.0-0.el9.noar 105 kB/s | 15 kB 00:00 2024-04-27T01:54:33.964 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-04-27T01:54:33.966 INFO:teuthology.orchestra.run.smithi052.stdout:(3/7): ceph-mgr-dashboard-18.2.0-0.el9.noarch.r 8.9 MB/s | 1.7 MB 00:00 2024-04-27T01:54:34.012 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-04-27T01:54:34.116 INFO:teuthology.orchestra.run.smithi052.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 40 kB/s | 9.0 kB 00:00 2024-04-27T01:54:34.146 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-04-27T01:54:34.233 INFO:teuthology.orchestra.run.smithi052.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 129 kB/s | 41 kB 00:00 2024-04-27T01:54:34.258 INFO:teuthology.orchestra.run.smithi052.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 128 kB/s | 31 kB 00:00 2024-04-27T01:54:34.315 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 28/33 2024-04-27T01:54:34.333 INFO:teuthology.orchestra.run.smithi052.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 868 kB/s | 188 kB 00:00 2024-04-27T01:54:34.337 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:54:34.337 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2024-04-27T01:54:34.337 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-04-27T01:54:34.341 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-2:18.2.0-0.el9.x86_64 29/33 2024-04-27T01:54:34.346 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-04-27T01:54:34.347 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-04-27T01:54:34.368 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-2:18.2.0-0.el9.x86_64 29/33 2024-04-27T01:54:34.368 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:54:34.368 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-27T01:54:34.368 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-27T01:54:34.368 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-27T01:54:34.368 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:34.402 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-04-27T01:54:34.402 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-04-27T01:54:34.694 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-04-27T01:54:34.768 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-27T01:54:34.803 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-27T01:54:34.859 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-27T01:54:34.897 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-27T01:54:34.926 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 5/7 2024-04-27T01:54:35.168 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 6/7 2024-04-27T01:54:35.196 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 7/7 2024-04-27T01:54:35.703 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 7/7 2024-04-27T01:54:35.704 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 1/7 2024-04-27T01:54:35.704 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 2/7 2024-04-27T01:54:35.704 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 3/7 2024-04-27T01:54:35.704 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-27T01:54:35.704 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-27T01:54:35.705 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-27T01:54:35.958 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-27T01:54:35.958 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:35.958 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-04-27T01:54:35.958 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 2024-04-27T01:54:35.958 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 2024-04-27T01:54:35.958 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 2024-04-27T01:54:35.958 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-27T01:54:35.959 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-27T01:54:35.959 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-27T01:54:35.959 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-27T01:54:35.959 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:35.959 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:54:35.969 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-2:18.2.0-0.el9.x86_64 30/33 2024-04-27T01:54:35.997 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-2:18.2.0-0.el9.x86_64 30/33 2024-04-27T01:54:35.997 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:54:35.997 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-27T01:54:35.997 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-27T01:54:35.997 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-27T01:54:35.997 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:36.143 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-27T01:54:36.471 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-2:18.2.0-0.el9.x86_64 31/33 2024-04-27T01:54:36.495 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-2:18.2.0-0.el9.x86_64 31/33 2024-04-27T01:54:36.495 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:54:36.495 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-27T01:54:36.495 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-27T01:54:36.495 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-27T01:54:36.495 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:36.673 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:57 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:54:36.705 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-2:18.2.0-0.el9.x86_64 32/33 2024-04-27T01:54:36.730 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-2:18.2.0-0.el9.x86_64 32/33 2024-04-27T01:54:36.730 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:54:36.730 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-27T01:54:36.730 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-27T01:54:36.730 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-27T01:54:36.730 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:36.796 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-2:18.2.0-0.el9.x86_64 33/33 2024-04-27T01:54:36.799 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:54:36.800 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:36.800 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-04-27T01:54:36.800 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:36.800 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-04-27T01:54:36.800 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-0.el9 ceph-noarch 7.4 M 2024-04-27T01:54:36.800 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-04-27T01:54:36.800 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-27T01:54:36.800 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-27T01:54:36.801 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-27T01:54:36.801 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-27T01:54:36.801 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-27T01:54:36.801 INFO:teuthology.orchestra.run.smithi052.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-27T01:54:36.801 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-27T01:54:36.801 INFO:teuthology.orchestra.run.smithi052.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-27T01:54:36.801 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-27T01:54:36.801 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-27T01:54:36.801 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-27T01:54:36.801 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:36.801 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-04-27T01:54:36.801 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:36.801 INFO:teuthology.orchestra.run.smithi052.stdout:Install 12 Packages 2024-04-27T01:54:36.801 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:36.802 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 38 M 2024-04-27T01:54:36.802 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 209 M 2024-04-27T01:54:36.802 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-04-27T01:54:37.309 INFO:teuthology.orchestra.run.smithi052.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.0-0. 45 MB/s | 7.4 MB 00:00 2024-04-27T01:54:37.357 INFO:teuthology.orchestra.run.smithi052.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 659 kB/s | 33 kB 00:00 2024-04-27T01:54:37.423 INFO:teuthology.orchestra.run.smithi052.stdout:(3/12): libquadmath-11.4.1-3.el9.x86_64.rpm 695 kB/s | 195 kB 00:00 2024-04-27T01:54:37.475 INFO:teuthology.orchestra.run.smithi052.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-04-27T01:54:37.558 INFO:teuthology.orchestra.run.smithi052.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 15 MB/s | 3.0 MB 00:00 2024-04-27T01:54:37.583 INFO:teuthology.orchestra.run.smithi052.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 324 kB/s | 35 kB 00:00 2024-04-27T01:54:37.616 INFO:teuthology.orchestra.run.smithi052.stdout:(7/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-04-27T01:54:37.684 INFO:teuthology.orchestra.run.smithi052.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 2.4 MB/s | 244 kB 00:00 2024-04-27T01:54:37.767 INFO:teuthology.orchestra.run.smithi052.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-04-27T01:54:37.843 INFO:teuthology.orchestra.run.smithi052.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 17 MB/s | 4.7 MB 00:00 2024-04-27T01:54:37.968 INFO:teuthology.orchestra.run.smithi052.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 5.1 MB 00:00 2024-04-27T01:54:38.376 INFO:teuthology.orchestra.run.smithi052.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-04-27T01:54:38.377 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:54:38.377 INFO:teuthology.orchestra.run.smithi052.stdout:Total 24 MB/s | 38 MB 00:01 2024-04-27T01:54:38.736 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-04-27T01:54:38.764 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-04-27T01:54:38.764 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-04-27T01:54:39.196 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-04-27T01:54:39.196 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-04-27T01:54:39.345 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-2:18.2.0-0.el9.x86_64 33/33 2024-04-27T01:54:39.345 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-2:18.2.0-0.el9.x86_64 1/33 2024-04-27T01:54:39.345 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-2:18.2.0-0.el9.x86_64 2/33 2024-04-27T01:54:39.345 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-2:18.2.0-0.el9.x86_64 3/33 2024-04-27T01:54:39.345 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-2:18.2.0-0.el9.x86_64 4/33 2024-04-27T01:54:39.345 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-2:18.2.0-0.el9.x86_64 5/33 2024-04-27T01:54:39.345 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 6/33 2024-04-27T01:54:39.345 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-04-27T01:54:39.346 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-04-27T01:54:39.346 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-04-27T01:54:39.346 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-04-27T01:54:39.346 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-04-27T01:54:39.346 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-04-27T01:54:39.346 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-04-27T01:54:39.346 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-04-27T01:54:39.346 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-04-27T01:54:39.346 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-27T01:54:39.346 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-27T01:54:39.346 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-27T01:54:39.346 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-27T01:54:39.346 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-27T01:54:39.346 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-27T01:54:39.347 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-27T01:54:39.347 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-27T01:54:39.347 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-27T01:54:39.347 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-27T01:54:39.347 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-27T01:54:39.347 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-27T01:54:39.347 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-27T01:54:39.347 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-27T01:54:39.347 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-27T01:54:39.347 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-27T01:54:39.347 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-27T01:54:39.986 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-04-27T01:54:40.056 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-27T01:54:40.100 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-27T01:54:40.206 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-27T01:54:40.243 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-27T01:54:40.296 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-27T01:54:40.297 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:40.297 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-27T01:54:40.297 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:40.297 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:40.297 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:40.297 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 2024-04-27T01:54:40.297 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:40.297 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:40.297 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-27T01:54:40.297 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-27T01:54:40.297 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-27T01:54:40.297 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-27T01:54:40.297 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-27T01:54:40.298 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-27T01:54:40.298 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-27T01:54:40.298 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-27T01:54:40.298 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-27T01:54:40.298 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-27T01:54:40.298 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-27T01:54:40.298 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-27T01:54:40.298 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-27T01:54:40.298 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-27T01:54:40.298 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-27T01:54:40.298 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply-3.11-14.el9.noarch 2024-04-27T01:54:40.299 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-27T01:54:40.299 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-27T01:54:40.299 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-27T01:54:40.299 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-27T01:54:40.299 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-27T01:54:40.299 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-27T01:54:40.299 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-27T01:54:40.299 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-27T01:54:40.299 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-27T01:54:40.299 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-27T01:54:40.299 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-27T01:54:40.299 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:40.299 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:54:40.418 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-27T01:54:40.449 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-27T01:54:40.528 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-27T01:54:40.686 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-base 2024-04-27T01:54:41.218 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-27T01:54:41.226 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:59 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:54:41.308 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-base-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-27T01:54:41.350 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-27T01:54:41.409 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:54:41.409 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-27T01:54:41.409 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:54:41.431 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install cephadm 2024-04-27T01:54:41.968 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:59 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:54:42.094 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:54:42.095 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:54:42.095 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-04-27T01:54:42.095 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:54:42.095 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-27T01:54:42.096 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm noarch 2:18.2.0-0.el9 ceph-noarch 209 k 2024-04-27T01:54:42.096 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:42.096 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-27T01:54:42.096 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:54:42.096 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-04-27T01:54:42.096 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:42.096 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 209 k 2024-04-27T01:54:42.096 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 200 k 2024-04-27T01:54:42.096 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-27T01:54:42.167 INFO:teuthology.orchestra.run.smithi029.stdout:cephadm-18.2.0-0.el9.noarch.rpm 2.9 MB/s | 209 kB 00:00 2024-04-27T01:54:42.168 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:54:42.168 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.8 MB/s | 209 kB 00:00 2024-04-27T01:54:42.168 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-27T01:54:42.173 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-27T01:54:42.174 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-27T01:54:42.176 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-27T01:54:42.176 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-27T01:54:42.247 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-27T01:54:42.650 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-27T01:54:42.716 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-27T01:54:42.811 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-27T01:54:43.191 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-27T01:54:43.442 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-27T01:54:43.442 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:43.442 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-27T01:54:43.442 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm-2:18.2.0-0.el9.noarch 2024-04-27T01:54:43.443 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:43.443 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:54:43.689 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-immutable-object-cache 2024-04-27T01:54:43.727 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-27T01:54:43.749 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 12/12 2024-04-27T01:54:44.219 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:02 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:54:44.344 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:54:44.345 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:54:44.345 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-04-27T01:54:44.345 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:54:44.345 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-27T01:54:44.345 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-0.el9 ceph 142 k 2024-04-27T01:54:44.345 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:44.345 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-27T01:54:44.345 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:54:44.345 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-04-27T01:54:44.345 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:44.345 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 142 k 2024-04-27T01:54:44.346 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 399 k 2024-04-27T01:54:44.346 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-27T01:54:44.423 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-immutable-object-cache-18.2.0-0.el9.x86_64 1.8 MB/s | 142 kB 00:00 2024-04-27T01:54:44.424 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:54:44.424 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.8 MB/s | 142 kB 00:00 2024-04-27T01:54:44.424 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-27T01:54:44.432 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-27T01:54:44.432 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-27T01:54:44.492 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-27T01:54:44.492 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-27T01:54:44.641 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-27T01:54:44.743 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:54:44.769 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:54:44.769 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:54:44.769 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-27T01:54:44.769 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:45.679 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:54:45.679 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:45.679 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-27T01:54:45.679 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 2024-04-27T01:54:45.680 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:45.680 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:54:45.943 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr 2024-04-27T01:54:46.221 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 12/12 2024-04-27T01:54:46.221 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 1/12 2024-04-27T01:54:46.222 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-27T01:54:46.222 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-27T01:54:46.222 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-27T01:54:46.222 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-27T01:54:46.222 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-27T01:54:46.222 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-27T01:54:46.222 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-27T01:54:46.222 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-27T01:54:46.222 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-27T01:54:46.222 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-27T01:54:46.480 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:04 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:54:46.561 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-mgr-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-27T01:54:46.604 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:54:46.605 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-27T01:54:46.606 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:54:46.649 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-27T01:54:46.650 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:46.650 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-04-27T01:54:46.650 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarch 2024-04-27T01:54:46.650 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-27T01:54:46.650 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-27T01:54:46.650 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-27T01:54:46.650 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-27T01:54:46.650 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-27T01:54:46.650 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-27T01:54:46.650 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-27T01:54:46.650 INFO:teuthology.orchestra.run.smithi052.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-27T01:54:46.651 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-27T01:54:46.651 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-27T01:54:46.651 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-27T01:54:46.651 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:46.651 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:54:46.695 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-dashboard 2024-04-27T01:54:47.001 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-rook 2024-04-27T01:54:47.228 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:05 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:54:47.353 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:54:47.354 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:54:47.354 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-27T01:54:47.354 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:54:47.354 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-27T01:54:47.354 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard noarch 2:18.2.0-0.el9 ceph-noarch 1.7 M 2024-04-27T01:54:47.355 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-27T01:54:47.355 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards noarch 2:18.2.0-0.el9 ceph-noarch 23 k 2024-04-27T01:54:47.355 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts noarch 2:18.2.0-0.el9 ceph-noarch 15 k 2024-04-27T01:54:47.355 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-27T01:54:47.355 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-27T01:54:47.355 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-27T01:54:47.355 INFO:teuthology.orchestra.run.smithi029.stdout:Installing weak dependencies: 2024-04-27T01:54:47.355 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-27T01:54:47.355 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:47.355 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-27T01:54:47.355 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:54:47.355 INFO:teuthology.orchestra.run.smithi029.stdout:Install 7 Packages 2024-04-27T01:54:47.355 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:47.356 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 2.0 M 2024-04-27T01:54:47.356 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 8.1 M 2024-04-27T01:54:47.356 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-27T01:54:47.534 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:08 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:54:47.620 INFO:teuthology.orchestra.run.smithi029.stdout:(1/7): ceph-grafana-dashboards-18.2.0-0.el9.noa 195 kB/s | 23 kB 00:00 2024-04-27T01:54:47.645 INFO:teuthology.orchestra.run.smithi029.stdout:(2/7): ceph-prometheus-alerts-18.2.0-0.el9.noar 102 kB/s | 15 kB 00:00 2024-04-27T01:54:47.660 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:54:47.661 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:47.661 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-04-27T01:54:47.661 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:47.661 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-04-27T01:54:47.661 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook noarch 2:18.2.0-0.el9 ceph-noarch 47 k 2024-04-27T01:54:47.661 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-04-27T01:54:47.661 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-27T01:54:47.661 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-27T01:54:47.662 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-27T01:54:47.662 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-27T01:54:47.662 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-27T01:54:47.662 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-27T01:54:47.662 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-27T01:54:47.662 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-27T01:54:47.662 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-27T01:54:47.662 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-27T01:54:47.662 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-27T01:54:47.662 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-27T01:54:47.662 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:47.662 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-04-27T01:54:47.663 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:47.663 INFO:teuthology.orchestra.run.smithi052.stdout:Install 13 Packages 2024-04-27T01:54:47.663 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:47.663 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 2.2 M 2024-04-27T01:54:47.663 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 26 M 2024-04-27T01:54:47.663 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-04-27T01:54:47.704 INFO:teuthology.orchestra.run.smithi029.stdout:(3/7): ceph-mgr-dashboard-18.2.0-0.el9.noarch.r 8.4 MB/s | 1.7 MB 00:00 2024-04-27T01:54:47.855 INFO:teuthology.orchestra.run.smithi029.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 39 kB/s | 9.0 kB 00:00 2024-04-27T01:54:47.972 INFO:teuthology.orchestra.run.smithi029.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 125 kB/s | 41 kB 00:00 2024-04-27T01:54:47.997 INFO:teuthology.orchestra.run.smithi029.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 126 kB/s | 31 kB 00:00 2024-04-27T01:54:48.081 INFO:teuthology.orchestra.run.smithi029.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 834 kB/s | 188 kB 00:00 2024-04-27T01:54:48.085 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:54:48.085 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.7 MB/s | 2.0 MB 00:00 2024-04-27T01:54:48.085 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-27T01:54:48.094 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-27T01:54:48.094 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-27T01:54:48.132 INFO:teuthology.orchestra.run.smithi052.stdout:(1/13): ceph-mgr-rook-18.2.0-0.el9.noarch.rpm 820 kB/s | 47 kB 00:00 2024-04-27T01:54:48.150 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-27T01:54:48.151 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-27T01:54:48.236 INFO:teuthology.orchestra.run.smithi052.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 171 kB/s | 26 kB 00:00 2024-04-27T01:54:48.249 INFO:teuthology.orchestra.run.smithi052.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 109 kB/s | 19 kB 00:00 2024-04-27T01:54:48.316 INFO:teuthology.orchestra.run.smithi052.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-04-27T01:54:48.341 INFO:teuthology.orchestra.run.smithi052.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.0 MB/s | 222 kB 00:00 2024-04-27T01:54:48.366 INFO:teuthology.orchestra.run.smithi052.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.3 MB/s | 279 kB 00:00 2024-04-27T01:54:48.392 INFO:teuthology.orchestra.run.smithi052.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 709 kB/s | 54 kB 00:00 2024-04-27T01:54:48.442 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-27T01:54:48.500 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-27T01:54:48.535 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-27T01:54:48.584 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-27T01:54:48.623 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-27T01:54:48.634 INFO:teuthology.orchestra.run.smithi052.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-04-27T01:54:48.667 INFO:teuthology.orchestra.run.smithi052.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 99 kB/s | 32 kB 00:00 2024-04-27T01:54:48.669 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 5/7 2024-04-27T01:54:48.868 INFO:teuthology.orchestra.run.smithi052.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 432 kB/s | 206 kB 00:00 2024-04-27T01:54:48.893 INFO:teuthology.orchestra.run.smithi052.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 262 kB/s | 59 kB 00:00 2024-04-27T01:54:48.910 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 6/7 2024-04-27T01:54:48.939 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 7/7 2024-04-27T01:54:49.052 INFO:teuthology.orchestra.run.smithi052.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 489 kB/s | 90 kB 00:00 2024-04-27T01:54:49.160 INFO:teuthology.orchestra.run.smithi052.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-04-27T01:54:49.161 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:54:49.161 INFO:teuthology.orchestra.run.smithi052.stdout:Total 1.5 MB/s | 2.2 MB 00:01 2024-04-27T01:54:49.232 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-04-27T01:54:49.248 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-04-27T01:54:49.248 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-04-27T01:54:49.340 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-04-27T01:54:49.341 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-04-27T01:54:49.540 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 7/7 2024-04-27T01:54:49.540 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 1/7 2024-04-27T01:54:49.540 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 2/7 2024-04-27T01:54:49.540 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 3/7 2024-04-27T01:54:49.540 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-27T01:54:49.540 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-27T01:54:49.540 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-27T01:54:49.796 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-27T01:54:49.796 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:49.796 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-27T01:54:49.796 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 2024-04-27T01:54:49.796 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 2024-04-27T01:54:49.797 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 2024-04-27T01:54:49.797 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-27T01:54:49.797 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-27T01:54:49.797 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-27T01:54:49.797 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-27T01:54:49.797 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:49.797 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:54:49.926 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-04-27T01:54:50.036 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-27T01:54:50.100 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-27T01:54:50.164 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-27T01:54:50.228 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-27T01:54:50.263 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-27T01:54:50.297 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-27T01:54:50.396 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-27T01:54:50.507 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-04-27T01:54:50.557 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-27T01:54:50.566 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:08 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:54:50.691 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:54:50.692 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:54:50.692 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-27T01:54:50.692 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:54:50.692 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-27T01:54:50.692 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-0.el9 ceph-noarch 7.4 M 2024-04-27T01:54:50.693 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-27T01:54:50.693 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-27T01:54:50.693 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-27T01:54:50.693 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-27T01:54:50.693 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-27T01:54:50.693 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-27T01:54:50.693 INFO:teuthology.orchestra.run.smithi029.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-27T01:54:50.693 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-27T01:54:50.693 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-27T01:54:50.693 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-27T01:54:50.693 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-27T01:54:50.694 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-27T01:54:50.694 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:50.694 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-27T01:54:50.694 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:54:50.694 INFO:teuthology.orchestra.run.smithi029.stdout:Install 12 Packages 2024-04-27T01:54:50.694 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:54:50.695 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 38 M 2024-04-27T01:54:50.695 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 209 M 2024-04-27T01:54:50.695 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-27T01:54:51.297 INFO:teuthology.orchestra.run.smithi029.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.0-0. 29 MB/s | 7.4 MB 00:00 2024-04-27T01:54:51.316 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-27T01:54:51.371 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-27T01:54:51.404 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-27T01:54:51.450 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-27T01:54:51.474 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-rook-2:18.2.0-0.el9.noarch 13/13 2024-04-27T01:54:51.681 INFO:teuthology.orchestra.run.smithi029.stdout:(2/12): libquadmath-11.4.1-3.el9.x86_64.rpm 306 kB/s | 195 kB 00:00 2024-04-27T01:54:51.773 INFO:teuthology.orchestra.run.smithi029.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-04-27T01:54:51.982 INFO:teuthology.orchestra.run.smithi029.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 48 kB/s | 33 kB 00:00 2024-04-27T01:54:52.226 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-0.el9.noarch 13/13 2024-04-27T01:54:52.226 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-rook-2:18.2.0-0.el9.noarch 1/13 2024-04-27T01:54:52.226 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-27T01:54:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-27T01:54:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-27T01:54:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-27T01:54:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-27T01:54:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-27T01:54:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-27T01:54:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-27T01:54:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-04-27T01:54:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-27T01:54:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-27T01:54:52.340 INFO:teuthology.orchestra.run.smithi029.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 32 kB/s | 18 kB 00:00 2024-04-27T01:54:52.546 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-27T01:54:52.546 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:52.546 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-04-27T01:54:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook-2:18.2.0-0.el9.noarch 2024-04-27T01:54:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-27T01:54:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-27T01:54:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-27T01:54:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-27T01:54:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-27T01:54:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-27T01:54:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-27T01:54:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-27T01:54:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-27T01:54:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-27T01:54:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-27T01:54:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-27T01:54:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:52.548 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:54:52.633 INFO:teuthology.orchestra.run.smithi029.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 54 kB/s | 35 kB 00:00 2024-04-27T01:54:52.767 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-cephadm 2024-04-27T01:54:53.310 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:14 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:54:53.432 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:54:53.432 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:53.432 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-04-27T01:54:53.433 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:53.433 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-04-27T01:54:53.433 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm noarch 2:18.2.0-0.el9 ceph-noarch 127 k 2024-04-27T01:54:53.433 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-04-27T01:54:53.433 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-27T01:54:53.433 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-27T01:54:53.433 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-27T01:54:53.433 INFO:teuthology.orchestra.run.smithi052.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-27T01:54:53.433 INFO:teuthology.orchestra.run.smithi052.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-27T01:54:53.433 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:53.433 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-04-27T01:54:53.433 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:53.433 INFO:teuthology.orchestra.run.smithi052.stdout:Install 6 Packages 2024-04-27T01:54:53.434 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:53.434 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 7.0 M 2024-04-27T01:54:53.434 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 33 M 2024-04-27T01:54:53.434 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-04-27T01:54:53.550 INFO:teuthology.orchestra.run.smithi029.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 266 kB/s | 244 kB 00:00 2024-04-27T01:54:53.797 INFO:teuthology.orchestra.run.smithi052.stdout:(1/6): ceph-mgr-cephadm-18.2.0-0.el9.noarch.rpm 1.8 MB/s | 127 kB 00:00 2024-04-27T01:54:53.964 INFO:teuthology.orchestra.run.smithi052.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.0 MB/s | 248 kB 00:00 2024-04-27T01:54:54.189 INFO:teuthology.orchestra.run.smithi052.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 6.0 MB 00:00 2024-04-27T01:54:54.314 INFO:teuthology.orchestra.run.smithi052.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 165 kB/s | 58 kB 00:00 2024-04-27T01:54:54.506 INFO:teuthology.orchestra.run.smithi052.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 773 kB/s | 548 kB 00:00 2024-04-27T01:54:54.531 INFO:teuthology.orchestra.run.smithi052.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 149 kB/s | 51 kB 00:00 2024-04-27T01:54:54.531 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:54:54.532 INFO:teuthology.orchestra.run.smithi052.stdout:Total 6.3 MB/s | 7.0 MB 00:01 2024-04-27T01:54:54.606 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-04-27T01:54:54.619 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-04-27T01:54:54.619 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-04-27T01:54:54.721 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-04-27T01:54:54.722 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-04-27T01:54:55.029 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-04-27T01:54:55.203 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-27T01:54:55.251 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-27T01:54:55.798 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-27T01:54:55.881 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-27T01:54:55.947 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-27T01:54:55.970 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 6/6 2024-04-27T01:54:56.125 INFO:teuthology.orchestra.run.smithi029.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 689 kB/s | 3.0 MB 00:04 2024-04-27T01:54:56.708 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 6/6 2024-04-27T01:54:56.739 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 1/6 2024-04-27T01:54:56.739 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-27T01:54:56.739 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-27T01:54:56.739 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-27T01:54:56.739 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-27T01:54:56.962 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-27T01:54:56.962 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:56.962 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-04-27T01:54:56.963 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 2024-04-27T01:54:56.963 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-27T01:54:56.963 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-27T01:54:56.963 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-27T01:54:56.963 INFO:teuthology.orchestra.run.smithi052.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-27T01:54:56.963 INFO:teuthology.orchestra.run.smithi052.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-27T01:54:56.963 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:56.963 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:54:57.042 INFO:teuthology.orchestra.run.smithi029.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 273 kB/s | 250 kB 00:00 2024-04-27T01:54:57.189 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-fuse 2024-04-27T01:54:57.718 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:18 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:54:57.841 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:54:57.841 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:57.841 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-04-27T01:54:57.842 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:57.842 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-04-27T01:54:57.842 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse x86_64 2:18.2.0-0.el9 ceph 835 k 2024-04-27T01:54:57.842 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-04-27T01:54:57.842 INFO:teuthology.orchestra.run.smithi052.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-27T01:54:57.842 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:57.842 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-04-27T01:54:57.842 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:54:57.842 INFO:teuthology.orchestra.run.smithi052.stdout:Install 2 Packages 2024-04-27T01:54:57.842 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:57.843 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 915 k 2024-04-27T01:54:57.843 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 2.7 M 2024-04-27T01:54:57.843 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-04-27T01:54:57.991 INFO:teuthology.orchestra.run.smithi052.stdout:(1/2): ceph-fuse-18.2.0-0.el9.x86_64.rpm 10 MB/s | 835 kB 00:00 2024-04-27T01:54:58.050 INFO:teuthology.orchestra.run.smithi052.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 584 kB/s | 80 kB 00:00 2024-04-27T01:54:58.051 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:54:58.051 INFO:teuthology.orchestra.run.smithi052.stdout:Total 4.3 MB/s | 915 kB 00:00 2024-04-27T01:54:58.062 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-04-27T01:54:58.085 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-04-27T01:54:58.085 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-04-27T01:54:58.170 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-04-27T01:54:58.170 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-04-27T01:54:58.365 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-04-27T01:54:58.495 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-27T01:54:58.560 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-fuse-2:18.2.0-0.el9.x86_64 2/2 2024-04-27T01:54:59.215 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-fuse-2:18.2.0-0.el9.x86_64 2/2 2024-04-27T01:54:59.215 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-fuse-2:18.2.0-0.el9.x86_64 1/2 2024-04-27T01:54:59.259 INFO:teuthology.orchestra.run.smithi029.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 701 kB/s | 4.7 MB 00:06 2024-04-27T01:54:59.462 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-27T01:54:59.462 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:59.462 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-04-27T01:54:59.463 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse-2:18.2.0-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-27T01:54:59.463 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:54:59.463 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:54:59.733 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados-devel 2024-04-27T01:55:00.266 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:21 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:55:00.392 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:55:00.393 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:55:00.393 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-04-27T01:55:00.393 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:55:00.393 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-04-27T01:55:00.393 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel x86_64 2:18.2.0-0.el9 ceph 126 k 2024-04-27T01:55:00.394 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:55:00.394 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-04-27T01:55:00.394 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:55:00.394 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-04-27T01:55:00.394 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:55:00.394 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 126 k 2024-04-27T01:55:00.394 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 456 k 2024-04-27T01:55:00.394 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-04-27T01:55:00.465 INFO:teuthology.orchestra.run.smithi052.stdout:librados-devel-18.2.0-0.el9.x86_64.rpm 1.7 MB/s | 126 kB 00:00 2024-04-27T01:55:00.466 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:55:00.466 INFO:teuthology.orchestra.run.smithi052.stdout:Total 1.7 MB/s | 126 kB 00:00 2024-04-27T01:55:00.466 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-04-27T01:55:00.474 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-04-27T01:55:00.474 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-04-27T01:55:00.538 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-04-27T01:55:00.538 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-04-27T01:55:00.618 INFO:teuthology.orchestra.run.smithi029.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 738 kB/s | 5.1 MB 00:07 2024-04-27T01:55:00.695 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-04-27T01:55:00.797 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:01.200 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:01.461 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:01.461 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:55:01.461 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-04-27T01:55:01.461 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel-2:18.2.0-0.el9.x86_64 2024-04-27T01:55:01.461 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:55:01.461 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:55:01.701 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs2 2024-04-27T01:55:02.231 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:23 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:55:02.314 INFO:teuthology.orchestra.run.smithi052.stdout:Package libcephfs2-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-27T01:55:02.357 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:55:02.358 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-04-27T01:55:02.358 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:55:02.411 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs-devel 2024-04-27T01:55:02.943 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:23 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:55:03.067 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:55:03.068 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:55:03.068 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-04-27T01:55:03.068 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:55:03.068 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-04-27T01:55:03.068 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel x86_64 2:18.2.0-0.el9 ceph 30 k 2024-04-27T01:55:03.068 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:55:03.069 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-04-27T01:55:03.069 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:55:03.069 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-04-27T01:55:03.069 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:55:03.069 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 30 k 2024-04-27T01:55:03.069 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 137 k 2024-04-27T01:55:03.069 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-04-27T01:55:03.127 INFO:teuthology.orchestra.run.smithi052.stdout:libcephfs-devel-18.2.0-0.el9.x86_64.rpm 526 kB/s | 30 kB 00:00 2024-04-27T01:55:03.128 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:55:03.128 INFO:teuthology.orchestra.run.smithi052.stdout:Total 514 kB/s | 30 kB 00:00 2024-04-27T01:55:03.128 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-04-27T01:55:03.134 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-04-27T01:55:03.134 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-04-27T01:55:03.135 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-04-27T01:55:03.135 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-04-27T01:55:03.213 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-04-27T01:55:03.325 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:03.660 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:03.868 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:03.868 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:55:03.868 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-04-27T01:55:03.869 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel-2:18.2.0-0.el9.x86_64 2024-04-27T01:55:03.869 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:55:03.869 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:55:04.046 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados2 2024-04-27T01:55:04.574 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:25 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:55:04.655 INFO:teuthology.orchestra.run.smithi052.stdout:Package librados2-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-27T01:55:04.699 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:55:04.700 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-04-27T01:55:04.700 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:55:04.764 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librbd1 2024-04-27T01:55:05.291 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:26 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:55:05.371 INFO:teuthology.orchestra.run.smithi052.stdout:Package librbd1-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-27T01:55:05.415 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:55:05.416 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-04-27T01:55:05.416 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:55:05.479 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rados 2024-04-27T01:55:06.011 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:27 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:55:06.092 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rados-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-27T01:55:06.134 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:55:06.135 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-04-27T01:55:06.136 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:55:06.195 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rgw 2024-04-27T01:55:06.725 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:27 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:55:06.807 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rgw-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-27T01:55:06.849 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:55:06.851 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-04-27T01:55:06.851 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:55:06.911 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-cephfs 2024-04-27T01:55:07.441 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:28 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:55:07.523 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-cephfs-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-27T01:55:07.565 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:55:07.566 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-04-27T01:55:07.567 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:55:07.621 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rbd 2024-04-27T01:55:08.147 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:29 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:55:08.228 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rbd-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-27T01:55:08.271 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:55:08.272 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-04-27T01:55:08.272 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:55:08.331 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-fuse 2024-04-27T01:55:08.860 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:29 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:55:08.984 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:55:08.985 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:55:08.985 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-04-27T01:55:08.985 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:55:08.985 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-04-27T01:55:08.986 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse x86_64 2:18.2.0-0.el9 ceph 86 k 2024-04-27T01:55:08.986 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:55:08.986 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-04-27T01:55:08.986 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:55:08.986 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-04-27T01:55:08.986 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:55:08.986 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 86 k 2024-04-27T01:55:08.986 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 230 k 2024-04-27T01:55:08.986 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-04-27T01:55:09.064 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-fuse-18.2.0-0.el9.x86_64.rpm 1.1 MB/s | 86 kB 00:00 2024-04-27T01:55:09.065 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:55:09.065 INFO:teuthology.orchestra.run.smithi052.stdout:Total 1.1 MB/s | 86 kB 00:00 2024-04-27T01:55:09.065 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-04-27T01:55:09.072 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-04-27T01:55:09.072 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-04-27T01:55:09.138 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-04-27T01:55:09.138 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-04-27T01:55:09.288 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-04-27T01:55:09.404 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:09.722 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:09.969 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:09.969 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:55:09.969 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-04-27T01:55:09.969 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse-2:18.2.0-0.el9.x86_64 2024-04-27T01:55:09.969 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:55:09.969 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:55:10.184 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-mirror 2024-04-27T01:55:10.715 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:31 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:55:10.840 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:55:10.841 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:55:10.841 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-04-27T01:55:10.841 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:55:10.841 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-04-27T01:55:10.841 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror x86_64 2:18.2.0-0.el9 ceph 3.0 M 2024-04-27T01:55:10.841 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:55:10.841 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-04-27T01:55:10.841 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:55:10.841 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-04-27T01:55:10.841 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:55:10.842 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 3.0 M 2024-04-27T01:55:10.842 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 12 M 2024-04-27T01:55:10.842 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-04-27T01:55:10.950 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-mirror-18.2.0-0.el9.x86_64.rpm 28 MB/s | 3.0 MB 00:00 2024-04-27T01:55:10.951 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:55:10.951 INFO:teuthology.orchestra.run.smithi052.stdout:Total 27 MB/s | 3.0 MB 00:00 2024-04-27T01:55:10.951 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-04-27T01:55:10.961 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-04-27T01:55:10.961 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-04-27T01:55:11.044 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-04-27T01:55:11.044 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-04-27T01:55:11.483 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-04-27T01:55:11.651 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:11.677 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:11.678 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:55:11.678 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-27T01:55:11.678 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-27T01:55:11.678 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-27T01:55:11.678 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:55:12.519 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:12.519 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:55:12.519 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-04-27T01:55:12.519 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror-2:18.2.0-0.el9.x86_64 2024-04-27T01:55:12.519 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:55:12.519 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:55:12.771 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-nbd 2024-04-27T01:55:13.302 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:34 ago on Sat 27 Apr 2024 01:53:39 AM UTC. 2024-04-27T01:55:13.427 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-27T01:55:13.428 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:55:13.428 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-04-27T01:55:13.429 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:55:13.429 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-04-27T01:55:13.429 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd x86_64 2:18.2.0-0.el9 ceph 169 k 2024-04-27T01:55:13.429 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:55:13.429 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-04-27T01:55:13.429 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-04-27T01:55:13.429 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-04-27T01:55:13.429 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:55:13.429 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 169 k 2024-04-27T01:55:13.429 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 490 k 2024-04-27T01:55:13.429 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-04-27T01:55:13.504 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-nbd-18.2.0-0.el9.x86_64.rpm 2.2 MB/s | 169 kB 00:00 2024-04-27T01:55:13.505 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:55:13.505 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.2 MB/s | 169 kB 00:00 2024-04-27T01:55:13.505 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-04-27T01:55:13.513 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-04-27T01:55:13.513 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-04-27T01:55:13.579 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-04-27T01:55:13.579 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-04-27T01:55:13.739 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-04-27T01:55:13.844 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:14.229 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:14.459 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:14.459 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:55:14.459 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-04-27T01:55:14.459 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd-2:18.2.0-0.el9.x86_64 2024-04-27T01:55:14.459 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:55:14.459 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-27T01:55:14.684 DEBUG:teuthology.parallel:result is None 2024-04-27T01:55:18.618 INFO:teuthology.orchestra.run.smithi029.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 771 kB/s | 16 MB 00:21 2024-04-27T01:55:18.620 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:55:18.620 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.4 MB/s | 38 MB 00:27 2024-04-27T01:55:18.982 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-27T01:55:19.012 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-27T01:55:19.012 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-27T01:55:19.449 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-27T01:55:19.449 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-27T01:55:20.234 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-27T01:55:20.307 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-27T01:55:20.351 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-27T01:55:20.502 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-27T01:55:20.545 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-27T01:55:20.731 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-27T01:55:20.761 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-27T01:55:20.850 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-27T01:55:21.548 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-27T01:55:21.690 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-27T01:55:23.012 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-27T01:55:24.128 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-27T01:55:24.161 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 12/12 2024-04-27T01:55:28.428 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 12/12 2024-04-27T01:55:28.428 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 1/12 2024-04-27T01:55:28.428 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-27T01:55:28.428 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-27T01:55:28.428 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-27T01:55:28.428 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-27T01:55:28.428 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-27T01:55:28.428 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-27T01:55:28.428 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-27T01:55:28.429 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-27T01:55:28.429 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-27T01:55:28.429 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-27T01:55:28.843 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-27T01:55:28.843 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:28.843 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-27T01:55:28.843 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarch 2024-04-27T01:55:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-27T01:55:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-27T01:55:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-27T01:55:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-27T01:55:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-27T01:55:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-27T01:55:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-27T01:55:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-27T01:55:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-27T01:55:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-27T01:55:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-27T01:55:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:28.844 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:55:29.292 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-rook 2024-04-27T01:55:29.831 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:47 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:55:29.956 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:55:29.958 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:29.958 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-27T01:55:29.958 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:29.958 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-27T01:55:29.958 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook noarch 2:18.2.0-0.el9 ceph-noarch 47 k 2024-04-27T01:55:29.958 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-27T01:55:29.958 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-27T01:55:29.958 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-27T01:55:29.958 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-27T01:55:29.958 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-27T01:55:29.959 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-27T01:55:29.959 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-27T01:55:29.959 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-27T01:55:29.959 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-27T01:55:29.959 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-27T01:55:29.959 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-27T01:55:29.959 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-27T01:55:29.959 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-27T01:55:29.959 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:29.959 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-27T01:55:29.959 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:29.959 INFO:teuthology.orchestra.run.smithi029.stdout:Install 13 Packages 2024-04-27T01:55:29.959 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:29.960 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 2.2 M 2024-04-27T01:55:29.960 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 26 M 2024-04-27T01:55:29.960 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-27T01:55:30.477 INFO:teuthology.orchestra.run.smithi029.stdout:(1/13): ceph-mgr-rook-18.2.0-0.el9.noarch.rpm 729 kB/s | 47 kB 00:00 2024-04-27T01:55:30.936 INFO:teuthology.orchestra.run.smithi029.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 37 kB/s | 19 kB 00:00 2024-04-27T01:55:31.011 INFO:teuthology.orchestra.run.smithi029.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 43 kB/s | 26 kB 00:00 2024-04-27T01:55:31.328 INFO:teuthology.orchestra.run.smithi029.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 261 kB/s | 222 kB 00:00 2024-04-27T01:55:31.754 INFO:teuthology.orchestra.run.smithi029.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 194 kB/s | 159 kB 00:00 2024-04-27T01:55:31.888 INFO:teuthology.orchestra.run.smithi029.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 241 kB/s | 32 kB 00:00 2024-04-27T01:55:31.923 INFO:teuthology.orchestra.run.smithi029.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 426 kB/s | 15 kB 00:00 2024-04-27T01:55:31.974 INFO:teuthology.orchestra.run.smithi029.stdout:(8/13): python3-requests-oauthlib-1.3.0-12.el9. 83 kB/s | 54 kB 00:00 2024-04-27T01:55:32.016 INFO:teuthology.orchestra.run.smithi029.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 278 kB/s | 279 kB 00:01 2024-04-27T01:55:32.050 INFO:teuthology.orchestra.run.smithi029.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 1.6 MB/s | 206 kB 00:00 2024-04-27T01:55:32.101 INFO:teuthology.orchestra.run.smithi029.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 8.1 MB/s | 1.0 MB 00:00 2024-04-27T01:55:32.126 INFO:teuthology.orchestra.run.smithi029.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 545 kB/s | 59 kB 00:00 2024-04-27T01:55:32.151 INFO:teuthology.orchestra.run.smithi029.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 902 kB/s | 90 kB 00:00 2024-04-27T01:55:32.154 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:55:32.154 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.0 MB/s | 2.2 MB 00:02 2024-04-27T01:55:32.226 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-27T01:55:32.244 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-27T01:55:32.244 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-27T01:55:32.338 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-27T01:55:32.338 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-27T01:55:32.929 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-27T01:55:33.118 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-27T01:55:33.182 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-27T01:55:33.258 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-27T01:55:33.301 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-27T01:55:33.345 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-27T01:55:33.452 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-27T01:55:33.572 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-04-27T01:55:33.634 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-27T01:55:34.399 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-27T01:55:34.469 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-27T01:55:34.502 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-27T01:55:34.549 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-27T01:55:34.573 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-rook-2:18.2.0-0.el9.noarch 13/13 2024-04-27T01:55:35.628 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-0.el9.noarch 13/13 2024-04-27T01:55:35.628 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-rook-2:18.2.0-0.el9.noarch 1/13 2024-04-27T01:55:35.628 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-27T01:55:35.628 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-27T01:55:35.628 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-27T01:55:35.628 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-27T01:55:35.628 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-27T01:55:35.629 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-27T01:55:35.629 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-27T01:55:35.629 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-27T01:55:35.629 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-04-27T01:55:35.629 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-27T01:55:35.629 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-27T01:55:35.941 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-27T01:55:35.941 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:35.941 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-27T01:55:35.941 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook-2:18.2.0-0.el9.noarch 2024-04-27T01:55:35.941 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-27T01:55:35.941 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-27T01:55:35.941 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-27T01:55:35.941 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-27T01:55:35.941 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-27T01:55:35.941 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-27T01:55:35.941 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-27T01:55:35.942 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-27T01:55:35.942 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-27T01:55:35.942 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-27T01:55:35.942 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-27T01:55:35.942 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-27T01:55:35.942 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:35.942 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:55:36.235 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-cephadm 2024-04-27T01:55:36.766 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:54 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:55:36.890 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:55:36.891 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:36.891 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-27T01:55:36.891 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:36.891 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-27T01:55:36.891 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm noarch 2:18.2.0-0.el9 ceph-noarch 127 k 2024-04-27T01:55:36.891 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-27T01:55:36.891 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-27T01:55:36.891 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-27T01:55:36.891 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-27T01:55:36.891 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-27T01:55:36.891 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-27T01:55:36.892 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:36.892 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-27T01:55:36.892 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:36.892 INFO:teuthology.orchestra.run.smithi029.stdout:Install 6 Packages 2024-04-27T01:55:36.892 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:36.892 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 7.0 M 2024-04-27T01:55:36.892 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 33 M 2024-04-27T01:55:36.893 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-27T01:55:37.477 INFO:teuthology.orchestra.run.smithi029.stdout:(1/6): ceph-mgr-cephadm-18.2.0-0.el9.noarch.rpm 1.6 MB/s | 127 kB 00:00 2024-04-27T01:55:37.602 INFO:teuthology.orchestra.run.smithi029.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-04-27T01:55:37.853 INFO:teuthology.orchestra.run.smithi029.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 6.0 MB 00:00 2024-04-27T01:55:37.878 INFO:teuthology.orchestra.run.smithi029.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 209 kB/s | 58 kB 00:00 2024-04-27T01:55:37.912 INFO:teuthology.orchestra.run.smithi029.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-04-27T01:55:38.081 INFO:teuthology.orchestra.run.smithi029.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 224 kB/s | 51 kB 00:00 2024-04-27T01:55:38.081 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:55:38.081 INFO:teuthology.orchestra.run.smithi029.stdout:Total 5.9 MB/s | 7.0 MB 00:01 2024-04-27T01:55:38.155 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-27T01:55:38.166 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-27T01:55:38.166 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-27T01:55:38.267 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-27T01:55:38.268 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-27T01:55:38.579 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-27T01:55:38.763 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-27T01:55:38.827 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-27T01:55:39.392 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-27T01:55:39.484 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-27T01:55:39.552 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-27T01:55:39.576 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 6/6 2024-04-27T01:55:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 6/6 2024-04-27T01:55:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 1/6 2024-04-27T01:55:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-27T01:55:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-27T01:55:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-27T01:55:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-27T01:55:40.823 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-27T01:55:40.823 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:40.823 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-27T01:55:40.823 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 2024-04-27T01:55:40.823 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-27T01:55:40.823 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-27T01:55:40.823 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-27T01:55:40.823 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-27T01:55:40.824 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-27T01:55:40.824 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:40.824 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:55:41.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-fuse 2024-04-27T01:55:41.646 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:59 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:55:41.770 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:55:41.771 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:41.771 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-04-27T01:55:41.771 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:41.771 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-27T01:55:41.771 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse x86_64 2:18.2.0-0.el9 ceph 835 k 2024-04-27T01:55:41.771 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-27T01:55:41.771 INFO:teuthology.orchestra.run.smithi029.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-27T01:55:41.771 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:41.771 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-27T01:55:41.771 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:41.771 INFO:teuthology.orchestra.run.smithi029.stdout:Install 2 Packages 2024-04-27T01:55:41.771 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:41.772 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 915 k 2024-04-27T01:55:41.772 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 2.7 M 2024-04-27T01:55:41.772 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-27T01:55:42.153 INFO:teuthology.orchestra.run.smithi029.stdout:(1/2): ceph-fuse-18.2.0-0.el9.x86_64.rpm 8.0 MB/s | 835 kB 00:00 2024-04-27T01:55:42.203 INFO:teuthology.orchestra.run.smithi029.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 526 kB/s | 80 kB 00:00 2024-04-27T01:55:42.204 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:55:42.204 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.1 MB/s | 915 kB 00:00 2024-04-27T01:55:42.216 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-27T01:55:42.239 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-27T01:55:42.240 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-27T01:55:42.325 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-27T01:55:42.326 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-27T01:55:42.516 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-27T01:55:42.664 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-27T01:55:42.738 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-2:18.2.0-0.el9.x86_64 2/2 2024-04-27T01:55:43.422 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-fuse-2:18.2.0-0.el9.x86_64 2/2 2024-04-27T01:55:43.422 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-2:18.2.0-0.el9.x86_64 1/2 2024-04-27T01:55:43.709 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-27T01:55:43.709 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:43.710 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-27T01:55:43.710 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-2:18.2.0-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-27T01:55:43.710 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:43.710 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:55:44.008 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados-devel 2024-04-27T01:55:44.544 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:02 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:55:44.670 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:55:44.671 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:44.671 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-04-27T01:55:44.671 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:44.671 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-27T01:55:44.671 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel x86_64 2:18.2.0-0.el9 ceph 126 k 2024-04-27T01:55:44.671 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:44.671 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-27T01:55:44.671 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:44.671 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-04-27T01:55:44.671 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:44.672 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 126 k 2024-04-27T01:55:44.672 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 456 k 2024-04-27T01:55:44.672 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-27T01:55:44.739 INFO:teuthology.orchestra.run.smithi029.stdout:librados-devel-18.2.0-0.el9.x86_64.rpm 1.8 MB/s | 126 kB 00:00 2024-04-27T01:55:44.740 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:55:44.740 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.8 MB/s | 126 kB 00:00 2024-04-27T01:55:44.741 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-27T01:55:44.747 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-27T01:55:44.747 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-27T01:55:44.813 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-27T01:55:44.813 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-27T01:55:44.965 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-27T01:55:45.108 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:45.531 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:45.745 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:45.746 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:45.746 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-27T01:55:45.746 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-2:18.2.0-0.el9.x86_64 2024-04-27T01:55:45.746 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:45.746 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:55:46.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs2 2024-04-27T01:55:46.586 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:04 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:55:46.667 INFO:teuthology.orchestra.run.smithi029.stdout:Package libcephfs2-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-27T01:55:46.710 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:55:46.712 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-27T01:55:46.712 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:55:46.786 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs-devel 2024-04-27T01:55:47.318 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:05 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:55:47.444 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:55:47.444 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:47.444 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-04-27T01:55:47.445 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:47.445 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-27T01:55:47.445 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel x86_64 2:18.2.0-0.el9 ceph 30 k 2024-04-27T01:55:47.445 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:47.445 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-27T01:55:47.445 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:47.445 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-04-27T01:55:47.445 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:47.445 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 30 k 2024-04-27T01:55:47.445 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 137 k 2024-04-27T01:55:47.445 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-27T01:55:47.519 INFO:teuthology.orchestra.run.smithi029.stdout:libcephfs-devel-18.2.0-0.el9.x86_64.rpm 416 kB/s | 30 kB 00:00 2024-04-27T01:55:47.519 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:55:47.520 INFO:teuthology.orchestra.run.smithi029.stdout:Total 408 kB/s | 30 kB 00:00 2024-04-27T01:55:47.520 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-27T01:55:47.523 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-27T01:55:47.523 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-27T01:55:47.526 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-27T01:55:47.526 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-27T01:55:47.616 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-27T01:55:47.725 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:48.125 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:48.338 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:48.338 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:48.338 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-27T01:55:48.338 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel-2:18.2.0-0.el9.x86_64 2024-04-27T01:55:48.338 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:48.338 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:55:48.545 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados2 2024-04-27T01:55:49.077 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:07 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:55:49.158 INFO:teuthology.orchestra.run.smithi029.stdout:Package librados2-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-27T01:55:49.201 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:55:49.202 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-27T01:55:49.202 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:55:49.268 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librbd1 2024-04-27T01:55:49.800 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:07 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:55:49.882 INFO:teuthology.orchestra.run.smithi029.stdout:Package librbd1-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-27T01:55:49.924 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:55:49.925 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-27T01:55:49.925 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:55:49.995 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rados 2024-04-27T01:55:50.532 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:08 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:55:50.613 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rados-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-27T01:55:50.659 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:55:50.659 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-27T01:55:50.659 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:55:50.727 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rgw 2024-04-27T01:55:51.260 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:09 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:55:51.341 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rgw-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-27T01:55:51.384 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:55:51.385 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-27T01:55:51.385 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:55:51.454 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-cephfs 2024-04-27T01:55:51.986 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:10 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:55:52.068 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-cephfs-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-27T01:55:52.111 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:55:52.112 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-27T01:55:52.112 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:55:52.184 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rbd 2024-04-27T01:55:52.718 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:10 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:55:52.799 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rbd-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-27T01:55:52.842 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:55:52.844 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-04-27T01:55:52.844 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:55:52.910 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-fuse 2024-04-27T01:55:53.445 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:11 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:55:53.569 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:55:53.569 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:53.569 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-04-27T01:55:53.569 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:53.569 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-27T01:55:53.569 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse x86_64 2:18.2.0-0.el9 ceph 86 k 2024-04-27T01:55:53.570 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:53.570 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-27T01:55:53.570 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:53.570 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-04-27T01:55:53.570 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:53.570 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 86 k 2024-04-27T01:55:53.571 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 230 k 2024-04-27T01:55:53.571 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-27T01:55:53.660 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-fuse-18.2.0-0.el9.x86_64.rpm 961 kB/s | 86 kB 00:00 2024-04-27T01:55:53.660 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:55:53.661 INFO:teuthology.orchestra.run.smithi029.stdout:Total 948 kB/s | 86 kB 00:00 2024-04-27T01:55:53.661 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-27T01:55:53.668 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-27T01:55:53.668 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-27T01:55:53.734 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-27T01:55:53.735 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-27T01:55:53.883 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-27T01:55:53.992 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:54.419 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:54.714 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:54.714 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:54.714 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-27T01:55:54.714 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-2:18.2.0-0.el9.x86_64 2024-04-27T01:55:54.714 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:54.714 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:55:54.993 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-mirror 2024-04-27T01:55:55.545 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:13 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:55:55.670 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:55:55.671 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:55.671 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-04-27T01:55:55.671 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:55.671 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-27T01:55:55.671 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror x86_64 2:18.2.0-0.el9 ceph 3.0 M 2024-04-27T01:55:55.672 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:55.672 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-27T01:55:55.672 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:55.672 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-04-27T01:55:55.672 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:55.672 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 3.0 M 2024-04-27T01:55:55.672 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 12 M 2024-04-27T01:55:55.672 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-27T01:55:55.801 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-mirror-18.2.0-0.el9.x86_64.rpm 23 MB/s | 3.0 MB 00:00 2024-04-27T01:55:55.801 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:55:55.802 INFO:teuthology.orchestra.run.smithi029.stdout:Total 23 MB/s | 3.0 MB 00:00 2024-04-27T01:55:55.802 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-27T01:55:55.812 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-27T01:55:55.813 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-27T01:55:55.896 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-27T01:55:55.896 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-27T01:55:56.330 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-27T01:55:56.572 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:56.602 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:56.602 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:55:56.602 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-27T01:55:56.602 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-27T01:55:56.603 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-27T01:55:56.603 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:57.580 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:57.580 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:57.580 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-27T01:55:57.581 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-2:18.2.0-0.el9.x86_64 2024-04-27T01:55:57.581 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:57.581 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:55:57.925 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-nbd 2024-04-27T01:55:58.458 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:16 ago on Sat 27 Apr 2024 01:53:42 AM UTC. 2024-04-27T01:55:58.582 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-27T01:55:58.583 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:58.583 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-04-27T01:55:58.583 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:58.584 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-27T01:55:58.584 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd x86_64 2:18.2.0-0.el9 ceph 169 k 2024-04-27T01:55:58.584 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:58.584 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-27T01:55:58.584 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-04-27T01:55:58.584 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-04-27T01:55:58.584 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:58.584 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 169 k 2024-04-27T01:55:58.584 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 490 k 2024-04-27T01:55:58.584 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-27T01:55:58.647 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-nbd-18.2.0-0.el9.x86_64.rpm 2.6 MB/s | 169 kB 00:00 2024-04-27T01:55:58.648 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:55:58.648 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.6 MB/s | 169 kB 00:00 2024-04-27T01:55:58.648 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-27T01:55:58.657 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-27T01:55:58.657 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-27T01:55:58.724 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-27T01:55:58.724 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-27T01:55:58.885 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-27T01:55:59.005 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:59.421 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:59.682 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-04-27T01:55:59.682 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:59.682 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-27T01:55:59.682 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-2:18.2.0-0.el9.x86_64 2024-04-27T01:55:59.682 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:55:59.682 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-27T01:55:59.972 DEBUG:teuthology.parallel:result is None 2024-04-27T01:55:59.972 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-27T01:55:59.972 INFO:teuthology.packaging:ref: None 2024-04-27T01:55:59.972 INFO:teuthology.packaging:tag: v18.2.0 2024-04-27T01:55:59.972 INFO:teuthology.packaging:branch: None 2024-04-27T01:55:59.972 INFO:teuthology.packaging:sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T01:55:59.973 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-04-27T01:56:00.187 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-27T01:56:00.210 INFO:teuthology.orchestra.run.smithi029.stdout:18.2.0-0.el9 2024-04-27T01:56:00.211 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-0.el9 2024-04-27T01:56:00.211 INFO:teuthology.task.install:The correct ceph version 18.2.0-0 is installed. 2024-04-27T01:56:00.213 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-27T01:56:00.213 INFO:teuthology.packaging:ref: None 2024-04-27T01:56:00.213 INFO:teuthology.packaging:tag: v18.2.0 2024-04-27T01:56:00.213 INFO:teuthology.packaging:branch: None 2024-04-27T01:56:00.213 INFO:teuthology.packaging:sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T01:56:00.214 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-04-27T01:56:00.426 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-27T01:56:00.452 INFO:teuthology.orchestra.run.smithi052.stdout:18.2.0-0.el9 2024-04-27T01:56:00.452 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-0.el9 2024-04-27T01:56:00.452 INFO:teuthology.task.install:The correct ceph version 18.2.0-0 is installed. 2024-04-27T01:56:00.453 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-27T01:56:00.454 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-27T01:56:00.454 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-27T01:56:00.482 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-27T01:56:00.482 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-27T01:56:00.522 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-27T01:56:00.523 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-27T01:56:00.523 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2024-04-27T01:56:00.550 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-27T01:56:00.616 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-27T01:56:00.617 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/daemon-helper 2024-04-27T01:56:00.645 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-27T01:56:00.714 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-27T01:56:00.715 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-27T01:56:00.715 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-27T01:56:00.741 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-27T01:56:00.834 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-27T01:56:00.835 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-27T01:56:00.863 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-27T01:56:00.929 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-27T01:56:00.929 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-27T01:56:00.930 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2024-04-27T01:56:00.955 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-27T01:56:01.020 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-27T01:56:01.020 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/stdin-killer 2024-04-27T01:56:01.045 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-27T01:56:01.109 INFO:teuthology.run_tasks:Running task print... 2024-04-27T01:56:01.120 INFO:teuthology.task.print:**** done install task... 2024-04-27T01:56:01.120 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-27T01:56:01.227 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.io/ceph/ceph:v18.2.0', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a'} 2024-04-27T01:56:01.228 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v18.2.0 2024-04-27T01:56:01.228 INFO:tasks.cephadm:Cluster fsid is 4c587076-0439-11ef-bc93-c7b262605968 2024-04-27T01:56:01.228 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-27T01:56:01.228 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-27T01:56:01.229 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi029': '172.21.15.29', 'mon.smithi052': '172.21.15.52'} 2024-04-27T01:56:01.229 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-27T01:56:01.229 DEBUG:teuthology.orchestra.run.smithi029:> sudo hostname $(hostname -s) 2024-04-27T01:56:01.256 DEBUG:teuthology.orchestra.run.smithi052:> sudo hostname $(hostname -s) 2024-04-27T01:56:01.281 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-04-27T01:56:01.281 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T01:56:01.495 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54991/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2447-gb22e2ebd', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2447.gb22e2ebd'}, 'url': 'https://4.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-25 23:00:56.935300', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-27T01:56:01.721 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref reef, sha1 5403a02127f27ec6c3309ef057522ab70cdeb9d2 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-04-27T01:56:01.723 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/reef/5403a02127f27ec6c3309ef057522ab70cdeb9d2/centos/9/x86_64/flavors/default/cephadm 2024-04-27T01:56:01.723 DEBUG:teuthology.orchestra.run.smithi029:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/reef/5403a02127f27ec6c3309ef057522ab70cdeb9d2/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-27T01:56:02.331 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r--. 1 ubuntu ubuntu 215333 Apr 27 01:56 /home/ubuntu/cephtest/cephadm 2024-04-27T01:56:02.331 DEBUG:teuthology.orchestra.run.smithi052:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/reef/5403a02127f27ec6c3309ef057522ab70cdeb9d2/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-27T01:56:02.730 INFO:teuthology.orchestra.run.smithi052.stdout:-rw-r--r--. 1 ubuntu ubuntu 215333 Apr 27 01:56 /home/ubuntu/cephtest/cephadm 2024-04-27T01:56:02.731 DEBUG:teuthology.orchestra.run.smithi029:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-27T01:56:02.750 DEBUG:teuthology.orchestra.run.smithi052:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-27T01:56:02.778 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v18.2.0 on all hosts... 2024-04-27T01:56:02.778 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 pull 2024-04-27T01:56:02.793 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 pull 2024-04-27T01:56:04.229 INFO:teuthology.orchestra.run.smithi052.stderr:Pulling container image quay.io/ceph/ceph:v18.2.0... 2024-04-27T01:56:05.480 INFO:teuthology.orchestra.run.smithi029.stderr:Pulling container image quay.io/ceph/ceph:v18.2.0... 2024-04-27T01:56:23.828 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-27T01:56:23.828 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph_version": "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)", 2024-04-27T01:56:23.829 INFO:teuthology.orchestra.run.smithi052.stdout: "image_id": "dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946", 2024-04-27T01:56:23.829 INFO:teuthology.orchestra.run.smithi052.stdout: "repo_digests": [ 2024-04-27T01:56:23.829 INFO:teuthology.orchestra.run.smithi052.stdout: "quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007", 2024-04-27T01:56:23.829 INFO:teuthology.orchestra.run.smithi052.stdout: "quay.io/ceph/ceph@sha256:d58cf65589d0abf9d5261cd46fb62be3bfb29098febc78c0fcdf116a15274d27" 2024-04-27T01:56:23.829 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-04-27T01:56:23.829 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-27T01:56:26.075 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-04-27T01:56:26.076 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_version": "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)", 2024-04-27T01:56:26.076 INFO:teuthology.orchestra.run.smithi029.stdout: "image_id": "dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946", 2024-04-27T01:56:26.076 INFO:teuthology.orchestra.run.smithi029.stdout: "repo_digests": [ 2024-04-27T01:56:26.076 INFO:teuthology.orchestra.run.smithi029.stdout: "quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007", 2024-04-27T01:56:26.076 INFO:teuthology.orchestra.run.smithi029.stdout: "quay.io/ceph/ceph@sha256:d58cf65589d0abf9d5261cd46fb62be3bfb29098febc78c0fcdf116a15274d27" 2024-04-27T01:56:26.076 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-04-27T01:56:26.076 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-04-27T01:56:26.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph 2024-04-27T01:56:26.155 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph 2024-04-27T01:56:26.187 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /etc/ceph 2024-04-27T01:56:26.213 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 777 /etc/ceph 2024-04-27T01:56:26.256 INFO:tasks.cephadm:Writing seed config... 2024-04-27T01:56:26.257 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-04-27T01:56:26.257 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-04-27T01:56:26.257 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-27T01:56:26.257 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-27T01:56:26.257 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-27T01:56:26.258 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-27T01:56:26.258 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-27T01:56:26.258 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-27T01:56:26.258 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-27T01:56:26.258 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-27T01:56:26.258 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-27T01:56:26.258 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-27T01:56:26.258 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-27T01:56:26.258 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-27T01:56:26.258 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-27T01:56:26.258 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-27T01:56:26.258 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-27T01:56:26.258 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-27T01:56:26.258 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-27T01:56:26.259 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-27T01:56:26.259 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-27T01:56:26.259 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-27T01:56:26.259 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-27T01:56:26.259 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-04-27T01:56:26.259 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-27T01:56:26.259 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-27T01:56:26.259 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-27T01:56:26.259 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-27T01:56:26.259 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-27T01:56:26.259 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-27T01:56:26.259 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-27T01:56:26.259 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-27T01:56:26.259 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-27T01:56:26.259 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-27T01:56:26.259 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-27T01:56:26.260 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-27T01:56:26.260 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-27T01:56:26.260 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-27T01:56:26.260 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-27T01:56:26.260 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-27T01:56:26.260 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-27T01:56:26.260 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-27T01:56:26.261 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-27T01:56:26.261 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-27T01:56:26.277 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 4c587076-0439-11ef-bc93-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-27T01:56:26.277 DEBUG:teuthology.orchestra.run.smithi029:mon.smithi029> sudo journalctl -f -n 0 -u ceph-4c587076-0439-11ef-bc93-c7b262605968@mon.smithi029.service 2024-04-27T01:56:26.322 INFO:tasks.cephadm:Bootstrapping... 2024-04-27T01:56:26.322 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 -v bootstrap --fsid 4c587076-0439-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.29 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-27T01:56:26.482 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:56:26.482 INFO:teuthology.orchestra.run.smithi029.stdout:cephadm ['--image', 'quay.io/ceph/ceph:v18.2.0', '-v', 'bootstrap', '--fsid', '4c587076-0439-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.29', '--skip-admin-label'] 2024-04-27T01:56:26.508 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-27T01:56:26.508 INFO:teuthology.orchestra.run.smithi029.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-27T01:56:26.508 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying podman|docker is present... 2024-04-27T01:56:26.533 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-27T01:56:26.533 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying lvm2 is present... 2024-04-27T01:56:26.533 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying time synchronization is in place... 2024-04-27T01:56:26.540 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-27T01:56:26.540 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-27T01:56:26.547 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-27T01:56:26.547 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-04-27T01:56:26.553 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout enabled 2024-04-27T01:56:26.559 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout active 2024-04-27T01:56:26.559 INFO:teuthology.orchestra.run.smithi029.stdout:Unit chronyd.service is enabled and running 2024-04-27T01:56:26.559 INFO:teuthology.orchestra.run.smithi029.stdout:Repeating the final host check... 2024-04-27T01:56:26.583 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-27T01:56:26.583 INFO:teuthology.orchestra.run.smithi029.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-27T01:56:26.583 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl is present 2024-04-27T01:56:26.583 INFO:teuthology.orchestra.run.smithi029.stdout:lvcreate is present 2024-04-27T01:56:26.589 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-27T01:56:26.590 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-27T01:56:26.595 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-27T01:56:26.596 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-04-27T01:56:26.602 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout enabled 2024-04-27T01:56:26.608 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout active 2024-04-27T01:56:26.608 INFO:teuthology.orchestra.run.smithi029.stdout:Unit chronyd.service is enabled and running 2024-04-27T01:56:26.608 INFO:teuthology.orchestra.run.smithi029.stdout:Host looks OK 2024-04-27T01:56:26.608 INFO:teuthology.orchestra.run.smithi029.stdout:Cluster fsid: 4c587076-0439-11ef-bc93-c7b262605968 2024-04-27T01:56:26.609 INFO:teuthology.orchestra.run.smithi029.stdout:Acquiring lock 139814898533376 on /run/cephadm/4c587076-0439-11ef-bc93-c7b262605968.lock 2024-04-27T01:56:26.609 INFO:teuthology.orchestra.run.smithi029.stdout:Lock 139814898533376 acquired on /run/cephadm/4c587076-0439-11ef-bc93-c7b262605968.lock 2024-04-27T01:56:26.609 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying IP 172.21.15.29 port 3300 ... 2024-04-27T01:56:26.610 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying IP 172.21.15.29 port 6789 ... 2024-04-27T01:56:26.610 INFO:teuthology.orchestra.run.smithi029.stdout:Base mon IP(s) is [172.21.15.29:3300, 172.21.15.29:6789], mon addrv is [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-04-27T01:56:26.613 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.29 metric 100 2024-04-27T01:56:26.613 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.29 metric 100 2024-04-27T01:56:26.615 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-27T01:56:26.616 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-27T01:56:26.616 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-27T01:56:26.618 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-27T01:56:26.618 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-27T01:56:26.618 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-27T01:56:26.618 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-27T01:56:26.618 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2024-04-27T01:56:26.618 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-27T01:56:26.619 INFO:teuthology.orchestra.run.smithi029.stdout:Mon IP `172.21.15.29` is in CIDR network `172.21.0.0/20` 2024-04-27T01:56:26.619 INFO:teuthology.orchestra.run.smithi029.stdout:Mon IP `172.21.15.29` is in CIDR network `172.21.0.0/20` 2024-04-27T01:56:26.619 INFO:teuthology.orchestra.run.smithi029.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-27T01:56:26.619 INFO:teuthology.orchestra.run.smithi029.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-27T01:56:26.620 INFO:teuthology.orchestra.run.smithi029.stdout:Pulling container image quay.io/ceph/ceph:v18.2.0... 2024-04-27T01:56:27.416 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 2024-04-27T01:56:27.416 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Trying to pull quay.io/ceph/ceph:v18.2.0... 2024-04-27T01:56:27.416 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Getting image source signatures 2024-04-27T01:56:27.416 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying blob sha256:46af8f5390d4e94fc57efb422ccb97bb53dfe5b948546bfc191b46557eb2dbd9 2024-04-27T01:56:27.416 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying blob sha256:3bd20aeff60302f668275dc2005d10679ae56492967a3a5a54fd3dde85333aec 2024-04-27T01:56:27.416 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying config sha256:dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 2024-04-27T01:56:27.416 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-27T01:56:28.407 INFO:teuthology.orchestra.run.smithi029.stdout:ceph: stdout ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) 2024-04-27T01:56:28.407 INFO:teuthology.orchestra.run.smithi029.stdout:Ceph version: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) 2024-04-27T01:56:28.407 INFO:teuthology.orchestra.run.smithi029.stdout:Extracting ceph user uid/gid from container image... 2024-04-27T01:56:29.327 INFO:teuthology.orchestra.run.smithi029.stdout:stat: stdout 167 167 2024-04-27T01:56:29.327 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial keys... 2024-04-27T01:56:30.214 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQBNWyxm476xJxAAa8ZaxgDn17pu5Hz+xrBmDw== 2024-04-27T01:56:34.018 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQBOWyxmHjOcJRAA4THi9R1XJfx13fAQkROppw== 2024-04-27T01:56:48.268 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQBWWyxmMFb6KxAAAuxEQYg6FS+qsvO8rbJudQ== 2024-04-27T01:56:48.269 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial monmap... 2024-04-27T01:56:50.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-27T01:56:50.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-04-27T01:56:50.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 4c587076-0439-11ef-bc93-c7b262605968 2024-04-27T01:56:50.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-27T01:56:50.306 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool for smithi029 [v2:172.21.15.29:3300,v1:172.21.15.29:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-27T01:56:50.307 INFO:teuthology.orchestra.run.smithi029.stdout:setting min_mon_release = pacific 2024-04-27T01:56:50.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: set fsid to 4c587076-0439-11ef-bc93-c7b262605968 2024-04-27T01:56:50.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-27T01:56:50.307 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:56:50.307 INFO:teuthology.orchestra.run.smithi029.stdout:Creating mon... 2024-04-27T01:56:51.303 INFO:teuthology.orchestra.run.smithi029.stdout:create mon.smithi029 on 2024-04-27T01:56:51.518 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-27T01:56:51.659 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-27T01:56:51.837 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-4c587076-0439-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-4c587076-0439-11ef-bc93-c7b262605968.target. 2024-04-27T01:56:51.837 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-4c587076-0439-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-4c587076-0439-11ef-bc93-c7b262605968.target. 2024-04-27T01:56:52.072 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-4c587076-0439-11ef-bc93-c7b262605968@mon.smithi029 2024-04-27T01:56:52.072 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to reset failed state of unit ceph-4c587076-0439-11ef-bc93-c7b262605968@mon.smithi029.service: Unit ceph-4c587076-0439-11ef-bc93-c7b262605968@mon.smithi029.service not loaded. 2024-04-27T01:56:52.250 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-4c587076-0439-11ef-bc93-c7b262605968.target.wants/ceph-4c587076-0439-11ef-bc93-c7b262605968@mon.smithi029.service → /etc/systemd/system/ceph-4c587076-0439-11ef-bc93-c7b262605968@.service. 2024-04-27T01:56:52.613 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:56:52 smithi029 podman[26369]: 2024-04-27 01:56:52.433815431 +0000 UTC m=+0.020836809 image pull dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 quay.io/ceph/ceph:v18.2.0 2024-04-27T01:56:52.785 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-27T01:56:52.786 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-04-27T01:56:52.794 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-27T01:56:52.794 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-04-27T01:56:52.794 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-04-27T01:56:52.794 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to enable service . firewalld.service is not available 2024-04-27T01:56:52.794 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mon to start... 2024-04-27T01:56:52.794 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mon... 2024-04-27T01:56:52.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:56:52 smithi029 podman[26369]: 2024-04-27 01:56:52.714533757 +0000 UTC m=+0.301555127 container init 5492b9a535d0ed8742771291c4a5caccb1aeec05aec8ca8d7dd837a664c481c6 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-27T01:56:52.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:56:52 smithi029 podman[26369]: 2024-04-27 01:56:52.717978887 +0000 UTC m=+0.305000247 container start 5492b9a535d0ed8742771291c4a5caccb1aeec05aec8ca8d7dd837a664c481c6 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029, RELEASE=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=-18.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-27T01:56:52.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:56:52 smithi029 ceph-mon[26396]: mkfs 4c587076-0439-11ef-bc93-c7b262605968 2024-04-27T01:56:52.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:56:52 smithi029 ceph-mon[26396]: mon.smithi029 is new leader, mons smithi029 in quorum (ranks 0) 2024-04-27T01:56:52.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:56:52 smithi029 bash[26369]: 5492b9a535d0ed8742771291c4a5caccb1aeec05aec8ca8d7dd837a664c481c6 2024-04-27T01:56:52.923 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:56:52 smithi029 systemd[1]: Started Ceph mon.smithi029 for 4c587076-0439-11ef-bc93-c7b262605968. 2024-04-27T01:56:54.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:56:53 smithi029 ceph-mon[26396]: mon.smithi029 is new leader, mons smithi029 in quorum (ranks 0) 2024-04-27T01:56:54.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:56:53 smithi029 ceph-mon[26396]: monmap e1: 1 mons at {smithi029=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0]} removed_ranks: {} 2024-04-27T01:56:54.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:56:53 smithi029 ceph-mon[26396]: fsmap 2024-04-27T01:56:54.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:56:53 smithi029 ceph-mon[26396]: osdmap e1: 0 total, 0 up, 0 in 2024-04-27T01:56:54.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:56:53 smithi029 ceph-mon[26396]: mgrmap e1: no daemons active 2024-04-27T01:56:54.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:56:53 smithi029 ceph-mon[26396]: from='client.? 172.21.15.29:0/2186657179' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-27T01:56:56.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout cluster: 2024-04-27T01:56:56.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout id: 4c587076-0439-11ef-bc93-c7b262605968 2024-04-27T01:56:56.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-27T01:56:56.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-04-27T01:56:56.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout services: 2024-04-27T01:56:56.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi029 (age 0.595992s) 2024-04-27T01:56:56.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-27T01:56:56.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-27T01:56:56.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-04-27T01:56:56.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout data: 2024-04-27T01:56:56.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-27T01:56:56.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-27T01:56:56.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-27T01:56:56.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout pgs: 2024-04-27T01:56:56.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-04-27T01:56:56.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.271+0000 7fb02432b700 1 Processor -- start 2024-04-27T01:56:56.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.271+0000 7fb02432b700 1 -- start start 2024-04-27T01:56:56.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.271+0000 7fb02432b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb01c104a00 0x7fb01c104e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:56:56.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.271+0000 7fb02432b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c105350 con 0x7fb01c104a00 2024-04-27T01:56:56.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.272+0000 7fb0220c7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb01c104a00 0x7fb01c104e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:56:56.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.272+0000 7fb0220c7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb01c104a00 0x7fb01c104e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46462/0 (socket says 172.21.15.29:46462) 2024-04-27T01:56:56.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.272+0000 7fb0220c7700 1 -- 172.21.15.29:0/2172853649 learned_addr learned my addr 172.21.15.29:0/2172853649 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:56:56.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.272+0000 7fb0220c7700 1 -- 172.21.15.29:0/2172853649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01c105490 con 0x7fb01c104a00 2024-04-27T01:56:56.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.272+0000 7fb0220c7700 1 --2- 172.21.15.29:0/2172853649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb01c104a00 0x7fb01c104e10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb00c00d180 tx=0x7fb00c00d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33ac768030bf9d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:56:56.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.273+0000 7fb0210c5700 1 -- 172.21.15.29:0/2172853649 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb00c010070 con 0x7fb01c104a00 2024-04-27T01:56:56.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.273+0000 7fb0210c5700 1 -- 172.21.15.29:0/2172853649 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb00c00dcf0 con 0x7fb01c104a00 2024-04-27T01:56:56.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.273+0000 7fb0210c5700 1 -- 172.21.15.29:0/2172853649 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb00c00b0d0 con 0x7fb01c104a00 2024-04-27T01:56:56.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.273+0000 7fb02432b700 1 -- 172.21.15.29:0/2172853649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb01c104a00 msgr2=0x7fb01c104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:56:56.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.273+0000 7fb02432b700 1 --2- 172.21.15.29:0/2172853649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb01c104a00 0x7fb01c104e10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb00c00d180 tx=0x7fb00c00d490 comp rx=0 tx=0).stop 2024-04-27T01:56:56.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.273+0000 7fb02432b700 1 -- 172.21.15.29:0/2172853649 shutdown_connections 2024-04-27T01:56:56.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.273+0000 7fb02432b700 1 --2- 172.21.15.29:0/2172853649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb01c104a00 0x7fb01c104e10 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:56:56.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.273+0000 7fb02432b700 1 -- 172.21.15.29:0/2172853649 >> 172.21.15.29:0/2172853649 conn(0x7fb01c100070 msgr2=0x7fb01c102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:56:56.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.273+0000 7fb02432b700 1 -- 172.21.15.29:0/2172853649 shutdown_connections 2024-04-27T01:56:56.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.273+0000 7fb02432b700 1 -- 172.21.15.29:0/2172853649 wait complete. 2024-04-27T01:56:56.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.274+0000 7fb02432b700 1 Processor -- start 2024-04-27T01:56:56.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.274+0000 7fb02432b700 1 -- start start 2024-04-27T01:56:56.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.274+0000 7fb02432b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb01c104a00 0x7fb01c196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:56:56.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.274+0000 7fb02432b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c197390 con 0x7fb01c104a00 2024-04-27T01:56:56.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.274+0000 7fb0220c7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb01c104a00 0x7fb01c196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:56:56.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.274+0000 7fb0220c7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb01c104a00 0x7fb01c196e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46468/0 (socket says 172.21.15.29:46468) 2024-04-27T01:56:56.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.274+0000 7fb0220c7700 1 -- 172.21.15.29:0/2186657179 learned_addr learned my addr 172.21.15.29:0/2186657179 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:56:56.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.275+0000 7fb0220c7700 1 -- 172.21.15.29:0/2186657179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00c00bd40 con 0x7fb01c104a00 2024-04-27T01:56:56.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.275+0000 7fb0220c7700 1 --2- 172.21.15.29:0/2186657179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb01c104a00 0x7fb01c196e50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb00c017040 tx=0x7fb00c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:56:56.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.275+0000 7fb01b7fe700 1 -- 172.21.15.29:0/2186657179 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb00c00b520 con 0x7fb01c104a00 2024-04-27T01:56:56.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.275+0000 7fb01b7fe700 1 -- 172.21.15.29:0/2186657179 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb00c009810 con 0x7fb01c104a00 2024-04-27T01:56:56.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.275+0000 7fb02432b700 1 -- 172.21.15.29:0/2186657179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb01c197590 con 0x7fb01c104a00 2024-04-27T01:56:56.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.275+0000 7fb01b7fe700 1 -- 172.21.15.29:0/2186657179 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb00c009b10 con 0x7fb01c104a00 2024-04-27T01:56:56.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.275+0000 7fb02432b700 1 -- 172.21.15.29:0/2186657179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb01c1979b0 con 0x7fb01c104a00 2024-04-27T01:56:56.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.275+0000 7fb01b7fe700 1 -- 172.21.15.29:0/2186657179 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb00c01b050 con 0x7fb01c104a00 2024-04-27T01:56:56.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.275+0000 7fb01b7fe700 1 -- 172.21.15.29:0/2186657179 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb00c0298d0 con 0x7fb01c104a00 2024-04-27T01:56:56.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.276+0000 7fb02432b700 1 -- 172.21.15.29:0/2186657179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb01c190660 con 0x7fb01c104a00 2024-04-27T01:56:56.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.279+0000 7fb01b7fe700 1 -- 172.21.15.29:0/2186657179 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fb00c02e060 con 0x7fb01c104a00 2024-04-27T01:56:56.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.352+0000 7fb02432b700 1 -- 172.21.15.29:0/2186657179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fb01c02cde0 con 0x7fb01c104a00 2024-04-27T01:56:56.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.353+0000 7fb01b7fe700 1 -- 172.21.15.29:0/2186657179 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fb00c024070 con 0x7fb01c104a00 2024-04-27T01:56:56.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.354+0000 7fb02432b700 1 -- 172.21.15.29:0/2186657179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb01c104a00 msgr2=0x7fb01c196e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:56:56.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.354+0000 7fb02432b700 1 --2- 172.21.15.29:0/2186657179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb01c104a00 0x7fb01c196e50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb00c017040 tx=0x7fb00c004ad0 comp rx=0 tx=0).stop 2024-04-27T01:56:56.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.354+0000 7fb02432b700 1 -- 172.21.15.29:0/2186657179 shutdown_connections 2024-04-27T01:56:56.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.354+0000 7fb02432b700 1 --2- 172.21.15.29:0/2186657179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb01c104a00 0x7fb01c196e50 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:56:56.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.354+0000 7fb02432b700 1 -- 172.21.15.29:0/2186657179 >> 172.21.15.29:0/2186657179 conn(0x7fb01c100070 msgr2=0x7fb01c102200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:56:56.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.354+0000 7fb02432b700 1 -- 172.21.15.29:0/2186657179 shutdown_connections 2024-04-27T01:56:56.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:53.354+0000 7fb02432b700 1 -- 172.21.15.29:0/2186657179 wait complete. 2024-04-27T01:56:56.824 INFO:teuthology.orchestra.run.smithi029.stdout:mon is available 2024-04-27T01:56:56.824 INFO:teuthology.orchestra.run.smithi029.stdout:Assimilating anything we can from ceph.conf... 2024-04-27T01:56:57.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:56:57 smithi029 ceph-mon[26396]: from='client.? 172.21.15.29:0/2725867476' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-27T01:56:57.702 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:56:57 smithi029 ceph-mon[26396]: from='client.? 172.21.15.29:0/2725867476' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-27T01:56:58.003 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-04-27T01:56:58.003 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [global] 2024-04-27T01:56:58.003 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout fsid = 4c587076-0439-11ef-bc93-c7b262605968 2024-04-27T01:56:58.003 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-04-27T01:56:58.003 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-27T01:56:58.003 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-27T01:56:58.003 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-27T01:56:58.003 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-27T01:56:58.003 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-04-27T01:56:58.003 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-27T01:56:58.003 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-27T01:56:58.003 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-04-27T01:56:58.004 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [osd] 2024-04-27T01:56:58.004 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-27T01:56:58.004 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-27T01:56:58.004 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-27T01:56:58.004 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-27T01:56:58.004 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-27T01:56:58.004 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.301+0000 7f7258c5c700 1 Processor -- start 2024-04-27T01:56:58.004 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.301+0000 7f7258c5c700 1 -- start start 2024-04-27T01:56:58.004 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.301+0000 7f7258c5c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7254074490 0x7f725410aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:56:58.004 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.301+0000 7f7258c5c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7254106490 con 0x7f7254074490 2024-04-27T01:56:58.004 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.302+0000 7f725259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7254074490 0x7f725410aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:56:58.004 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.302+0000 7f725259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7254074490 0x7f725410aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33382/0 (socket says 172.21.15.29:33382) 2024-04-27T01:56:58.004 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.302+0000 7f725259c700 1 -- 172.21.15.29:0/2091625669 learned_addr learned my addr 172.21.15.29:0/2091625669 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:56:58.005 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.302+0000 7f725259c700 1 -- 172.21.15.29:0/2091625669 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725410afe0 con 0x7f7254074490 2024-04-27T01:56:58.005 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.302+0000 7f725259c700 1 --2- 172.21.15.29:0/2091625669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7254074490 0x7f725410aaa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f72440098a0 tx=0x7f7244009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b9895d02ec56e24 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:56:58.005 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.302+0000 7f725159a700 1 -- 172.21.15.29:0/2091625669 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7244004180 con 0x7f7254074490 2024-04-27T01:56:58.005 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.302+0000 7f725159a700 1 -- 172.21.15.29:0/2091625669 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72440042e0 con 0x7f7254074490 2024-04-27T01:56:58.005 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.302+0000 7f725159a700 1 -- 172.21.15.29:0/2091625669 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72440036a0 con 0x7f7254074490 2024-04-27T01:56:58.005 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.303+0000 7f7258c5c700 1 -- 172.21.15.29:0/2091625669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7254074490 msgr2=0x7f725410aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:56:58.005 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.303+0000 7f7258c5c700 1 --2- 172.21.15.29:0/2091625669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7254074490 0x7f725410aaa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f72440098a0 tx=0x7f7244009bb0 comp rx=0 tx=0).stop 2024-04-27T01:56:58.005 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.303+0000 7f7258c5c700 1 -- 172.21.15.29:0/2091625669 shutdown_connections 2024-04-27T01:56:58.005 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.303+0000 7f7258c5c700 1 --2- 172.21.15.29:0/2091625669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7254074490 0x7f725410aaa0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:56:58.005 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.303+0000 7f7258c5c700 1 -- 172.21.15.29:0/2091625669 >> 172.21.15.29:0/2091625669 conn(0x7f7254100070 msgr2=0x7f7254102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:56:58.005 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.303+0000 7f7258c5c700 1 -- 172.21.15.29:0/2091625669 shutdown_connections 2024-04-27T01:56:58.005 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.303+0000 7f7258c5c700 1 -- 172.21.15.29:0/2091625669 wait complete. 2024-04-27T01:56:58.005 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.304+0000 7f7258c5c700 1 Processor -- start 2024-04-27T01:56:58.005 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.304+0000 7f7258c5c700 1 -- start start 2024-04-27T01:56:58.006 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.304+0000 7f7258c5c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f725419b200 0x7f725419b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:56:58.006 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.304+0000 7f7258c5c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725419bb50 con 0x7f725419b200 2024-04-27T01:56:58.006 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.304+0000 7f725259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f725419b200 0x7f725419b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:56:58.006 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.304+0000 7f725259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f725419b200 0x7f725419b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33386/0 (socket says 172.21.15.29:33386) 2024-04-27T01:56:58.006 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.304+0000 7f725259c700 1 -- 172.21.15.29:0/2725867476 learned_addr learned my addr 172.21.15.29:0/2725867476 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:56:58.006 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.304+0000 7f725259c700 1 -- 172.21.15.29:0/2725867476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7244009550 con 0x7f725419b200 2024-04-27T01:56:58.006 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.305+0000 7f725259c700 1 --2- 172.21.15.29:0/2725867476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f725419b200 0x7f725419b610 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f7244004000 tx=0x7f7244004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:56:58.006 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.305+0000 7f72437fe700 1 -- 172.21.15.29:0/2725867476 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7244003ce0 con 0x7f725419b200 2024-04-27T01:56:58.006 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.305+0000 7f72437fe700 1 -- 172.21.15.29:0/2725867476 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7244020070 con 0x7f725419b200 2024-04-27T01:56:58.006 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.305+0000 7f7258c5c700 1 -- 172.21.15.29:0/2725867476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f725419bd50 con 0x7f725419b200 2024-04-27T01:56:58.006 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.305+0000 7f72437fe700 1 -- 172.21.15.29:0/2725867476 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7244011630 con 0x7f725419b200 2024-04-27T01:56:58.006 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.305+0000 7f7258c5c700 1 -- 172.21.15.29:0/2725867476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f725419e9b0 con 0x7f725419b200 2024-04-27T01:56:58.006 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.305+0000 7f72437fe700 1 -- 172.21.15.29:0/2725867476 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7244022030 con 0x7f725419b200 2024-04-27T01:56:58.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.305+0000 7f72437fe700 1 -- 172.21.15.29:0/2725867476 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7244025d00 con 0x7f725419b200 2024-04-27T01:56:58.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.306+0000 7f7258c5c700 1 -- 172.21.15.29:0/2725867476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7254194ff0 con 0x7f725419b200 2024-04-27T01:56:58.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.308+0000 7f72437fe700 1 -- 172.21.15.29:0/2725867476 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f7244025480 con 0x7f725419b200 2024-04-27T01:56:58.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.379+0000 7f7258c5c700 1 -- 172.21.15.29:0/2725867476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f7254062380 con 0x7f725419b200 2024-04-27T01:56:58.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.384+0000 7f72437fe700 1 -- 172.21.15.29:0/2725867476 <== mon.0 v2:172.21.15.29:3300/0 7 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7244011c20 con 0x7f725419b200 2024-04-27T01:56:58.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.384+0000 7f72437fe700 1 -- 172.21.15.29:0/2725867476 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f724401c500 con 0x7f725419b200 2024-04-27T01:56:58.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.385+0000 7f7258c5c700 1 -- 172.21.15.29:0/2725867476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f725419b200 msgr2=0x7f725419b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:56:58.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.385+0000 7f7258c5c700 1 --2- 172.21.15.29:0/2725867476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f725419b200 0x7f725419b610 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f7244004000 tx=0x7f7244004750 comp rx=0 tx=0).stop 2024-04-27T01:56:58.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.385+0000 7f7258c5c700 1 -- 172.21.15.29:0/2725867476 shutdown_connections 2024-04-27T01:56:58.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.385+0000 7f7258c5c700 1 --2- 172.21.15.29:0/2725867476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f725419b200 0x7f725419b610 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:56:58.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.385+0000 7f7258c5c700 1 -- 172.21.15.29:0/2725867476 >> 172.21.15.29:0/2725867476 conn(0x7f7254100070 msgr2=0x7f72541019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:56:58.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.385+0000 7f7258c5c700 1 -- 172.21.15.29:0/2725867476 shutdown_connections 2024-04-27T01:56:58.007 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:57.385+0000 7f7258c5c700 1 -- 172.21.15.29:0/2725867476 wait complete. 2024-04-27T01:56:58.007 INFO:teuthology.orchestra.run.smithi029.stdout:Generating new minimal ceph.conf... 2024-04-27T01:56:58.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:56:58 smithi029 ceph-mon[26396]: from='client.? 172.21.15.29:0/321302039' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:56:59.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.479+0000 7f2acf5cd700 1 Processor -- start 2024-04-27T01:56:59.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.480+0000 7f2acf5cd700 1 -- start start 2024-04-27T01:56:59.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.480+0000 7f2acf5cd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac8104a10 0x7f2ac8104e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:56:59.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.480+0000 7f2acf5cd700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac8105360 con 0x7f2ac8104a10 2024-04-27T01:56:59.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.480+0000 7f2acd369700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac8104a10 0x7f2ac8104e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:56:59.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.480+0000 7f2acd369700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac8104a10 0x7f2ac8104e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33388/0 (socket says 172.21.15.29:33388) 2024-04-27T01:56:59.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.480+0000 7f2acd369700 1 -- 172.21.15.29:0/988565290 learned_addr learned my addr 172.21.15.29:0/988565290 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:56:59.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.480+0000 7f2acd369700 1 -- 172.21.15.29:0/988565290 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac81054a0 con 0x7f2ac8104a10 2024-04-27T01:56:59.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.480+0000 7f2acd369700 1 --2- 172.21.15.29:0/988565290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac8104a10 0x7f2ac8104e20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2ac40098a0 tx=0x7f2ac4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb83f7b91bed0a31 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:56:59.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.481+0000 7f2abbfff700 1 -- 172.21.15.29:0/988565290 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ac4004180 con 0x7f2ac8104a10 2024-04-27T01:56:59.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.481+0000 7f2abbfff700 1 -- 172.21.15.29:0/988565290 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2ac40042e0 con 0x7f2ac8104a10 2024-04-27T01:56:59.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.481+0000 7f2abbfff700 1 -- 172.21.15.29:0/988565290 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ac4011a50 con 0x7f2ac8104a10 2024-04-27T01:56:59.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.481+0000 7f2acf5cd700 1 -- 172.21.15.29:0/988565290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac8104a10 msgr2=0x7f2ac8104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:56:59.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.481+0000 7f2acf5cd700 1 --2- 172.21.15.29:0/988565290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac8104a10 0x7f2ac8104e20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2ac40098a0 tx=0x7f2ac4009bb0 comp rx=0 tx=0).stop 2024-04-27T01:56:59.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.481+0000 7f2acf5cd700 1 -- 172.21.15.29:0/988565290 shutdown_connections 2024-04-27T01:56:59.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.481+0000 7f2acf5cd700 1 --2- 172.21.15.29:0/988565290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac8104a10 0x7f2ac8104e20 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:56:59.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.481+0000 7f2acf5cd700 1 -- 172.21.15.29:0/988565290 >> 172.21.15.29:0/988565290 conn(0x7f2ac8100060 msgr2=0x7f2ac8102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:56:59.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.481+0000 7f2acf5cd700 1 -- 172.21.15.29:0/988565290 shutdown_connections 2024-04-27T01:56:59.127 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.481+0000 7f2acf5cd700 1 -- 172.21.15.29:0/988565290 wait complete. 2024-04-27T01:56:59.127 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.482+0000 7f2acf5cd700 1 Processor -- start 2024-04-27T01:56:59.127 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.482+0000 7f2acf5cd700 1 -- start start 2024-04-27T01:56:59.127 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.482+0000 7f2acf5cd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac8104a10 0x7f2ac818fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:56:59.127 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.482+0000 7f2acf5cd700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac81900f0 con 0x7f2ac8104a10 2024-04-27T01:56:59.127 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.482+0000 7f2acd369700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac8104a10 0x7f2ac818fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:56:59.127 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.482+0000 7f2acd369700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac8104a10 0x7f2ac818fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33402/0 (socket says 172.21.15.29:33402) 2024-04-27T01:56:59.127 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.482+0000 7f2acd369700 1 -- 172.21.15.29:0/321302039 learned_addr learned my addr 172.21.15.29:0/321302039 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:56:59.127 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.482+0000 7f2acd369700 1 -- 172.21.15.29:0/321302039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac4009550 con 0x7f2ac8104a10 2024-04-27T01:56:59.127 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.483+0000 7f2acd369700 1 --2- 172.21.15.29:0/321302039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac8104a10 0x7f2ac818fbb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4000c00 tx=0x7f2ac4011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:56:59.127 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.483+0000 7f2aba7fc700 1 -- 172.21.15.29:0/321302039 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ac4003b90 con 0x7f2ac8104a10 2024-04-27T01:56:59.127 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.483+0000 7f2aba7fc700 1 -- 172.21.15.29:0/321302039 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2ac40248b0 con 0x7f2ac8104a10 2024-04-27T01:56:59.127 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.483+0000 7f2acf5cd700 1 -- 172.21.15.29:0/321302039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac818e200 con 0x7f2ac8104a10 2024-04-27T01:56:59.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.483+0000 7f2aba7fc700 1 -- 172.21.15.29:0/321302039 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ac401a8e0 con 0x7f2ac8104a10 2024-04-27T01:56:59.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.483+0000 7f2acf5cd700 1 -- 172.21.15.29:0/321302039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac818e620 con 0x7f2ac8104a10 2024-04-27T01:56:59.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.483+0000 7f2aba7fc700 1 -- 172.21.15.29:0/321302039 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2ac402d3f0 con 0x7f2ac8104a10 2024-04-27T01:56:59.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.483+0000 7f2aba7fc700 1 -- 172.21.15.29:0/321302039 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2ac4019450 con 0x7f2ac8104a10 2024-04-27T01:56:59.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.484+0000 7f2acf5cd700 1 -- 172.21.15.29:0/321302039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac8062380 con 0x7f2ac8104a10 2024-04-27T01:56:59.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.486+0000 7f2aba7fc700 1 -- 172.21.15.29:0/321302039 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f2ac4024db0 con 0x7f2ac8104a10 2024-04-27T01:56:59.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.558+0000 7f2acf5cd700 1 -- 172.21.15.29:0/321302039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f2ac818f0b0 con 0x7f2ac8104a10 2024-04-27T01:56:59.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.559+0000 7f2aba7fc700 1 -- 172.21.15.29:0/321302039 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f2ac4024ad0 con 0x7f2ac8104a10 2024-04-27T01:56:59.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.559+0000 7f2acf5cd700 1 -- 172.21.15.29:0/321302039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac8104a10 msgr2=0x7f2ac818fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:56:59.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.559+0000 7f2acf5cd700 1 --2- 172.21.15.29:0/321302039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac8104a10 0x7f2ac818fbb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4000c00 tx=0x7f2ac4011ab0 comp rx=0 tx=0).stop 2024-04-27T01:56:59.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.560+0000 7f2acf5cd700 1 -- 172.21.15.29:0/321302039 shutdown_connections 2024-04-27T01:56:59.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.560+0000 7f2acf5cd700 1 --2- 172.21.15.29:0/321302039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ac8104a10 0x7f2ac818fbb0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:56:59.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.560+0000 7f2acf5cd700 1 -- 172.21.15.29:0/321302039 >> 172.21.15.29:0/321302039 conn(0x7f2ac8100060 msgr2=0x7f2ac8079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:56:59.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.560+0000 7f2acf5cd700 1 -- 172.21.15.29:0/321302039 shutdown_connections 2024-04-27T01:56:59.129 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:56:58.560+0000 7f2acf5cd700 1 -- 172.21.15.29:0/321302039 wait complete. 2024-04-27T01:56:59.129 INFO:teuthology.orchestra.run.smithi029.stdout:Restarting the monitor... 2024-04-27T01:56:59.398 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:56:59 smithi029 systemd[1]: Stopping Ceph mon.smithi029 for 4c587076-0439-11ef-bc93-c7b262605968... 2024-04-27T01:56:59.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26392]: 2024-04-27T01:56:59.415+0000 7f38b65dc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi029 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:56:59.702 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26392]: 2024-04-27T01:56:59.415+0000 7f38b65dc700 -1 mon.smithi029@0(leader) e1 *** Got Signal Terminated *** 2024-04-27T01:56:59.702 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:56:59 smithi029 podman[26644]: 2024-04-27 01:56:59.529064254 +0000 UTC m=+0.214949119 container died 5492b9a535d0ed8742771291c4a5caccb1aeec05aec8ca8d7dd837a664c481c6 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-27T01:57:00.117 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:56:59 smithi029 podman[26644]: 2024-04-27 01:56:59.875649212 +0000 UTC m=+0.561534079 container cleanup 5492b9a535d0ed8742771291c4a5caccb1aeec05aec8ca8d7dd837a664c481c6 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029, org.label-schema.license=GPLv2, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.0, ceph=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=HEAD) 2024-04-27T01:57:00.117 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:56:59 smithi029 bash[26644]: ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029 2024-04-27T01:57:00.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 podman[26658]: 2024-04-27 01:57:00.135957694 +0000 UTC m=+0.604884088 container remove 5492b9a535d0ed8742771291c4a5caccb1aeec05aec8ca8d7dd837a664c481c6 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, ceph=True, CEPH_POINT_RELEASE=-18.2.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-27T01:57:00.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 systemd[1]: ceph-4c587076-0439-11ef-bc93-c7b262605968@mon.smithi029.service: Deactivated successfully. 2024-04-27T01:57:00.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 systemd[1]: Stopped Ceph mon.smithi029 for 4c587076-0439-11ef-bc93-c7b262605968. 2024-04-27T01:57:00.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 systemd[1]: Starting Ceph mon.smithi029 for 4c587076-0439-11ef-bc93-c7b262605968... 2024-04-27T01:57:00.749 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 podman[26750]: 2024-04-27 01:57:00.421199999 +0000 UTC m=+0.018295157 image pull dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 quay.io/ceph/ceph:v18.2.0 2024-04-27T01:57:00.750 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 podman[26750]: 2024-04-27 01:57:00.531046835 +0000 UTC m=+0.128142003 container create 33130493e80ded41989df919b6ae566012e0ee93d10a59b55f00bb543ccc9c59 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20231212, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-27T01:57:00.750 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 podman[26750]: 2024-04-27 01:57:00.734742836 +0000 UTC m=+0.331837962 container init 33130493e80ded41989df919b6ae566012e0ee93d10a59b55f00bb543ccc9c59 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.0) 2024-04-27T01:57:00.750 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 podman[26750]: 2024-04-27 01:57:00.737625618 +0000 UTC m=+0.334720750 container start 33130493e80ded41989df919b6ae566012e0ee93d10a59b55f00bb543ccc9c59 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212, RELEASE=HEAD, ceph=True, GIT_BRANCH=HEAD) 2024-04-27T01:57:00.750 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: set uid:gid to 167:167 (ceph:ceph) 2024-04-27T01:57:00.750 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable), process ceph-mon, pid 2 2024-04-27T01:57:00.750 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: pidfile_write: ignore empty --pid-file 2024-04-27T01:57:00.750 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: load: jerasure load: lrc 2024-04-27T01:57:00.750 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: RocksDB version: 7.9.2 2024-04-27T01:57:00.750 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Git sha 0 2024-04-27T01:57:00.750 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Compile date 2023-08-03 19:21:13 2024-04-27T01:57:00.750 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: DB SUMMARY 2024-04-27T01:57:00.751 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: DB Session ID: 02WVA901GTEJOW8FKLQC 2024-04-27T01:57:00.751 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: CURRENT file: CURRENT 2024-04-27T01:57:00.751 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: IDENTITY file: IDENTITY 2024-04-27T01:57:00.751 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-27T01:57:00.751 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi029/store.db dir, Total Num: 1, files: 000008.sst 2024-04-27T01:57:00.751 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi029/store.db: 000009.log size: 98766 ; 2024-04-27T01:57:00.751 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.error_if_exists: 0 2024-04-27T01:57:00.751 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.create_if_missing: 0 2024-04-27T01:57:00.751 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.paranoid_checks: 1 2024-04-27T01:57:00.752 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-27T01:57:00.752 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-27T01:57:00.752 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-27T01:57:00.752 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.env: 0x55d403b39720 2024-04-27T01:57:00.752 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.fs: PosixFileSystem 2024-04-27T01:57:00.752 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.info_log: 0x55d4053cd2c0 2024-04-27T01:57:00.752 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_file_opening_threads: 16 2024-04-27T01:57:00.753 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.statistics: (nil) 2024-04-27T01:57:00.753 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.use_fsync: 0 2024-04-27T01:57:00.753 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_log_file_size: 0 2024-04-27T01:57:00.753 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-27T01:57:00.753 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-27T01:57:00.753 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.keep_log_file_num: 1000 2024-04-27T01:57:00.753 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.recycle_log_file_num: 0 2024-04-27T01:57:00.753 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.allow_fallocate: 1 2024-04-27T01:57:00.753 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.allow_mmap_reads: 0 2024-04-27T01:57:00.753 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.allow_mmap_writes: 0 2024-04-27T01:57:00.753 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.use_direct_reads: 0 2024-04-27T01:57:00.753 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-27T01:57:00.753 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.create_missing_column_families: 0 2024-04-27T01:57:00.754 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.db_log_dir: 2024-04-27T01:57:00.754 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.wal_dir: 2024-04-27T01:57:00.754 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-27T01:57:00.754 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-27T01:57:00.754 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-27T01:57:00.754 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-27T01:57:00.754 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-27T01:57:00.754 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-27T01:57:00.754 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.advise_random_on_open: 1 2024-04-27T01:57:00.754 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.db_write_buffer_size: 0 2024-04-27T01:57:00.754 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.write_buffer_manager: 0x55d40463e3c0 2024-04-27T01:57:00.754 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-27T01:57:00.754 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-27T01:57:00.754 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-27T01:57:00.755 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.rate_limiter: (nil) 2024-04-27T01:57:00.755 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-27T01:57:00.755 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.wal_recovery_mode: 2 2024-04-27T01:57:00.755 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.enable_thread_tracking: 0 2024-04-27T01:57:00.755 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.enable_pipelined_write: 0 2024-04-27T01:57:00.755 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.unordered_write: 0 2024-04-27T01:57:00.755 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-27T01:57:00.755 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-27T01:57:00.755 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-27T01:57:00.755 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-27T01:57:00.755 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.row_cache: None 2024-04-27T01:57:00.755 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.wal_filter: None 2024-04-27T01:57:00.755 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-27T01:57:00.755 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.allow_ingest_behind: 0 2024-04-27T01:57:00.756 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.two_write_queues: 0 2024-04-27T01:57:00.756 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.manual_wal_flush: 0 2024-04-27T01:57:00.756 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.wal_compression: 0 2024-04-27T01:57:00.756 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.atomic_flush: 0 2024-04-27T01:57:00.756 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-27T01:57:00.756 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-27T01:57:00.756 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-27T01:57:00.756 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.log_readahead_size: 0 2024-04-27T01:57:00.756 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-27T01:57:00.756 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.best_efforts_recovery: 0 2024-04-27T01:57:00.756 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-27T01:57:00.756 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-27T01:57:00.756 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.allow_data_in_errors: 0 2024-04-27T01:57:00.756 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.db_host_id: __hostname__ 2024-04-27T01:57:00.757 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-27T01:57:00.757 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_background_jobs: 2 2024-04-27T01:57:00.757 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_background_compactions: -1 2024-04-27T01:57:00.757 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_subcompactions: 1 2024-04-27T01:57:00.757 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-27T01:57:00.757 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-27T01:57:00.757 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-27T01:57:00.757 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_total_wal_size: 0 2024-04-27T01:57:00.757 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-27T01:57:00.757 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-27T01:57:00.757 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-27T01:57:00.757 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-27T01:57:00.757 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_open_files: -1 2024-04-27T01:57:00.757 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.bytes_per_sync: 0 2024-04-27T01:57:00.758 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-27T01:57:00.758 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-27T01:57:00.758 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compaction_readahead_size: 0 2024-04-27T01:57:00.758 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_background_flushes: -1 2024-04-27T01:57:00.758 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Compression algorithms supported: 2024-04-27T01:57:00.758 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-27T01:57:00.758 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: kZSTD supported: 0 2024-04-27T01:57:00.758 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: kXpressCompression supported: 0 2024-04-27T01:57:00.758 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: kLZ4HCCompression supported: 1 2024-04-27T01:57:00.758 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: kZlibCompression supported: 1 2024-04-27T01:57:00.758 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: kSnappyCompression supported: 1 2024-04-27T01:57:00.758 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: kLZ4Compression supported: 1 2024-04-27T01:57:00.758 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: kBZip2Compression supported: 0 2024-04-27T01:57:00.759 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-27T01:57:00.759 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-27T01:57:00.759 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi029/store.db/MANIFEST-000010 2024-04-27T01:57:00.759 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-27T01:57:00.759 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-27T01:57:00.759 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.merge_operator: 2024-04-27T01:57:00.759 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compaction_filter: None 2024-04-27T01:57:00.759 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compaction_filter_factory: None 2024-04-27T01:57:00.759 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.sst_partitioner_factory: None 2024-04-27T01:57:00.759 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-27T01:57:00.759 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-27T01:57:00.759 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d4053cd3c0) 2024-04-27T01:57:00.759 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: cache_index_and_filter_blocks: 1 2024-04-27T01:57:00.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-27T01:57:00.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-27T01:57:00.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: pin_top_level_index_and_filter: 1 2024-04-27T01:57:00.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: index_type: 0 2024-04-27T01:57:00.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: data_block_index_type: 0 2024-04-27T01:57:00.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: index_shortening: 1 2024-04-27T01:57:00.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-27T01:57:00.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: checksum: 4 2024-04-27T01:57:00.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: no_block_cache: 0 2024-04-27T01:57:00.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_cache: 0x55d4046df1f0 2024-04-27T01:57:00.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_cache_name: BinnedLRUCache 2024-04-27T01:57:00.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_cache_options: 2024-04-27T01:57:00.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: capacity : 536870912 2024-04-27T01:57:00.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: num_shard_bits : 4 2024-04-27T01:57:00.761 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: strict_capacity_limit : 0 2024-04-27T01:57:00.761 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: high_pri_pool_ratio: 0.000 2024-04-27T01:57:00.761 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_cache_compressed: (nil) 2024-04-27T01:57:00.761 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: persistent_cache: (nil) 2024-04-27T01:57:00.761 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_size: 4096 2024-04-27T01:57:00.761 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_size_deviation: 10 2024-04-27T01:57:00.761 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_restart_interval: 16 2024-04-27T01:57:00.761 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: index_block_restart_interval: 1 2024-04-27T01:57:00.761 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: metadata_block_size: 4096 2024-04-27T01:57:00.761 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: partition_filters: 0 2024-04-27T01:57:00.761 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: use_delta_encoding: 1 2024-04-27T01:57:00.761 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: filter_policy: bloomfilter 2024-04-27T01:57:00.761 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: whole_key_filtering: 1 2024-04-27T01:57:00.761 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: verify_compression: 0 2024-04-27T01:57:00.762 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: read_amp_bytes_per_bit: 0 2024-04-27T01:57:00.762 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: format_version: 5 2024-04-27T01:57:00.762 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: enable_index_compression: 1 2024-04-27T01:57:00.762 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_align: 0 2024-04-27T01:57:00.762 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: max_auto_readahead_size: 262144 2024-04-27T01:57:00.762 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: prepopulate_block_cache: 0 2024-04-27T01:57:00.762 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: initial_auto_readahead_size: 8192 2024-04-27T01:57:00.762 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: num_file_reads_for_auto_readahead: 2 2024-04-27T01:57:00.762 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.write_buffer_size: 33554432 2024-04-27T01:57:00.762 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_write_buffer_number: 2 2024-04-27T01:57:00.762 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compression: NoCompression 2024-04-27T01:57:00.762 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.bottommost_compression: Disabled 2024-04-27T01:57:00.762 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.prefix_extractor: nullptr 2024-04-27T01:57:00.762 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-27T01:57:00.763 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.num_levels: 7 2024-04-27T01:57:00.763 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-27T01:57:00.763 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-27T01:57:00.763 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-27T01:57:00.763 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-27T01:57:00.763 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-27T01:57:00.763 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-27T01:57:00.763 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-27T01:57:00.763 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-27T01:57:00.763 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-27T01:57:00.763 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-27T01:57:00.763 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-27T01:57:00.763 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-27T01:57:00.763 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-27T01:57:00.764 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compression_opts.level: 32767 2024-04-27T01:57:00.764 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compression_opts.strategy: 0 2024-04-27T01:57:00.764 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-27T01:57:00.764 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-27T01:57:00.764 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-27T01:57:00.764 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-27T01:57:00.764 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compression_opts.enabled: false 2024-04-27T01:57:00.764 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-27T01:57:00.764 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-27T01:57:00.764 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-27T01:57:00.764 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-27T01:57:00.765 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.target_file_size_base: 67108864 2024-04-27T01:57:00.765 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-27T01:57:00.765 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-27T01:57:00.765 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-27T01:57:00.765 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-27T01:57:00.766 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-27T01:57:00.766 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-27T01:57:00.766 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-27T01:57:00.766 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-27T01:57:00.766 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-27T01:57:00.766 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-27T01:57:00.766 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-27T01:57:00.766 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-27T01:57:00.766 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-27T01:57:00.766 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-27T01:57:00.766 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.arena_block_size: 1048576 2024-04-27T01:57:00.766 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-27T01:57:00.766 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-27T01:57:00.766 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.disable_auto_compactions: 0 2024-04-27T01:57:00.767 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-27T01:57:00.767 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-27T01:57:00.767 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-27T01:57:00.767 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-27T01:57:00.767 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-27T01:57:00.767 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-27T01:57:00.767 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-27T01:57:00.767 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-27T01:57:00.767 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-27T01:57:00.767 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-27T01:57:00.767 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.table_properties_collectors: 2024-04-27T01:57:00.767 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.inplace_update_support: 0 2024-04-27T01:57:00.767 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-27T01:57:00.767 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-27T01:57:00.767 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-27T01:57:00.768 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-27T01:57:00.768 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.bloom_locality: 0 2024-04-27T01:57:00.768 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.max_successive_merges: 0 2024-04-27T01:57:00.768 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-27T01:57:00.768 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.paranoid_file_checks: 0 2024-04-27T01:57:00.768 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.force_consistency_checks: 1 2024-04-27T01:57:00.768 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.report_bg_io_stats: 0 2024-04-27T01:57:00.768 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.ttl: 2592000 2024-04-27T01:57:00.768 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-27T01:57:00.768 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-27T01:57:00.768 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-27T01:57:00.768 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.enable_blob_files: false 2024-04-27T01:57:00.768 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.min_blob_size: 0 2024-04-27T01:57:00.768 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.blob_file_size: 268435456 2024-04-27T01:57:00.768 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-27T01:57:00.769 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-27T01:57:00.769 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-27T01:57:00.769 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-27T01:57:00.769 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-27T01:57:00.769 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.blob_file_starting_level: 0 2024-04-27T01:57:00.769 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-27T01:57:00.769 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi029/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-27T01:57:00.769 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-27T01:57:00.769 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 54c39251-90af-49a0-a0f2-32bc4fe9ffff 2024-04-27T01:57:00.769 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714183020766556, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-27T01:57:00.769 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-27T01:57:00.769 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714183020767623, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93766, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 323, "table_properties": {"data_size": 91788, "index_size": 260, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13592, "raw_average_key_size": 49, "raw_value_size": 84384, "raw_average_value_size": 307, "num_data_blocks": 11, "num_entries": 274, "num_filter_entries": 274, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714183020, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "54c39251-90af-49a0-a0f2-32bc4fe9ffff", "db_session_id": "02WVA901GTEJOW8FKLQC", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-27T01:57:00.769 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714183020767710, "job": 1, "event": "recovery_finished"} 2024-04-27T01:57:00.770 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-27T01:57:00.770 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi029/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-27T01:57:00.770 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d40477c000 2024-04-27T01:57:00.770 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: DB pointer 0x55d404768000 2024-04-27T01:57:00.770 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-27T01:57:00.770 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-27T01:57:00.770 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: ** DB Stats ** 2024-04-27T01:57:00.770 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T01:57:00.770 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-27T01:57:00.770 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-27T01:57:00.770 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T01:57:00.770 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-27T01:57:00.770 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-27T01:57:00.771 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T01:57:00.771 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: 2024-04-27T01:57:00.771 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: ** Compaction Stats [default] ** 2024-04-27T01:57:00.771 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-27T01:57:00.771 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-27T01:57:00.771 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: L0 2/0 93.44 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-27T01:57:00.771 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Sum 2/0 93.44 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-27T01:57:00.771 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-27T01:57:00.771 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: 2024-04-27T01:57:00.771 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: ** Compaction Stats [default] ** 2024-04-27T01:57:00.771 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-27T01:57:00.771 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-27T01:57:00.771 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 137.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-27T01:57:00.771 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: 2024-04-27T01:57:00.772 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-27T01:57:00.772 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: 2024-04-27T01:57:00.772 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T01:57:00.772 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-27T01:57:00.772 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-27T01:57:00.772 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-27T01:57:00.772 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-27T01:57:00.772 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-27T01:57:00.772 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Cumulative compaction: 0.00 GB write, 28.20 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T01:57:00.772 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Interval compaction: 0.00 GB write, 28.20 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T01:57:00.772 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-27T01:57:00.772 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Block cache BinnedLRUCache@0x55d4046df1f0#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-27T01:57:00.772 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-04-27T01:57:00.772 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: 2024-04-27T01:57:00.773 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-27T01:57:00.773 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: starting mon.smithi029 rank 0 at public addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] at bind addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi029 fsid 4c587076-0439-11ef-bc93-c7b262605968 2024-04-27T01:57:00.773 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: mon.smithi029@-1(???) e1 preinit fsid 4c587076-0439-11ef-bc93-c7b262605968 2024-04-27T01:57:00.773 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: mon.smithi029@-1(???).mds e1 new map 2024-04-27T01:57:00.773 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: mon.smithi029@-1(???).mds e1 print_map 2024-04-27T01:57:00.773 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: e1 2024-04-27T01:57:00.773 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T01:57:00.773 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T01:57:00.773 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: legacy client fscid: -1 2024-04-27T01:57:00.773 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: 2024-04-27T01:57:00.773 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: No filesystems configured 2024-04-27T01:57:00.773 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: mon.smithi029@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-27T01:57:00.773 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: mon.smithi029@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T01:57:00.773 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: mon.smithi029@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T01:57:00.773 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: mon.smithi029@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T01:57:00.774 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: mon.smithi029@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-27T01:57:00.774 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: expand_channel_meta expand map: {default=false} 2024-04-27T01:57:00.774 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: expand_channel_meta from 'false' to 'false' 2024-04-27T01:57:00.774 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: expand_channel_meta expanded map: {default=false} 2024-04-27T01:57:00.774 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: expand_channel_meta expand map: {default=info} 2024-04-27T01:57:00.774 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: expand_channel_meta from 'info' to 'info' 2024-04-27T01:57:00.774 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: expand_channel_meta expanded map: {default=info} 2024-04-27T01:57:00.774 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: expand_channel_meta expand map: {default=daemon} 2024-04-27T01:57:00.774 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: expand_channel_meta from 'daemon' to 'daemon' 2024-04-27T01:57:00.774 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: expand_channel_meta expanded map: {default=daemon} 2024-04-27T01:57:00.774 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: expand_channel_meta expand map: {default=debug} 2024-04-27T01:57:00.774 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: expand_channel_meta from 'debug' to 'debug' 2024-04-27T01:57:00.774 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: expand_channel_meta expanded map: {default=debug} 2024-04-27T01:57:00.837 INFO:teuthology.orchestra.run.smithi029.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-27T01:57:01.053 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: mon.smithi029 is new leader, mons smithi029 in quorum (ranks 0) 2024-04-27T01:57:01.054 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: monmap e1: 1 mons at {smithi029=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0]} removed_ranks: {} 2024-04-27T01:57:01.054 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: fsmap 2024-04-27T01:57:01.054 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: osdmap e1: 0 total, 0 up, 0 in 2024-04-27T01:57:01.054 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 ceph-mon[26769]: mgrmap e1: no daemons active 2024-04-27T01:57:01.054 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 bash[26750]: 33130493e80ded41989df919b6ae566012e0ee93d10a59b55f00bb543ccc9c59 2024-04-27T01:57:01.054 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:00 smithi029 systemd[1]: Started Ceph mon.smithi029 for 4c587076-0439-11ef-bc93-c7b262605968. 2024-04-27T01:57:02.381 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.264+0000 7fd914ca5700 1 Processor -- start 2024-04-27T01:57:02.381 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.264+0000 7fd914ca5700 1 -- start start 2024-04-27T01:57:02.381 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.264+0000 7fd914ca5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd910105b10 0x7fd910105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:02.381 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.264+0000 7fd914ca5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd910106460 con 0x7fd910105b10 2024-04-27T01:57:02.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.264+0000 7fd90e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd910105b10 0x7fd910105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:02.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.264+0000 7fd90e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd910105b10 0x7fd910105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33418/0 (socket says 172.21.15.29:33418) 2024-04-27T01:57:02.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.264+0000 7fd90e59c700 1 -- 172.21.15.29:0/3192941265 learned_addr learned my addr 172.21.15.29:0/3192941265 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:02.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.265+0000 7fd90e59c700 1 -- 172.21.15.29:0/3192941265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9101065a0 con 0x7fd910105b10 2024-04-27T01:57:02.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.265+0000 7fd90e59c700 1 --2- 172.21.15.29:0/3192941265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd910105b10 0x7fd910105f20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd9000098a0 tx=0x7fd900009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61e8e056781e45f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:02.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.265+0000 7fd90d59a700 1 -- 172.21.15.29:0/3192941265 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd900004180 con 0x7fd910105b10 2024-04-27T01:57:02.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.265+0000 7fd90d59a700 1 -- 172.21.15.29:0/3192941265 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd9000042e0 con 0x7fd910105b10 2024-04-27T01:57:02.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.265+0000 7fd90d59a700 1 -- 172.21.15.29:0/3192941265 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd900011a50 con 0x7fd910105b10 2024-04-27T01:57:02.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.266+0000 7fd914ca5700 1 -- 172.21.15.29:0/3192941265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd910105b10 msgr2=0x7fd910105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:02.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.266+0000 7fd914ca5700 1 --2- 172.21.15.29:0/3192941265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd910105b10 0x7fd910105f20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd9000098a0 tx=0x7fd900009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:02.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.266+0000 7fd914ca5700 1 -- 172.21.15.29:0/3192941265 shutdown_connections 2024-04-27T01:57:02.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.266+0000 7fd914ca5700 1 --2- 172.21.15.29:0/3192941265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd910105b10 0x7fd910105f20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:02.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.266+0000 7fd914ca5700 1 -- 172.21.15.29:0/3192941265 >> 172.21.15.29:0/3192941265 conn(0x7fd910101180 msgr2=0x7fd910103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:02.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.266+0000 7fd914ca5700 1 -- 172.21.15.29:0/3192941265 shutdown_connections 2024-04-27T01:57:02.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.266+0000 7fd914ca5700 1 -- 172.21.15.29:0/3192941265 wait complete. 2024-04-27T01:57:02.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.267+0000 7fd914ca5700 1 Processor -- start 2024-04-27T01:57:02.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.267+0000 7fd914ca5700 1 -- start start 2024-04-27T01:57:02.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.267+0000 7fd914ca5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd910105b10 0x7fd910199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:02.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.267+0000 7fd914ca5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd910199d90 con 0x7fd910105b10 2024-04-27T01:57:02.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.267+0000 7fd90e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd910105b10 0x7fd910199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:02.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.267+0000 7fd90e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd910105b10 0x7fd910199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33430/0 (socket says 172.21.15.29:33430) 2024-04-27T01:57:02.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.267+0000 7fd90e59c700 1 -- 172.21.15.29:0/3675777890 learned_addr learned my addr 172.21.15.29:0/3675777890 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:02.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.267+0000 7fd90e59c700 1 -- 172.21.15.29:0/3675777890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd900009550 con 0x7fd910105b10 2024-04-27T01:57:02.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.267+0000 7fd90e59c700 1 --2- 172.21.15.29:0/3675777890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd910105b10 0x7fd910199850 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd900000c00 tx=0x7fd900011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:02.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.268+0000 7fd8ff7fe700 1 -- 172.21.15.29:0/3675777890 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd900003b90 con 0x7fd910105b10 2024-04-27T01:57:02.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.268+0000 7fd8ff7fe700 1 -- 172.21.15.29:0/3675777890 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd9000248b0 con 0x7fd910105b10 2024-04-27T01:57:02.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.268+0000 7fd914ca5700 1 -- 172.21.15.29:0/3675777890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd910199f90 con 0x7fd910105b10 2024-04-27T01:57:02.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.268+0000 7fd8ff7fe700 1 -- 172.21.15.29:0/3675777890 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd90001a8e0 con 0x7fd910105b10 2024-04-27T01:57:02.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.268+0000 7fd914ca5700 1 -- 172.21.15.29:0/3675777890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd91019a3b0 con 0x7fd910105b10 2024-04-27T01:57:02.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.268+0000 7fd8ff7fe700 1 -- 172.21.15.29:0/3675777890 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd90002d3f0 con 0x7fd910105b10 2024-04-27T01:57:02.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.268+0000 7fd8ff7fe700 1 -- 172.21.15.29:0/3675777890 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd9000194a0 con 0x7fd910105b10 2024-04-27T01:57:02.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.269+0000 7fd914ca5700 1 -- 172.21.15.29:0/3675777890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd910062380 con 0x7fd910105b10 2024-04-27T01:57:02.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.271+0000 7fd8ff7fe700 1 -- 172.21.15.29:0/3675777890 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fd900024db0 con 0x7fd910105b10 2024-04-27T01:57:02.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.342+0000 7fd914ca5700 1 -- 172.21.15.29:0/3675777890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fd910107980 con 0x7fd910105b10 2024-04-27T01:57:02.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.345+0000 7fd8ff7fe700 1 -- 172.21.15.29:0/3675777890 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd900024a20 con 0x7fd910105b10 2024-04-27T01:57:02.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.346+0000 7fd914ca5700 1 -- 172.21.15.29:0/3675777890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd910105b10 msgr2=0x7fd910199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:02.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.346+0000 7fd914ca5700 1 --2- 172.21.15.29:0/3675777890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd910105b10 0x7fd910199850 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd900000c00 tx=0x7fd900011ab0 comp rx=0 tx=0).stop 2024-04-27T01:57:02.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.346+0000 7fd914ca5700 1 -- 172.21.15.29:0/3675777890 shutdown_connections 2024-04-27T01:57:02.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.346+0000 7fd914ca5700 1 --2- 172.21.15.29:0/3675777890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd910105b10 0x7fd910199850 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:02.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.346+0000 7fd914ca5700 1 -- 172.21.15.29:0/3675777890 >> 172.21.15.29:0/3675777890 conn(0x7fd910101180 msgr2=0x7fd910101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:02.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.346+0000 7fd914ca5700 1 -- 172.21.15.29:0/3675777890 shutdown_connections 2024-04-27T01:57:02.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:01.346+0000 7fd914ca5700 1 -- 172.21.15.29:0/3675777890 wait complete. 2024-04-27T01:57:02.386 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-27T01:57:02.407 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-27T01:57:02.407 INFO:teuthology.orchestra.run.smithi029.stdout:Creating mgr... 2024-04-27T01:57:02.407 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-27T01:57:02.408 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-27T01:57:02.408 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying port 0.0.0.0:8443 ... 2024-04-27T01:57:02.683 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-4c587076-0439-11ef-bc93-c7b262605968@mgr.smithi029.ewvdzs 2024-04-27T01:57:02.683 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to reset failed state of unit ceph-4c587076-0439-11ef-bc93-c7b262605968@mgr.smithi029.ewvdzs.service: Unit ceph-4c587076-0439-11ef-bc93-c7b262605968@mgr.smithi029.ewvdzs.service not loaded. 2024-04-27T01:57:02.690 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:02 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/3675777890' entity='client.admin' 2024-04-27T01:57:02.841 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-4c587076-0439-11ef-bc93-c7b262605968.target.wants/ceph-4c587076-0439-11ef-bc93-c7b262605968@mgr.smithi029.ewvdzs.service → /etc/systemd/system/ceph-4c587076-0439-11ef-bc93-c7b262605968@.service. 2024-04-27T01:57:03.425 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-27T01:57:03.425 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-04-27T01:57:03.433 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-27T01:57:03.433 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-04-27T01:57:03.433 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-04-27T01:57:03.433 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to enable service . firewalld.service is not available 2024-04-27T01:57:03.442 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-27T01:57:03.442 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-04-27T01:57:03.450 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-27T01:57:03.450 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-04-27T01:57:03.450 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-04-27T01:57:03.450 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-04-27T01:57:03.450 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr to start... 2024-04-27T01:57:03.450 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr... 2024-04-27T01:57:04.297 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:04 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/4144994745' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-27T01:57:04.670 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-04-27T01:57:04.670 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-04-27T01:57:04.671 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "4c587076-0439-11ef-bc93-c7b262605968", 2024-04-27T01:57:04.671 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-04-27T01:57:04.671 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-27T01:57:04.671 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-27T01:57:04.671 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-27T01:57:04.671 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:04.671 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-27T01:57:04.671 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-27T01:57:04.671 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-04-27T01:57:04.671 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:57:04.671 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-27T01:57:04.671 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "smithi029" 2024-04-27T01:57:04.671 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:57:04.671 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-27T01:57:04.671 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-27T01:57:04.672 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:57:04.672 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-27T01:57:04.672 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-27T01:57:04.672 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:04.672 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-27T01:57:04.672 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:57:04.672 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-27T01:57:04.672 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-27T01:57:04.672 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-27T01:57:04.672 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-27T01:57:04.672 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-27T01:57:04.672 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-27T01:57:04.672 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:04.672 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-27T01:57:04.672 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-27T01:57:04.673 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-27T01:57:04.673 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-27T01:57:04.673 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-27T01:57:04.673 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-27T01:57:04.673 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-27T01:57:04.673 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-27T01:57:04.673 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-27T01:57:04.673 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:04.673 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-27T01:57:04.673 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:57:04.673 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-27T01:57:04.673 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-27T01:57:04.673 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:04.673 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-27T01:57:04.673 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-27T01:57:04.674 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-27T01:57:04.674 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-27T01:57:04.674 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-27T01:57:04.674 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-27T01:57:04.674 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-04-27T01:57:04.674 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:57:04.674 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-27T01:57:04.674 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:04.674 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-27T01:57:04.674 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:57:04.674 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-04-27T01:56:52.759060+0000", 2024-04-27T01:57:04.674 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-27T01:57:04.674 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:04.674 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-27T01:57:04.675 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-04-27T01:57:04.675 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.903+0000 7f3857976700 1 Processor -- start 2024-04-27T01:57:04.675 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.903+0000 7f3857976700 1 -- start start 2024-04-27T01:57:04.675 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.903+0000 7f3857976700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3850104440 0x7f3850104850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:04.675 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.903+0000 7f3857976700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38500744d0 con 0x7f3850104440 2024-04-27T01:57:04.675 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.903+0000 7f3855712700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3850104440 0x7f3850104850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:04.675 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.904+0000 7f3855712700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3850104440 0x7f3850104850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33456/0 (socket says 172.21.15.29:33456) 2024-04-27T01:57:04.675 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.904+0000 7f3855712700 1 -- 172.21.15.29:0/2579525808 learned_addr learned my addr 172.21.15.29:0/2579525808 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:04.675 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.904+0000 7f3855712700 1 -- 172.21.15.29:0/2579525808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3850074610 con 0x7f3850104440 2024-04-27T01:57:04.675 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.904+0000 7f3855712700 1 --2- 172.21.15.29:0/2579525808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3850104440 0x7f3850104850 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f384c0098a0 tx=0x7f384c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fb9dee7e8cbdb51 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:04.675 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.904+0000 7f3847fff700 1 -- 172.21.15.29:0/2579525808 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f384c004180 con 0x7f3850104440 2024-04-27T01:57:04.675 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.904+0000 7f3847fff700 1 -- 172.21.15.29:0/2579525808 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f384c0042e0 con 0x7f3850104440 2024-04-27T01:57:04.675 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.904+0000 7f3847fff700 1 -- 172.21.15.29:0/2579525808 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f384c011a50 con 0x7f3850104440 2024-04-27T01:57:04.675 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.905+0000 7f3857976700 1 -- 172.21.15.29:0/2579525808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3850104440 msgr2=0x7f3850104850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:04.676 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.905+0000 7f3857976700 1 --2- 172.21.15.29:0/2579525808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3850104440 0x7f3850104850 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f384c0098a0 tx=0x7f384c009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:04.676 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.905+0000 7f3857976700 1 -- 172.21.15.29:0/2579525808 shutdown_connections 2024-04-27T01:57:04.676 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.905+0000 7f3857976700 1 --2- 172.21.15.29:0/2579525808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3850104440 0x7f3850104850 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:04.676 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.905+0000 7f3857976700 1 -- 172.21.15.29:0/2579525808 >> 172.21.15.29:0/2579525808 conn(0x7f3850100070 msgr2=0x7f3850102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:04.676 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.905+0000 7f3857976700 1 -- 172.21.15.29:0/2579525808 shutdown_connections 2024-04-27T01:57:04.676 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.905+0000 7f3857976700 1 -- 172.21.15.29:0/2579525808 wait complete. 2024-04-27T01:57:04.676 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.906+0000 7f3857976700 1 Processor -- start 2024-04-27T01:57:04.676 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.906+0000 7f3857976700 1 -- start start 2024-04-27T01:57:04.676 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.906+0000 7f3857976700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3850104440 0x7f385019b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:04.676 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.906+0000 7f3857976700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385019bef0 con 0x7f3850104440 2024-04-27T01:57:04.676 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.906+0000 7f3855712700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3850104440 0x7f385019b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:04.676 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.906+0000 7f3855712700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3850104440 0x7f385019b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33466/0 (socket says 172.21.15.29:33466) 2024-04-27T01:57:04.676 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.906+0000 7f3855712700 1 -- 172.21.15.29:0/4144994745 learned_addr learned my addr 172.21.15.29:0/4144994745 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:04.676 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.906+0000 7f3855712700 1 -- 172.21.15.29:0/4144994745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f384c009550 con 0x7f3850104440 2024-04-27T01:57:04.677 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.906+0000 7f3855712700 1 --2- 172.21.15.29:0/4144994745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3850104440 0x7f385019b9b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f384c000c00 tx=0x7f384c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:04.677 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.907+0000 7f38467fc700 1 -- 172.21.15.29:0/4144994745 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f384c003b90 con 0x7f3850104440 2024-04-27T01:57:04.677 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.907+0000 7f38467fc700 1 -- 172.21.15.29:0/4144994745 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f384c0248b0 con 0x7f3850104440 2024-04-27T01:57:04.677 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.907+0000 7f38467fc700 1 -- 172.21.15.29:0/4144994745 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f384c01a8e0 con 0x7f3850104440 2024-04-27T01:57:04.677 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.907+0000 7f3857976700 1 -- 172.21.15.29:0/4144994745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f385019c150 con 0x7f3850104440 2024-04-27T01:57:04.677 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.907+0000 7f3857976700 1 -- 172.21.15.29:0/4144994745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f385019c5d0 con 0x7f3850104440 2024-04-27T01:57:04.677 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.907+0000 7f38467fc700 1 -- 172.21.15.29:0/4144994745 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f384c02d3f0 con 0x7f3850104440 2024-04-27T01:57:04.677 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.907+0000 7f38467fc700 1 -- 172.21.15.29:0/4144994745 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f384c019450 con 0x7f3850104440 2024-04-27T01:57:04.677 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.908+0000 7f3857976700 1 -- 172.21.15.29:0/4144994745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3850195220 con 0x7f3850104440 2024-04-27T01:57:04.677 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.910+0000 7f38467fc700 1 -- 172.21.15.29:0/4144994745 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f384c024a20 con 0x7f3850104440 2024-04-27T01:57:04.677 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.985+0000 7f3857976700 1 -- 172.21.15.29:0/4144994745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f385002cc30 con 0x7f3850104440 2024-04-27T01:57:04.677 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.985+0000 7f38467fc700 1 -- 172.21.15.29:0/4144994745 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f384c035090 con 0x7f3850104440 2024-04-27T01:57:04.677 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.986+0000 7f3857976700 1 -- 172.21.15.29:0/4144994745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3850104440 msgr2=0x7f385019b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:04.677 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.986+0000 7f3857976700 1 --2- 172.21.15.29:0/4144994745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3850104440 0x7f385019b9b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f384c000c00 tx=0x7f384c011ab0 comp rx=0 tx=0).stop 2024-04-27T01:57:04.678 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.986+0000 7f3857976700 1 -- 172.21.15.29:0/4144994745 shutdown_connections 2024-04-27T01:57:04.678 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.986+0000 7f3857976700 1 --2- 172.21.15.29:0/4144994745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3850104440 0x7f385019b9b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:04.678 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.986+0000 7f3857976700 1 -- 172.21.15.29:0/4144994745 >> 172.21.15.29:0/4144994745 conn(0x7f3850100070 msgr2=0x7f3850190000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:04.678 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.986+0000 7f3857976700 1 -- 172.21.15.29:0/4144994745 shutdown_connections 2024-04-27T01:57:04.678 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:03.986+0000 7f3857976700 1 -- 172.21.15.29:0/4144994745 wait complete. 2024-04-27T01:57:04.678 INFO:teuthology.orchestra.run.smithi029.stdout:mgr not available, waiting (1/15)... 2024-04-27T01:57:07.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:07 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/830863800' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-27T01:57:07.785 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-04-27T01:57:07.786 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-04-27T01:57:07.786 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "4c587076-0439-11ef-bc93-c7b262605968", 2024-04-27T01:57:07.786 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-04-27T01:57:07.786 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-27T01:57:07.786 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-27T01:57:07.786 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-27T01:57:07.786 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:07.786 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-27T01:57:07.786 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-27T01:57:07.786 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-04-27T01:57:07.786 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:57:07.786 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-27T01:57:07.787 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "smithi029" 2024-04-27T01:57:07.787 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:57:07.787 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-27T01:57:07.787 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-27T01:57:07.787 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:57:07.787 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-27T01:57:07.787 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-27T01:57:07.787 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:07.787 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-27T01:57:07.787 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:57:07.787 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-27T01:57:07.787 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-27T01:57:07.787 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-27T01:57:07.795 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-27T01:57:07.795 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-27T01:57:07.795 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-27T01:57:07.795 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:07.796 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-27T01:57:07.796 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-27T01:57:07.796 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-27T01:57:07.796 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-27T01:57:07.796 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-27T01:57:07.796 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-27T01:57:07.796 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-27T01:57:07.796 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-27T01:57:07.796 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-27T01:57:07.796 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:07.796 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-27T01:57:07.796 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:57:07.796 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-27T01:57:07.797 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-27T01:57:07.797 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:07.797 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-27T01:57:07.797 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-27T01:57:07.797 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-27T01:57:07.797 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-27T01:57:07.797 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-27T01:57:07.797 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-27T01:57:07.797 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-04-27T01:57:07.797 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:57:07.797 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-27T01:57:07.797 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:07.797 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-27T01:57:07.797 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:57:07.798 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-04-27T01:56:52.759060+0000", 2024-04-27T01:57:07.798 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-27T01:57:07.798 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:07.798 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-27T01:57:07.798 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-04-27T01:57:07.798 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.120+0000 7f904a3bf700 1 Processor -- start 2024-04-27T01:57:07.798 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.120+0000 7f904a3bf700 1 -- start start 2024-04-27T01:57:07.798 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.120+0000 7f904a3bf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9044106610 0x7f90441089f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:07.798 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.120+0000 7f904a3bf700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9044074510 con 0x7f9044106610 2024-04-27T01:57:07.798 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.120+0000 7f9043fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9044106610 0x7f90441089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:07.798 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.120+0000 7f9043fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9044106610 0x7f90441089f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57872/0 (socket says 172.21.15.29:57872) 2024-04-27T01:57:07.798 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.120+0000 7f9043fff700 1 -- 172.21.15.29:0/3804698850 learned_addr learned my addr 172.21.15.29:0/3804698850 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:07.798 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.121+0000 7f9043fff700 1 -- 172.21.15.29:0/3804698850 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9044074650 con 0x7f9044106610 2024-04-27T01:57:07.798 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.121+0000 7f9043fff700 1 --2- 172.21.15.29:0/3804698850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9044106610 0x7f90441089f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f90340098a0 tx=0x7f9034009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c81bbf310adeb4ea server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:07.799 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.121+0000 7f9042ffd700 1 -- 172.21.15.29:0/3804698850 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9034004180 con 0x7f9044106610 2024-04-27T01:57:07.799 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.121+0000 7f9042ffd700 1 -- 172.21.15.29:0/3804698850 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f90340042e0 con 0x7f9044106610 2024-04-27T01:57:07.799 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.121+0000 7f9042ffd700 1 -- 172.21.15.29:0/3804698850 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9034011a50 con 0x7f9044106610 2024-04-27T01:57:07.799 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.122+0000 7f904a3bf700 1 -- 172.21.15.29:0/3804698850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9044106610 msgr2=0x7f90441089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:07.799 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.122+0000 7f904a3bf700 1 --2- 172.21.15.29:0/3804698850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9044106610 0x7f90441089f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f90340098a0 tx=0x7f9034009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:07.799 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.122+0000 7f904a3bf700 1 -- 172.21.15.29:0/3804698850 shutdown_connections 2024-04-27T01:57:07.799 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.122+0000 7f904a3bf700 1 --2- 172.21.15.29:0/3804698850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9044106610 0x7f90441089f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:07.799 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.122+0000 7f904a3bf700 1 -- 172.21.15.29:0/3804698850 >> 172.21.15.29:0/3804698850 conn(0x7f9044100070 msgr2=0x7f9044102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:07.799 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.122+0000 7f904a3bf700 1 -- 172.21.15.29:0/3804698850 shutdown_connections 2024-04-27T01:57:07.799 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.122+0000 7f904a3bf700 1 -- 172.21.15.29:0/3804698850 wait complete. 2024-04-27T01:57:07.799 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.123+0000 7f904a3bf700 1 Processor -- start 2024-04-27T01:57:07.799 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.123+0000 7f904a3bf700 1 -- start start 2024-04-27T01:57:07.799 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.123+0000 7f904a3bf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9044106610 0x7f9044199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:07.799 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.123+0000 7f904a3bf700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9044199d90 con 0x7f9044106610 2024-04-27T01:57:07.800 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.123+0000 7f9043fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9044106610 0x7f9044199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:07.800 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.123+0000 7f9043fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9044106610 0x7f9044199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57876/0 (socket says 172.21.15.29:57876) 2024-04-27T01:57:07.800 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.123+0000 7f9043fff700 1 -- 172.21.15.29:0/830863800 learned_addr learned my addr 172.21.15.29:0/830863800 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:07.800 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.124+0000 7f9043fff700 1 -- 172.21.15.29:0/830863800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9034009550 con 0x7f9044106610 2024-04-27T01:57:07.800 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.124+0000 7f9043fff700 1 --2- 172.21.15.29:0/830863800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9044106610 0x7f9044199850 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9034006af0 tx=0x7f9034011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:07.800 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.124+0000 7f90417fa700 1 -- 172.21.15.29:0/830863800 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9034003b90 con 0x7f9044106610 2024-04-27T01:57:07.800 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.124+0000 7f904a3bf700 1 -- 172.21.15.29:0/830863800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9044199f90 con 0x7f9044106610 2024-04-27T01:57:07.800 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.124+0000 7f90417fa700 1 -- 172.21.15.29:0/830863800 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f903401a8a0 con 0x7f9044106610 2024-04-27T01:57:07.800 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.124+0000 7f90417fa700 1 -- 172.21.15.29:0/830863800 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9034019880 con 0x7f9044106610 2024-04-27T01:57:07.800 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.124+0000 7f904a3bf700 1 -- 172.21.15.29:0/830863800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f904419a3b0 con 0x7f9044106610 2024-04-27T01:57:07.800 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.125+0000 7f90417fa700 1 -- 172.21.15.29:0/830863800 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f903402d440 con 0x7f9044106610 2024-04-27T01:57:07.800 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.125+0000 7f90417fa700 1 -- 172.21.15.29:0/830863800 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9034024860 con 0x7f9044106610 2024-04-27T01:57:07.800 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.125+0000 7f904a3bf700 1 -- 172.21.15.29:0/830863800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9044062380 con 0x7f9044106610 2024-04-27T01:57:07.800 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.128+0000 7f90417fa700 1 -- 172.21.15.29:0/830863800 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f9034019e40 con 0x7f9044106610 2024-04-27T01:57:07.801 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.202+0000 7f904a3bf700 1 -- 172.21.15.29:0/830863800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f904419cee0 con 0x7f9044106610 2024-04-27T01:57:07.801 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.203+0000 7f90417fa700 1 -- 172.21.15.29:0/830863800 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f903401aa10 con 0x7f9044106610 2024-04-27T01:57:07.801 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.204+0000 7f904a3bf700 1 -- 172.21.15.29:0/830863800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9044106610 msgr2=0x7f9044199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:07.801 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.204+0000 7f904a3bf700 1 --2- 172.21.15.29:0/830863800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9044106610 0x7f9044199850 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9034006af0 tx=0x7f9034011a90 comp rx=0 tx=0).stop 2024-04-27T01:57:07.801 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.204+0000 7f904a3bf700 1 -- 172.21.15.29:0/830863800 shutdown_connections 2024-04-27T01:57:07.801 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.204+0000 7f904a3bf700 1 --2- 172.21.15.29:0/830863800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9044106610 0x7f9044199850 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:07.801 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.204+0000 7f904a3bf700 1 -- 172.21.15.29:0/830863800 >> 172.21.15.29:0/830863800 conn(0x7f9044100070 msgr2=0x7f9044101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:07.801 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.204+0000 7f904a3bf700 1 -- 172.21.15.29:0/830863800 shutdown_connections 2024-04-27T01:57:07.801 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:07.204+0000 7f904a3bf700 1 -- 172.21.15.29:0/830863800 wait complete. 2024-04-27T01:57:07.801 INFO:teuthology.orchestra.run.smithi029.stdout:mgr not available, waiting (2/15)... 2024-04-27T01:57:10.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:10 smithi029 ceph-mon[26769]: Activating manager daemon smithi029.ewvdzs 2024-04-27T01:57:10.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:10 smithi029 ceph-mon[26769]: mgrmap e2: smithi029.ewvdzs(active, starting, since 0.0031832s) 2024-04-27T01:57:10.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:10 smithi029 ceph-mon[26769]: from='mgr.14100 172.21.15.29:0/2391980838' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T01:57:10.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:10 smithi029 ceph-mon[26769]: from='mgr.14100 172.21.15.29:0/2391980838' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T01:57:10.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:10 smithi029 ceph-mon[26769]: from='mgr.14100 172.21.15.29:0/2391980838' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T01:57:10.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:10 smithi029 ceph-mon[26769]: from='mgr.14100 172.21.15.29:0/2391980838' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-04-27T01:57:10.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:10 smithi029 ceph-mon[26769]: from='mgr.14100 172.21.15.29:0/2391980838' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mgr metadata", "who": "smithi029.ewvdzs", "id": "smithi029.ewvdzs"}]: dispatch 2024-04-27T01:57:10.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:10 smithi029 ceph-mon[26769]: Manager daemon smithi029.ewvdzs is now available 2024-04-27T01:57:10.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:10 smithi029 ceph-mon[26769]: from='mgr.14100 172.21.15.29:0/2391980838' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.ewvdzs/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:57:10.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:10 smithi029 ceph-mon[26769]: from='mgr.14100 172.21.15.29:0/2391980838' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.ewvdzs/trash_purge_schedule"}]: dispatch 2024-04-27T01:57:10.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:10 smithi029 ceph-mon[26769]: from='mgr.14100 172.21.15.29:0/2391980838' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:10.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:10 smithi029 ceph-mon[26769]: from='mgr.14100 172.21.15.29:0/2391980838' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:10.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:10 smithi029 ceph-mon[26769]: from='mgr.14100 172.21.15.29:0/2391980838' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:10.874 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-04-27T01:57:10.874 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-04-27T01:57:10.874 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "4c587076-0439-11ef-bc93-c7b262605968", 2024-04-27T01:57:10.874 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-04-27T01:57:10.874 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-27T01:57:10.875 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-27T01:57:10.875 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-27T01:57:10.875 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:10.875 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-27T01:57:10.875 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-27T01:57:10.875 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-04-27T01:57:10.875 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:57:10.875 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-27T01:57:10.875 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "smithi029" 2024-04-27T01:57:10.875 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:57:10.876 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-04-27T01:57:10.876 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-27T01:57:10.876 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:57:10.876 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-27T01:57:10.876 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-27T01:57:10.876 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:10.876 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-27T01:57:10.876 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:57:10.876 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-27T01:57:10.876 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-27T01:57:10.876 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-27T01:57:10.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-27T01:57:10.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-27T01:57:10.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-27T01:57:10.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:10.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-27T01:57:10.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-27T01:57:10.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-27T01:57:10.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-27T01:57:10.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-27T01:57:10.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-27T01:57:10.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-27T01:57:10.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-27T01:57:10.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-27T01:57:10.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:10.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-27T01:57:10.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:57:10.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-27T01:57:10.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-27T01:57:10.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:10.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-27T01:57:10.887 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-27T01:57:10.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-27T01:57:10.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-27T01:57:10.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-27T01:57:10.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-27T01:57:10.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-04-27T01:57:10.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:57:10.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-27T01:57:10.888 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:10.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-27T01:57:10.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:57:10.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-04-27T01:56:52.759060+0000", 2024-04-27T01:57:10.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-27T01:57:10.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:10.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-27T01:57:10.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-04-27T01:57:10.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.232+0000 7f9356bb5700 1 Processor -- start 2024-04-27T01:57:10.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.232+0000 7f9356bb5700 1 -- start start 2024-04-27T01:57:10.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.232+0000 7f9356bb5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9350105b10 0x7f9350105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:10.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.232+0000 7f9356bb5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9350106460 con 0x7f9350105b10 2024-04-27T01:57:10.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.232+0000 7f9354951700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9350105b10 0x7f9350105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:10.889 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.233+0000 7f9354951700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9350105b10 0x7f9350105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57960/0 (socket says 172.21.15.29:57960) 2024-04-27T01:57:10.890 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.233+0000 7f9354951700 1 -- 172.21.15.29:0/1291044418 learned_addr learned my addr 172.21.15.29:0/1291044418 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:10.890 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.233+0000 7f9354951700 1 -- 172.21.15.29:0/1291044418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93501065a0 con 0x7f9350105b10 2024-04-27T01:57:10.890 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.233+0000 7f9354951700 1 --2- 172.21.15.29:0/1291044418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9350105b10 0x7f9350105f20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f93440098a0 tx=0x7f9344009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6c76e2ce30aaf47 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:10.890 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.233+0000 7f934f7fe700 1 -- 172.21.15.29:0/1291044418 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9344004180 con 0x7f9350105b10 2024-04-27T01:57:10.890 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.233+0000 7f934f7fe700 1 -- 172.21.15.29:0/1291044418 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f93440042e0 con 0x7f9350105b10 2024-04-27T01:57:10.890 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.233+0000 7f934f7fe700 1 -- 172.21.15.29:0/1291044418 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9344011a50 con 0x7f9350105b10 2024-04-27T01:57:10.890 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.234+0000 7f9356bb5700 1 -- 172.21.15.29:0/1291044418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9350105b10 msgr2=0x7f9350105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:10.890 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.234+0000 7f9356bb5700 1 --2- 172.21.15.29:0/1291044418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9350105b10 0x7f9350105f20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f93440098a0 tx=0x7f9344009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:10.890 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.234+0000 7f9356bb5700 1 -- 172.21.15.29:0/1291044418 shutdown_connections 2024-04-27T01:57:10.890 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.234+0000 7f9356bb5700 1 --2- 172.21.15.29:0/1291044418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9350105b10 0x7f9350105f20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:10.890 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.234+0000 7f9356bb5700 1 -- 172.21.15.29:0/1291044418 >> 172.21.15.29:0/1291044418 conn(0x7f9350101160 msgr2=0x7f9350103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:10.890 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.234+0000 7f9356bb5700 1 -- 172.21.15.29:0/1291044418 shutdown_connections 2024-04-27T01:57:10.891 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.234+0000 7f9356bb5700 1 -- 172.21.15.29:0/1291044418 wait complete. 2024-04-27T01:57:10.891 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.234+0000 7f9356bb5700 1 Processor -- start 2024-04-27T01:57:10.891 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.234+0000 7f9356bb5700 1 -- start start 2024-04-27T01:57:10.891 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.235+0000 7f9356bb5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9350105b10 0x7f9350199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:10.891 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.235+0000 7f9356bb5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9350199d70 con 0x7f9350105b10 2024-04-27T01:57:10.891 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.235+0000 7f9354951700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9350105b10 0x7f9350199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:10.891 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.235+0000 7f9354951700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9350105b10 0x7f9350199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57974/0 (socket says 172.21.15.29:57974) 2024-04-27T01:57:10.891 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.235+0000 7f9354951700 1 -- 172.21.15.29:0/1975209318 learned_addr learned my addr 172.21.15.29:0/1975209318 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:10.891 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.235+0000 7f9354951700 1 -- 172.21.15.29:0/1975209318 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9344009550 con 0x7f9350105b10 2024-04-27T01:57:10.891 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.235+0000 7f9354951700 1 --2- 172.21.15.29:0/1975209318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9350105b10 0x7f9350199830 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f93440119e0 tx=0x7f9344011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:10.891 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.235+0000 7f934dffb700 1 -- 172.21.15.29:0/1975209318 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9344003b90 con 0x7f9350105b10 2024-04-27T01:57:10.891 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.235+0000 7f9356bb5700 1 -- 172.21.15.29:0/1975209318 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9350199f70 con 0x7f9350105b10 2024-04-27T01:57:10.891 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.235+0000 7f934dffb700 1 -- 172.21.15.29:0/1975209318 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f934401a8a0 con 0x7f9350105b10 2024-04-27T01:57:10.891 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.235+0000 7f934dffb700 1 -- 172.21.15.29:0/1975209318 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9344019930 con 0x7f9350105b10 2024-04-27T01:57:10.892 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.236+0000 7f9356bb5700 1 -- 172.21.15.29:0/1975209318 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f935019a390 con 0x7f9350105b10 2024-04-27T01:57:10.892 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.237+0000 7f9356bb5700 1 -- 172.21.15.29:0/1975209318 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9350062380 con 0x7f9350105b10 2024-04-27T01:57:10.892 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.237+0000 7f934dffb700 1 -- 172.21.15.29:0/1975209318 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 2) v1 ==== 44840+0+0 (secure 0 0 0) 0x7f934401aa10 con 0x7f9350105b10 2024-04-27T01:57:10.892 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.237+0000 7f934dffb700 1 -- 172.21.15.29:0/1975209318 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f934404c220 con 0x7f9350105b10 2024-04-27T01:57:10.892 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.239+0000 7f934dffb700 1 -- 172.21.15.29:0/1975209318 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f9344019bf0 con 0x7f9350105b10 2024-04-27T01:57:10.892 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.313+0000 7f9356bb5700 1 -- 172.21.15.29:0/1975209318 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9350107980 con 0x7f9350105b10 2024-04-27T01:57:10.892 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.313+0000 7f934dffb700 1 -- 172.21.15.29:0/1975209318 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f9344018350 con 0x7f9350105b10 2024-04-27T01:57:10.892 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.314+0000 7f9356bb5700 1 -- 172.21.15.29:0/1975209318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9350105b10 msgr2=0x7f9350199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:10.892 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.314+0000 7f9356bb5700 1 --2- 172.21.15.29:0/1975209318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9350105b10 0x7f9350199830 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f93440119e0 tx=0x7f9344011ac0 comp rx=0 tx=0).stop 2024-04-27T01:57:10.892 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.314+0000 7f9356bb5700 1 -- 172.21.15.29:0/1975209318 shutdown_connections 2024-04-27T01:57:10.892 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.314+0000 7f9356bb5700 1 --2- 172.21.15.29:0/1975209318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9350105b10 0x7f9350199830 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:10.892 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.314+0000 7f9356bb5700 1 -- 172.21.15.29:0/1975209318 >> 172.21.15.29:0/1975209318 conn(0x7f9350101160 msgr2=0x7f9350101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:10.892 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.314+0000 7f9356bb5700 1 -- 172.21.15.29:0/1975209318 shutdown_connections 2024-04-27T01:57:10.893 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:10.315+0000 7f9356bb5700 1 -- 172.21.15.29:0/1975209318 wait complete. 2024-04-27T01:57:10.893 INFO:teuthology.orchestra.run.smithi029.stdout:mgr not available, waiting (3/15)... 2024-04-27T01:57:11.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:11 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/1975209318' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-27T01:57:11.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:11 smithi029 ceph-mon[26769]: mgrmap e3: smithi029.ewvdzs(active, since 1.00774s) 2024-04-27T01:57:13.317 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:13 smithi029 ceph-mon[26769]: mgrmap e4: smithi029.ewvdzs(active, since 2s) 2024-04-27T01:57:14.230 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-04-27T01:57:14.230 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-04-27T01:57:14.230 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "4c587076-0439-11ef-bc93-c7b262605968", 2024-04-27T01:57:14.230 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-04-27T01:57:14.230 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-27T01:57:14.230 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-27T01:57:14.230 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-27T01:57:14.230 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:14.230 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-27T01:57:14.230 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-27T01:57:14.230 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-04-27T01:57:14.231 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:57:14.231 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-27T01:57:14.231 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "smithi029" 2024-04-27T01:57:14.231 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:57:14.231 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-27T01:57:14.231 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-27T01:57:14.231 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:57:14.231 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-27T01:57:14.231 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-27T01:57:14.231 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:14.231 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-27T01:57:14.231 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:57:14.231 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-27T01:57:14.231 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-27T01:57:14.231 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-27T01:57:14.232 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-27T01:57:14.232 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-27T01:57:14.232 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-27T01:57:14.232 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:14.232 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-27T01:57:14.232 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-27T01:57:14.232 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-27T01:57:14.232 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-27T01:57:14.232 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-27T01:57:14.232 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-27T01:57:14.232 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-27T01:57:14.232 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-27T01:57:14.232 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-27T01:57:14.232 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:14.233 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-27T01:57:14.233 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:57:14.233 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-27T01:57:14.233 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-27T01:57:14.233 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:14.233 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-27T01:57:14.233 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-27T01:57:14.233 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-27T01:57:14.233 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-27T01:57:14.233 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-27T01:57:14.233 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-27T01:57:14.233 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-04-27T01:57:14.233 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:57:14.233 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-27T01:57:14.233 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:14.234 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-27T01:57:14.234 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:57:14.234 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-04-27T01:56:52.759060+0000", 2024-04-27T01:57:14.234 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-27T01:57:14.234 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:57:14.234 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-27T01:57:14.234 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-04-27T01:57:14.234 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.347+0000 7f167018c700 1 Processor -- start 2024-04-27T01:57:14.234 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.347+0000 7f167018c700 1 -- start start 2024-04-27T01:57:14.234 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.347+0000 7f167018c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f166807bdd0 0x7f166807a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:14.234 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.347+0000 7f167018c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166807a810 con 0x7f166807bdd0 2024-04-27T01:57:14.234 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.347+0000 7f166df28700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f166807bdd0 0x7f166807a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:14.234 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.347+0000 7f166df28700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f166807bdd0 0x7f166807a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57986/0 (socket says 172.21.15.29:57986) 2024-04-27T01:57:14.234 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.347+0000 7f166df28700 1 -- 172.21.15.29:0/1474442598 learned_addr learned my addr 172.21.15.29:0/1474442598 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:14.235 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.348+0000 7f166df28700 1 -- 172.21.15.29:0/1474442598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166807a950 con 0x7f166807bdd0 2024-04-27T01:57:14.235 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.348+0000 7f166df28700 1 --2- 172.21.15.29:0/1474442598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f166807bdd0 0x7f166807a2d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1660009e80 tx=0x7f166000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a877663fcbd8c6d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:14.235 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.348+0000 7f166cf26700 1 -- 172.21.15.29:0/1474442598 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f166000e070 con 0x7f166807bdd0 2024-04-27T01:57:14.235 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.348+0000 7f166cf26700 1 -- 172.21.15.29:0/1474442598 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1660003e90 con 0x7f166807bdd0 2024-04-27T01:57:14.235 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.348+0000 7f166cf26700 1 -- 172.21.15.29:0/1474442598 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1660004380 con 0x7f166807bdd0 2024-04-27T01:57:14.235 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.349+0000 7f167018c700 1 -- 172.21.15.29:0/1474442598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f166807bdd0 msgr2=0x7f166807a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:14.235 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.349+0000 7f167018c700 1 --2- 172.21.15.29:0/1474442598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f166807bdd0 0x7f166807a2d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1660009e80 tx=0x7f166000b330 comp rx=0 tx=0).stop 2024-04-27T01:57:14.235 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.349+0000 7f167018c700 1 -- 172.21.15.29:0/1474442598 shutdown_connections 2024-04-27T01:57:14.235 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.349+0000 7f167018c700 1 --2- 172.21.15.29:0/1474442598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f166807bdd0 0x7f166807a2d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:14.235 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.349+0000 7f167018c700 1 -- 172.21.15.29:0/1474442598 >> 172.21.15.29:0/1474442598 conn(0x7f1668101180 msgr2=0x7f1668103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:14.235 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.349+0000 7f167018c700 1 -- 172.21.15.29:0/1474442598 shutdown_connections 2024-04-27T01:57:14.235 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.349+0000 7f167018c700 1 -- 172.21.15.29:0/1474442598 wait complete. 2024-04-27T01:57:14.235 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.349+0000 7f167018c700 1 Processor -- start 2024-04-27T01:57:14.235 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.349+0000 7f167018c700 1 -- start start 2024-04-27T01:57:14.236 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.350+0000 7f167018c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f166807bdd0 0x7f166819ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:14.236 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.350+0000 7f167018c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166819bf70 con 0x7f166807bdd0 2024-04-27T01:57:14.236 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.350+0000 7f166df28700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f166807bdd0 0x7f166819ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:14.236 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.350+0000 7f166df28700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f166807bdd0 0x7f166819ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57998/0 (socket says 172.21.15.29:57998) 2024-04-27T01:57:14.236 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.350+0000 7f166df28700 1 -- 172.21.15.29:0/1688437387 learned_addr learned my addr 172.21.15.29:0/1688437387 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:14.236 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.350+0000 7f166df28700 1 -- 172.21.15.29:0/1688437387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1660009b30 con 0x7f166807bdd0 2024-04-27T01:57:14.236 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.350+0000 7f166df28700 1 --2- 172.21.15.29:0/1688437387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f166807bdd0 0x7f166819ba30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1660009530 tx=0x7f1660003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:14.236 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.350+0000 7f1666ffd700 1 -- 172.21.15.29:0/1688437387 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f166000e040 con 0x7f166807bdd0 2024-04-27T01:57:14.236 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.350+0000 7f1666ffd700 1 -- 172.21.15.29:0/1688437387 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1660004850 con 0x7f166807bdd0 2024-04-27T01:57:14.236 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.350+0000 7f167018c700 1 -- 172.21.15.29:0/1688437387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f166819c170 con 0x7f166807bdd0 2024-04-27T01:57:14.236 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.350+0000 7f1666ffd700 1 -- 172.21.15.29:0/1688437387 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f166001b680 con 0x7f166807bdd0 2024-04-27T01:57:14.236 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.350+0000 7f167018c700 1 -- 172.21.15.29:0/1688437387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f166819c5f0 con 0x7f166807bdd0 2024-04-27T01:57:14.236 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.351+0000 7f1666ffd700 1 -- 172.21.15.29:0/1688437387 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) v1 ==== 45073+0+0 (secure 0 0 0) 0x7f1660019040 con 0x7f166807bdd0 2024-04-27T01:57:14.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.351+0000 7f1666ffd700 1 --2- 172.21.15.29:0/1688437387 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f16500383a0 0x7f165003a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:14.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.351+0000 7f1666ffd700 1 -- 172.21.15.29:0/1688437387 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f166004bac0 con 0x7f166807bdd0 2024-04-27T01:57:14.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.352+0000 7f166d727700 1 --2- 172.21.15.29:0/1688437387 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f16500383a0 0x7f165003a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:14.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.352+0000 7f167018c700 1 -- 172.21.15.29:0/1688437387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1668062380 con 0x7f166807bdd0 2024-04-27T01:57:14.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.352+0000 7f166d727700 1 --2- 172.21.15.29:0/1688437387 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f16500383a0 0x7f165003a850 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f165c006fd0 tx=0x7f165c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:14.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.357+0000 7f1666ffd700 1 -- 172.21.15.29:0/1688437387 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1660029070 con 0x7f166807bdd0 2024-04-27T01:57:14.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.646+0000 7f167018c700 1 -- 172.21.15.29:0/1688437387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f166819f0c0 con 0x7f166807bdd0 2024-04-27T01:57:14.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.647+0000 7f1666ffd700 1 -- 172.21.15.29:0/1688437387 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f166001f070 con 0x7f166807bdd0 2024-04-27T01:57:14.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.649+0000 7f167018c700 1 -- 172.21.15.29:0/1688437387 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f16500383a0 msgr2=0x7f165003a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:14.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.649+0000 7f167018c700 1 --2- 172.21.15.29:0/1688437387 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f16500383a0 0x7f165003a850 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f165c006fd0 tx=0x7f165c006e40 comp rx=0 tx=0).stop 2024-04-27T01:57:14.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.649+0000 7f167018c700 1 -- 172.21.15.29:0/1688437387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f166807bdd0 msgr2=0x7f166819ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:14.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.649+0000 7f167018c700 1 --2- 172.21.15.29:0/1688437387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f166807bdd0 0x7f166819ba30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1660009530 tx=0x7f1660003d00 comp rx=0 tx=0).stop 2024-04-27T01:57:14.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.649+0000 7f167018c700 1 -- 172.21.15.29:0/1688437387 shutdown_connections 2024-04-27T01:57:14.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.649+0000 7f167018c700 1 --2- 172.21.15.29:0/1688437387 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f16500383a0 0x7f165003a850 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:14.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.649+0000 7f167018c700 1 --2- 172.21.15.29:0/1688437387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f166807bdd0 0x7f166819ba30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:14.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.649+0000 7f167018c700 1 -- 172.21.15.29:0/1688437387 >> 172.21.15.29:0/1688437387 conn(0x7f1668101180 msgr2=0x7f1668101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:14.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.649+0000 7f167018c700 1 -- 172.21.15.29:0/1688437387 shutdown_connections 2024-04-27T01:57:14.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:13.649+0000 7f167018c700 1 -- 172.21.15.29:0/1688437387 wait complete. 2024-04-27T01:57:14.238 INFO:teuthology.orchestra.run.smithi029.stdout:mgr is available 2024-04-27T01:57:14.343 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:14 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/1688437387' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-27T01:57:15.488 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:15 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/2295984218' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-27T01:57:15.498 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-04-27T01:57:15.498 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [global] 2024-04-27T01:57:15.498 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout fsid = 4c587076-0439-11ef-bc93-c7b262605968 2024-04-27T01:57:15.498 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-27T01:57:15.498 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-27T01:57:15.498 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-27T01:57:15.498 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-27T01:57:15.499 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-04-27T01:57:15.499 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-27T01:57:15.499 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-27T01:57:15.499 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-04-27T01:57:15.499 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [osd] 2024-04-27T01:57:15.499 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-27T01:57:15.499 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-27T01:57:15.499 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-27T01:57:15.499 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-27T01:57:15.499 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-27T01:57:15.499 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.671+0000 7f05aa2af700 1 Processor -- start 2024-04-27T01:57:15.499 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.671+0000 7f05aa2af700 1 -- start start 2024-04-27T01:57:15.499 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.671+0000 7f05aa2af700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a4104a10 0x7f05a4104e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:15.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.671+0000 7f05aa2af700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a4105360 con 0x7f05a4104a10 2024-04-27T01:57:15.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.671+0000 7f05a3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a4104a10 0x7f05a4104e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:15.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.671+0000 7f05a3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a4104a10 0x7f05a4104e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58004/0 (socket says 172.21.15.29:58004) 2024-04-27T01:57:15.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.671+0000 7f05a3fff700 1 -- 172.21.15.29:0/4062282704 learned_addr learned my addr 172.21.15.29:0/4062282704 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:15.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.671+0000 7f05a3fff700 1 -- 172.21.15.29:0/4062282704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a41054a0 con 0x7f05a4104a10 2024-04-27T01:57:15.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.672+0000 7f05a3fff700 1 --2- 172.21.15.29:0/4062282704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a4104a10 0x7f05a4104e20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f05940098a0 tx=0x7f0594009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6114de87be146a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:15.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.672+0000 7f05a2ffd700 1 -- 172.21.15.29:0/4062282704 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0594004180 con 0x7f05a4104a10 2024-04-27T01:57:15.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.672+0000 7f05a2ffd700 1 -- 172.21.15.29:0/4062282704 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f05940042e0 con 0x7f05a4104a10 2024-04-27T01:57:15.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.672+0000 7f05a2ffd700 1 -- 172.21.15.29:0/4062282704 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0594011a50 con 0x7f05a4104a10 2024-04-27T01:57:15.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.672+0000 7f05aa2af700 1 -- 172.21.15.29:0/4062282704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a4104a10 msgr2=0x7f05a4104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:15.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.672+0000 7f05aa2af700 1 --2- 172.21.15.29:0/4062282704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a4104a10 0x7f05a4104e20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f05940098a0 tx=0x7f0594009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:15.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.672+0000 7f05aa2af700 1 -- 172.21.15.29:0/4062282704 shutdown_connections 2024-04-27T01:57:15.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.672+0000 7f05aa2af700 1 --2- 172.21.15.29:0/4062282704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a4104a10 0x7f05a4104e20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:15.500 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.672+0000 7f05aa2af700 1 -- 172.21.15.29:0/4062282704 >> 172.21.15.29:0/4062282704 conn(0x7f05a4100060 msgr2=0x7f05a4102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:15.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.672+0000 7f05aa2af700 1 -- 172.21.15.29:0/4062282704 shutdown_connections 2024-04-27T01:57:15.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.672+0000 7f05aa2af700 1 -- 172.21.15.29:0/4062282704 wait complete. 2024-04-27T01:57:15.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.673+0000 7f05aa2af700 1 Processor -- start 2024-04-27T01:57:15.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.673+0000 7f05aa2af700 1 -- start start 2024-04-27T01:57:15.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.673+0000 7f05aa2af700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a4104a10 0x7f05a418fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:15.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.673+0000 7f05aa2af700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a41900f0 con 0x7f05a4104a10 2024-04-27T01:57:15.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.674+0000 7f05a3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a4104a10 0x7f05a418fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:15.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.674+0000 7f05a3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a4104a10 0x7f05a418fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58006/0 (socket says 172.21.15.29:58006) 2024-04-27T01:57:15.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.674+0000 7f05a3fff700 1 -- 172.21.15.29:0/2295984218 learned_addr learned my addr 172.21.15.29:0/2295984218 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:15.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.674+0000 7f05a3fff700 1 -- 172.21.15.29:0/2295984218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0594009550 con 0x7f05a4104a10 2024-04-27T01:57:15.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.674+0000 7f05a3fff700 1 --2- 172.21.15.29:0/2295984218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a4104a10 0x7f05a418fbb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0594000c00 tx=0x7f0594011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:15.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.674+0000 7f05a17fa700 1 -- 172.21.15.29:0/2295984218 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0594003b90 con 0x7f05a4104a10 2024-04-27T01:57:15.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.674+0000 7f05a17fa700 1 -- 172.21.15.29:0/2295984218 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f05940248b0 con 0x7f05a4104a10 2024-04-27T01:57:15.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.674+0000 7f05aa2af700 1 -- 172.21.15.29:0/2295984218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a418e200 con 0x7f05a4104a10 2024-04-27T01:57:15.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.674+0000 7f05a17fa700 1 -- 172.21.15.29:0/2295984218 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f059401a8e0 con 0x7f05a4104a10 2024-04-27T01:57:15.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.674+0000 7f05aa2af700 1 -- 172.21.15.29:0/2295984218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a418e620 con 0x7f05a4104a10 2024-04-27T01:57:15.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.675+0000 7f05a17fa700 1 -- 172.21.15.29:0/2295984218 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) v1 ==== 45073+0+0 (secure 0 0 0) 0x7f0594024440 con 0x7f05a4104a10 2024-04-27T01:57:15.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.675+0000 7f05aa2af700 1 -- 172.21.15.29:0/2295984218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a4062380 con 0x7f05a4104a10 2024-04-27T01:57:15.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.675+0000 7f05a17fa700 1 --2- 172.21.15.29:0/2295984218 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f058c0383f0 0x7f058c03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:15.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.675+0000 7f05a17fa700 1 -- 172.21.15.29:0/2295984218 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f059404bf80 con 0x7f05a4104a10 2024-04-27T01:57:15.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.675+0000 7f05a37fe700 1 --2- 172.21.15.29:0/2295984218 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f058c0383f0 0x7f058c03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:15.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.676+0000 7f05a37fe700 1 --2- 172.21.15.29:0/2295984218 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f058c0383f0 0x7f058c03a8a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0598006fd0 tx=0x7f0598006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:15.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.680+0000 7f05a17fa700 1 -- 172.21.15.29:0/2295984218 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f05940189a0 con 0x7f05a4104a10 2024-04-27T01:57:15.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.880+0000 7f05aa2af700 1 -- 172.21.15.29:0/2295984218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f05a4106880 con 0x7f05a4104a10 2024-04-27T01:57:15.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.880+0000 7f05a17fa700 1 -- 172.21.15.29:0/2295984218 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f059404bda0 con 0x7f05a4104a10 2024-04-27T01:57:15.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.883+0000 7f05aa2af700 1 -- 172.21.15.29:0/2295984218 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f058c0383f0 msgr2=0x7f058c03a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:15.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.883+0000 7f05aa2af700 1 --2- 172.21.15.29:0/2295984218 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f058c0383f0 0x7f058c03a8a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0598006fd0 tx=0x7f0598006e40 comp rx=0 tx=0).stop 2024-04-27T01:57:15.503 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.883+0000 7f05aa2af700 1 -- 172.21.15.29:0/2295984218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a4104a10 msgr2=0x7f05a418fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:15.503 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.883+0000 7f05aa2af700 1 --2- 172.21.15.29:0/2295984218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a4104a10 0x7f05a418fbb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0594000c00 tx=0x7f0594011ab0 comp rx=0 tx=0).stop 2024-04-27T01:57:15.503 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.883+0000 7f05aa2af700 1 -- 172.21.15.29:0/2295984218 shutdown_connections 2024-04-27T01:57:15.503 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.883+0000 7f05aa2af700 1 --2- 172.21.15.29:0/2295984218 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f058c0383f0 0x7f058c03a8a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:15.503 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.883+0000 7f05aa2af700 1 --2- 172.21.15.29:0/2295984218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05a4104a10 0x7f05a418fbb0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:15.503 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.883+0000 7f05aa2af700 1 -- 172.21.15.29:0/2295984218 >> 172.21.15.29:0/2295984218 conn(0x7f05a4100060 msgr2=0x7f05a4079950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:15.503 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.883+0000 7f05aa2af700 1 -- 172.21.15.29:0/2295984218 shutdown_connections 2024-04-27T01:57:15.503 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:14.883+0000 7f05aa2af700 1 -- 172.21.15.29:0/2295984218 wait complete. 2024-04-27T01:57:15.503 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling cephadm module... 2024-04-27T01:57:17.108 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:16 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/2573953325' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-27T01:57:17.562 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.941+0000 7f72cdf7e700 1 Processor -- start 2024-04-27T01:57:17.562 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.941+0000 7f72cdf7e700 1 -- start start 2024-04-27T01:57:17.562 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.941+0000 7f72cdf7e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c8105b10 0x7f72c8105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:17.562 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.941+0000 7f72cdf7e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c8106460 con 0x7f72c8105b10 2024-04-27T01:57:17.562 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.942+0000 7f72c77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c8105b10 0x7f72c8105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:17.562 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.942+0000 7f72c77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c8105b10 0x7f72c8105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58008/0 (socket says 172.21.15.29:58008) 2024-04-27T01:57:17.562 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.942+0000 7f72c77fe700 1 -- 172.21.15.29:0/2746545968 learned_addr learned my addr 172.21.15.29:0/2746545968 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:17.563 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.942+0000 7f72c77fe700 1 -- 172.21.15.29:0/2746545968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c81065a0 con 0x7f72c8105b10 2024-04-27T01:57:17.563 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.942+0000 7f72c77fe700 1 --2- 172.21.15.29:0/2746545968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c8105b10 0x7f72c8105f20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f72b80098a0 tx=0x7f72b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f82e5ab4f5d4a807 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:17.563 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.942+0000 7f72c67fc700 1 -- 172.21.15.29:0/2746545968 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72b8004180 con 0x7f72c8105b10 2024-04-27T01:57:17.563 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.942+0000 7f72c67fc700 1 -- 172.21.15.29:0/2746545968 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f72b80042e0 con 0x7f72c8105b10 2024-04-27T01:57:17.563 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.942+0000 7f72c67fc700 1 -- 172.21.15.29:0/2746545968 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72b8011a50 con 0x7f72c8105b10 2024-04-27T01:57:17.563 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.943+0000 7f72cdf7e700 1 -- 172.21.15.29:0/2746545968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c8105b10 msgr2=0x7f72c8105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:17.563 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.943+0000 7f72cdf7e700 1 --2- 172.21.15.29:0/2746545968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c8105b10 0x7f72c8105f20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f72b80098a0 tx=0x7f72b8009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:17.563 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.943+0000 7f72cdf7e700 1 -- 172.21.15.29:0/2746545968 shutdown_connections 2024-04-27T01:57:17.563 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.943+0000 7f72cdf7e700 1 --2- 172.21.15.29:0/2746545968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c8105b10 0x7f72c8105f20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:17.563 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.943+0000 7f72cdf7e700 1 -- 172.21.15.29:0/2746545968 >> 172.21.15.29:0/2746545968 conn(0x7f72c8101180 msgr2=0x7f72c8103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:17.563 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.943+0000 7f72cdf7e700 1 -- 172.21.15.29:0/2746545968 shutdown_connections 2024-04-27T01:57:17.563 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.943+0000 7f72cdf7e700 1 -- 172.21.15.29:0/2746545968 wait complete. 2024-04-27T01:57:17.563 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.944+0000 7f72cdf7e700 1 Processor -- start 2024-04-27T01:57:17.563 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.944+0000 7f72cdf7e700 1 -- start start 2024-04-27T01:57:17.564 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.944+0000 7f72cdf7e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c8105b10 0x7f72c8199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:17.564 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.944+0000 7f72cdf7e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c8199d90 con 0x7f72c8105b10 2024-04-27T01:57:17.564 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.944+0000 7f72c77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c8105b10 0x7f72c8199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:17.564 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.944+0000 7f72c77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c8105b10 0x7f72c8199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58018/0 (socket says 172.21.15.29:58018) 2024-04-27T01:57:17.564 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.944+0000 7f72c77fe700 1 -- 172.21.15.29:0/2573953325 learned_addr learned my addr 172.21.15.29:0/2573953325 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:17.564 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.944+0000 7f72c77fe700 1 -- 172.21.15.29:0/2573953325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b8009550 con 0x7f72c8105b10 2024-04-27T01:57:17.564 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.945+0000 7f72c77fe700 1 --2- 172.21.15.29:0/2573953325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c8105b10 0x7f72c8199850 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f72b80119e0 tx=0x7f72b8011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:17.564 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.945+0000 7f72c4ff9700 1 -- 172.21.15.29:0/2573953325 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72b8003b90 con 0x7f72c8105b10 2024-04-27T01:57:17.564 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.945+0000 7f72c4ff9700 1 -- 172.21.15.29:0/2573953325 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f72b801a8a0 con 0x7f72c8105b10 2024-04-27T01:57:17.564 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.945+0000 7f72cdf7e700 1 -- 172.21.15.29:0/2573953325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72c8199f90 con 0x7f72c8105b10 2024-04-27T01:57:17.564 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.945+0000 7f72c4ff9700 1 -- 172.21.15.29:0/2573953325 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72b8019930 con 0x7f72c8105b10 2024-04-27T01:57:17.564 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.945+0000 7f72cdf7e700 1 -- 172.21.15.29:0/2573953325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72c819a3b0 con 0x7f72c8105b10 2024-04-27T01:57:17.564 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.946+0000 7f72c4ff9700 1 -- 172.21.15.29:0/2573953325 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) v1 ==== 45073+0+0 (secure 0 0 0) 0x7f72b8019bf0 con 0x7f72c8105b10 2024-04-27T01:57:17.564 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.946+0000 7f72c4ff9700 1 --2- 172.21.15.29:0/2573953325 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f72b0038360 0x7f72b003a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:17.565 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.946+0000 7f72cdf7e700 1 -- 172.21.15.29:0/2573953325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72c8062380 con 0x7f72c8105b10 2024-04-27T01:57:17.565 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.946+0000 7f72c4ff9700 1 -- 172.21.15.29:0/2573953325 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f72b804c6c0 con 0x7f72c8105b10 2024-04-27T01:57:17.565 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.946+0000 7f72c6ffd700 1 --2- 172.21.15.29:0/2573953325 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f72b0038360 0x7f72b003a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:17.565 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.946+0000 7f72c6ffd700 1 --2- 172.21.15.29:0/2573953325 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f72b0038360 0x7f72b003a810 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f72bc006fd0 tx=0x7f72bc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:17.565 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:15.951+0000 7f72c4ff9700 1 -- 172.21.15.29:0/2573953325 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f72b802d440 con 0x7f72c8105b10 2024-04-27T01:57:17.565 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:16.205+0000 7f72cdf7e700 1 -- 172.21.15.29:0/2573953325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f72c819d550 con 0x7f72c8105b10 2024-04-27T01:57:17.565 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:16.780+0000 7f72c4ff9700 1 -- 172.21.15.29:0/2573953325 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 5) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f72b802c3a0 con 0x7f72c8105b10 2024-04-27T01:57:17.565 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:16.781+0000 7f72c4ff9700 1 -- 172.21.15.29:0/2573953325 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f72b802d9e0 con 0x7f72c8105b10 2024-04-27T01:57:17.565 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:16.784+0000 7f72cdf7e700 1 -- 172.21.15.29:0/2573953325 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f72b0038360 msgr2=0x7f72b003a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:17.565 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:16.784+0000 7f72cdf7e700 1 --2- 172.21.15.29:0/2573953325 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f72b0038360 0x7f72b003a810 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f72bc006fd0 tx=0x7f72bc006e40 comp rx=0 tx=0).stop 2024-04-27T01:57:17.565 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:16.784+0000 7f72cdf7e700 1 -- 172.21.15.29:0/2573953325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c8105b10 msgr2=0x7f72c8199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:17.565 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:16.784+0000 7f72cdf7e700 1 --2- 172.21.15.29:0/2573953325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c8105b10 0x7f72c8199850 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f72b80119e0 tx=0x7f72b8011ac0 comp rx=0 tx=0).stop 2024-04-27T01:57:17.565 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:16.784+0000 7f72cdf7e700 1 -- 172.21.15.29:0/2573953325 shutdown_connections 2024-04-27T01:57:17.566 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:16.784+0000 7f72cdf7e700 1 --2- 172.21.15.29:0/2573953325 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f72b0038360 0x7f72b003a810 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:17.566 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:16.784+0000 7f72cdf7e700 1 --2- 172.21.15.29:0/2573953325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c8105b10 0x7f72c8199850 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:17.566 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:16.784+0000 7f72cdf7e700 1 -- 172.21.15.29:0/2573953325 >> 172.21.15.29:0/2573953325 conn(0x7f72c8101180 msgr2=0x7f72c8101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:17.566 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:16.784+0000 7f72cdf7e700 1 -- 172.21.15.29:0/2573953325 shutdown_connections 2024-04-27T01:57:17.566 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:16.784+0000 7f72cdf7e700 1 -- 172.21.15.29:0/2573953325 wait complete. 2024-04-27T01:57:17.823 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:17 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/2573953325' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-27T01:57:17.823 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:17 smithi029 ceph-mon[26769]: mgrmap e5: smithi029.ewvdzs(active, since 6s) 2024-04-27T01:57:18.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:18 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/1582771251' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-27T01:57:18.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-04-27T01:57:18.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-27T01:57:18.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-27T01:57:18.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "active_name": "smithi029.ewvdzs", 2024-04-27T01:57:18.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-27T01:57:18.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-04-27T01:57:18.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.062+0000 7f51d2005700 1 Processor -- start 2024-04-27T01:57:18.969 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.062+0000 7f51d2005700 1 -- start start 2024-04-27T01:57:18.969 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.062+0000 7f51d2005700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51cc072aa0 0x7f51cc070fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:18.969 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.062+0000 7f51d2005700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc0714e0 con 0x7f51cc072aa0 2024-04-27T01:57:18.969 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.063+0000 7f51cb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51cc072aa0 0x7f51cc070fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:18.969 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.063+0000 7f51cb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51cc072aa0 0x7f51cc070fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50098/0 (socket says 172.21.15.29:50098) 2024-04-27T01:57:18.969 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.063+0000 7f51cb7fe700 1 -- 172.21.15.29:0/1606184490 learned_addr learned my addr 172.21.15.29:0/1606184490 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:18.969 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.063+0000 7f51cb7fe700 1 -- 172.21.15.29:0/1606184490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51cc071620 con 0x7f51cc072aa0 2024-04-27T01:57:18.969 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.064+0000 7f51cb7fe700 1 --2- 172.21.15.29:0/1606184490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51cc072aa0 0x7f51cc070fa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f51c4007840 tx=0x7f51c4007b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca75735d1fff500b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:18.969 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.064+0000 7f51ca7fc700 1 -- 172.21.15.29:0/1606184490 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51c4011070 con 0x7f51cc072aa0 2024-04-27T01:57:18.969 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.064+0000 7f51ca7fc700 1 -- 172.21.15.29:0/1606184490 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f51c4003a20 con 0x7f51cc072aa0 2024-04-27T01:57:18.969 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.065+0000 7f51d2005700 1 -- 172.21.15.29:0/1606184490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51cc072aa0 msgr2=0x7f51cc070fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:18.970 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.065+0000 7f51d2005700 1 --2- 172.21.15.29:0/1606184490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51cc072aa0 0x7f51cc070fa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f51c4007840 tx=0x7f51c4007b50 comp rx=0 tx=0).stop 2024-04-27T01:57:18.970 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.066+0000 7f51d2005700 1 -- 172.21.15.29:0/1606184490 shutdown_connections 2024-04-27T01:57:18.970 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.066+0000 7f51d2005700 1 --2- 172.21.15.29:0/1606184490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51cc072aa0 0x7f51cc070fa0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:18.970 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.066+0000 7f51d2005700 1 -- 172.21.15.29:0/1606184490 >> 172.21.15.29:0/1606184490 conn(0x7f51cc06c9d0 msgr2=0x7f51cc06ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:18.970 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.066+0000 7f51d2005700 1 -- 172.21.15.29:0/1606184490 shutdown_connections 2024-04-27T01:57:18.970 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.066+0000 7f51d2005700 1 -- 172.21.15.29:0/1606184490 wait complete. 2024-04-27T01:57:18.970 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.067+0000 7f51d2005700 1 Processor -- start 2024-04-27T01:57:18.970 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.067+0000 7f51d2005700 1 -- start start 2024-04-27T01:57:18.970 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.068+0000 7f51d2005700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51cc072aa0 0x7f51cc0898e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:18.970 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.068+0000 7f51d2005700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc089e20 con 0x7f51cc072aa0 2024-04-27T01:57:18.970 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.068+0000 7f51cb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51cc072aa0 0x7f51cc0898e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:18.970 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.068+0000 7f51cb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51cc072aa0 0x7f51cc0898e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50104/0 (socket says 172.21.15.29:50104) 2024-04-27T01:57:18.971 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.068+0000 7f51cb7fe700 1 -- 172.21.15.29:0/1582771251 learned_addr learned my addr 172.21.15.29:0/1582771251 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:18.971 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.068+0000 7f51cb7fe700 1 -- 172.21.15.29:0/1582771251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c40074f0 con 0x7f51cc072aa0 2024-04-27T01:57:18.971 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.069+0000 7f51cb7fe700 1 --2- 172.21.15.29:0/1582771251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51cc072aa0 0x7f51cc0898e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f51c4004130 tx=0x7f51c40174c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:18.971 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.069+0000 7f51c8ff9700 1 -- 172.21.15.29:0/1582771251 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51c4011050 con 0x7f51cc072aa0 2024-04-27T01:57:18.971 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.069+0000 7f51d2005700 1 -- 172.21.15.29:0/1582771251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51cc086340 con 0x7f51cc072aa0 2024-04-27T01:57:18.971 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.069+0000 7f51d2005700 1 -- 172.21.15.29:0/1582771251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51cc086760 con 0x7f51cc072aa0 2024-04-27T01:57:18.971 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.069+0000 7f51c8ff9700 1 -- 172.21.15.29:0/1582771251 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f51c400c2c0 con 0x7f51cc072aa0 2024-04-27T01:57:18.971 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.069+0000 7f51c8ff9700 1 -- 172.21.15.29:0/1582771251 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51c40047b0 con 0x7f51cc072aa0 2024-04-27T01:57:18.971 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.071+0000 7f51c8ff9700 1 -- 172.21.15.29:0/1582771251 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f51c4015070 con 0x7f51cc072aa0 2024-04-27T01:57:18.971 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.071+0000 7f51c8ff9700 1 --2- 172.21.15.29:0/1582771251 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f51b8038440 0x7f51b803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:18.971 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.071+0000 7f51c8ff9700 1 -- 172.21.15.29:0/1582771251 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f51c404b5c0 con 0x7f51cc072aa0 2024-04-27T01:57:18.971 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.071+0000 7f51caffd700 1 -- 172.21.15.29:0/1582771251 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f51b8038440 msgr2=0x7f51b803a8f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3122572856 2024-04-27T01:57:18.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.072+0000 7f51caffd700 1 --2- 172.21.15.29:0/1582771251 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f51b8038440 0x7f51b803a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-27T01:57:18.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.075+0000 7f51d2005700 1 -- 172.21.15.29:0/1582771251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51ac005320 con 0x7f51cc072aa0 2024-04-27T01:57:18.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.086+0000 7f51c8ff9700 1 -- 172.21.15.29:0/1582771251 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f51c4027cf0 con 0x7f51cc072aa0 2024-04-27T01:57:18.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.273+0000 7f51caffd700 1 -- 172.21.15.29:0/1582771251 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f51b8038440 msgr2=0x7f51b803a8f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3122572856 2024-04-27T01:57:18.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.273+0000 7f51caffd700 1 --2- 172.21.15.29:0/1582771251 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f51b8038440 0x7f51b803a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-27T01:57:18.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.355+0000 7f51d2005700 1 -- 172.21.15.29:0/1582771251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f51ac006200 con 0x7f51cc072aa0 2024-04-27T01:57:18.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.356+0000 7f51c8ff9700 1 -- 172.21.15.29:0/1582771251 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f51c4027b10 con 0x7f51cc072aa0 2024-04-27T01:57:18.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.358+0000 7f51d2005700 1 -- 172.21.15.29:0/1582771251 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f51b8038440 msgr2=0x7f51b803a8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-27T01:57:18.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.358+0000 7f51d2005700 1 --2- 172.21.15.29:0/1582771251 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f51b8038440 0x7f51b803a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:18.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.358+0000 7f51d2005700 1 -- 172.21.15.29:0/1582771251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51cc072aa0 msgr2=0x7f51cc0898e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:18.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.358+0000 7f51d2005700 1 --2- 172.21.15.29:0/1582771251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51cc072aa0 0x7f51cc0898e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f51c4004130 tx=0x7f51c40174c0 comp rx=0 tx=0).stop 2024-04-27T01:57:18.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.358+0000 7f51d2005700 1 -- 172.21.15.29:0/1582771251 shutdown_connections 2024-04-27T01:57:18.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.358+0000 7f51d2005700 1 --2- 172.21.15.29:0/1582771251 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f51b8038440 0x7f51b803a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:18.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.358+0000 7f51d2005700 1 --2- 172.21.15.29:0/1582771251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51cc072aa0 0x7f51cc0898e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:18.973 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.358+0000 7f51d2005700 1 -- 172.21.15.29:0/1582771251 >> 172.21.15.29:0/1582771251 conn(0x7f51cc06c9d0 msgr2=0x7f51cc06d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:18.973 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.358+0000 7f51d2005700 1 -- 172.21.15.29:0/1582771251 shutdown_connections 2024-04-27T01:57:18.973 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:18.358+0000 7f51d2005700 1 -- 172.21.15.29:0/1582771251 wait complete. 2024-04-27T01:57:18.973 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for the mgr to restart... 2024-04-27T01:57:18.973 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr epoch 5... 2024-04-27T01:57:23.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:23 smithi029 ceph-mon[26769]: Active manager daemon smithi029.ewvdzs restarted 2024-04-27T01:57:23.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:23 smithi029 ceph-mon[26769]: Activating manager daemon smithi029.ewvdzs 2024-04-27T01:57:23.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:23 smithi029 ceph-mon[26769]: osdmap e2: 0 total, 0 up, 0 in 2024-04-27T01:57:23.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:23 smithi029 ceph-mon[26769]: mgrmap e6: smithi029.ewvdzs(active, starting, since 0.00335979s) 2024-04-27T01:57:23.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:23 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-04-27T01:57:23.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:23 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mgr metadata", "who": "smithi029.ewvdzs", "id": "smithi029.ewvdzs"}]: dispatch 2024-04-27T01:57:23.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:23 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T01:57:23.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:23 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T01:57:23.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:23 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T01:57:23.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:23 smithi029 ceph-mon[26769]: Manager daemon smithi029.ewvdzs is now available 2024-04-27T01:57:23.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:23 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:23.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:23 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:23.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:23 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:57:23.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:23 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:57:23.953 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:23 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:57:23.953 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:23 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.ewvdzs/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:57:23.953 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:23 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.ewvdzs/trash_purge_schedule"}]: dispatch 2024-04-27T01:57:24.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:24 smithi029 ceph-mon[26769]: Found migration_current of "None". Setting to last migration. 2024-04-27T01:57:24.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:24 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:24.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:24 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:24.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:24 smithi029 ceph-mon[26769]: mgrmap e7: smithi029.ewvdzs(active, since 1.00544s) 2024-04-27T01:57:25.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-04-27T01:57:25.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-27T01:57:25.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-27T01:57:25.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-04-27T01:57:25.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.445+0000 7f0e33d0c700 1 Processor -- start 2024-04-27T01:57:25.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.445+0000 7f0e33d0c700 1 -- start start 2024-04-27T01:57:25.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.446+0000 7f0e33d0c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e2c105b60 0x7f0e2c105f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:25.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.446+0000 7f0e33d0c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c1064b0 con 0x7f0e2c105b60 2024-04-27T01:57:25.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.446+0000 7f0e31aa8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e2c105b60 0x7f0e2c105f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:25.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.446+0000 7f0e31aa8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e2c105b60 0x7f0e2c105f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50112/0 (socket says 172.21.15.29:50112) 2024-04-27T01:57:25.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.446+0000 7f0e31aa8700 1 -- 172.21.15.29:0/3035574505 learned_addr learned my addr 172.21.15.29:0/3035574505 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:25.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.446+0000 7f0e31aa8700 1 -- 172.21.15.29:0/3035574505 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2c1065f0 con 0x7f0e2c105b60 2024-04-27T01:57:25.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.446+0000 7f0e31aa8700 1 --2- 172.21.15.29:0/3035574505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e2c105b60 0x7f0e2c105f70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0e280098a0 tx=0x7f0e28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f76461bbe4976 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:25.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.446+0000 7f0e30aa6700 1 -- 172.21.15.29:0/3035574505 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e28004180 con 0x7f0e2c105b60 2024-04-27T01:57:25.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.447+0000 7f0e30aa6700 1 -- 172.21.15.29:0/3035574505 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0e280042e0 con 0x7f0e2c105b60 2024-04-27T01:57:25.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.447+0000 7f0e30aa6700 1 -- 172.21.15.29:0/3035574505 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e28011a50 con 0x7f0e2c105b60 2024-04-27T01:57:25.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.447+0000 7f0e33d0c700 1 -- 172.21.15.29:0/3035574505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e2c105b60 msgr2=0x7f0e2c105f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:25.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.447+0000 7f0e33d0c700 1 --2- 172.21.15.29:0/3035574505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e2c105b60 0x7f0e2c105f70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0e280098a0 tx=0x7f0e28009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:25.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.447+0000 7f0e33d0c700 1 -- 172.21.15.29:0/3035574505 shutdown_connections 2024-04-27T01:57:25.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.447+0000 7f0e33d0c700 1 --2- 172.21.15.29:0/3035574505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e2c105b60 0x7f0e2c105f70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:25.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.447+0000 7f0e33d0c700 1 -- 172.21.15.29:0/3035574505 >> 172.21.15.29:0/3035574505 conn(0x7f0e2c1011b0 msgr2=0x7f0e2c1035e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:25.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.447+0000 7f0e33d0c700 1 -- 172.21.15.29:0/3035574505 shutdown_connections 2024-04-27T01:57:25.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.447+0000 7f0e33d0c700 1 -- 172.21.15.29:0/3035574505 wait complete. 2024-04-27T01:57:25.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.448+0000 7f0e33d0c700 1 Processor -- start 2024-04-27T01:57:25.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.448+0000 7f0e33d0c700 1 -- start start 2024-04-27T01:57:25.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.448+0000 7f0e33d0c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e2c105b60 0x7f0e2c199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:25.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.448+0000 7f0e33d0c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c199da0 con 0x7f0e2c105b60 2024-04-27T01:57:25.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.448+0000 7f0e31aa8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e2c105b60 0x7f0e2c199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:25.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.448+0000 7f0e31aa8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e2c105b60 0x7f0e2c199860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50122/0 (socket says 172.21.15.29:50122) 2024-04-27T01:57:25.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.448+0000 7f0e31aa8700 1 -- 172.21.15.29:0/1354498556 learned_addr learned my addr 172.21.15.29:0/1354498556 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:25.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.449+0000 7f0e31aa8700 1 -- 172.21.15.29:0/1354498556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e28009550 con 0x7f0e2c105b60 2024-04-27T01:57:25.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.449+0000 7f0e31aa8700 1 --2- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e2c105b60 0x7f0e2c199860 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0e280119e0 tx=0x7f0e28011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:25.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.449+0000 7f0e22ffd700 1 -- 172.21.15.29:0/1354498556 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e28003b90 con 0x7f0e2c105b60 2024-04-27T01:57:25.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.449+0000 7f0e22ffd700 1 -- 172.21.15.29:0/1354498556 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0e2801a8a0 con 0x7f0e2c105b60 2024-04-27T01:57:25.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.449+0000 7f0e33d0c700 1 -- 172.21.15.29:0/1354498556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2c199fa0 con 0x7f0e2c105b60 2024-04-27T01:57:25.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.449+0000 7f0e22ffd700 1 -- 172.21.15.29:0/1354498556 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e28019930 con 0x7f0e2c105b60 2024-04-27T01:57:25.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.449+0000 7f0e33d0c700 1 -- 172.21.15.29:0/1354498556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2c19a3c0 con 0x7f0e2c105b60 2024-04-27T01:57:25.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.450+0000 7f0e22ffd700 1 -- 172.21.15.29:0/1354498556 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f0e28019ba0 con 0x7f0e2c105b60 2024-04-27T01:57:25.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.450+0000 7f0e22ffd700 1 --2- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f0e18038400 0x7f0e1803a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:25.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.450+0000 7f0e22ffd700 1 -- 172.21.15.29:0/1354498556 --> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0e1803afc0 con 0x7f0e18038400 2024-04-27T01:57:25.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.450+0000 7f0e22ffd700 1 -- 172.21.15.29:0/1354498556 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0e2804c6f0 con 0x7f0e2c105b60 2024-04-27T01:57:25.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.450+0000 7f0e312a7700 1 -- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f0e18038400 msgr2=0x7f0e1803a8b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3122572856 2024-04-27T01:57:25.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.450+0000 7f0e312a7700 1 --2- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f0e18038400 0x7f0e1803a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-27T01:57:25.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.651+0000 7f0e312a7700 1 -- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f0e18038400 msgr2=0x7f0e1803a8b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3122572856 2024-04-27T01:57:25.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:19.651+0000 7f0e312a7700 1 --2- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f0e18038400 0x7f0e1803a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-27T01:57:25.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:20.052+0000 7f0e312a7700 1 -- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f0e18038400 msgr2=0x7f0e1803a8b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3122572856 2024-04-27T01:57:25.257 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:20.052+0000 7f0e312a7700 1 --2- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f0e18038400 0x7f0e1803a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-27T01:57:25.257 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:20.853+0000 7f0e312a7700 1 -- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f0e18038400 msgr2=0x7f0e1803a8b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3122572856 2024-04-27T01:57:25.257 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:20.853+0000 7f0e312a7700 1 --2- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f0e18038400 0x7f0e1803a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-27T01:57:25.257 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:22.454+0000 7f0e312a7700 1 -- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f0e18038400 msgr2=0x7f0e1803a8b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3122572856 2024-04-27T01:57:25.257 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:22.454+0000 7f0e312a7700 1 --2- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f0e18038400 0x7f0e1803a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-27T01:57:25.257 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:23.639+0000 7f0e22ffd700 1 -- 172.21.15.29:0/1354498556 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mgrmap(e 6) v1 ==== 44851+0+0 (secure 0 0 0) 0x7f0e2802c3a0 con 0x7f0e2c105b60 2024-04-27T01:57:25.257 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:23.639+0000 7f0e22ffd700 1 -- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f0e18038400 msgr2=0x7f0e1803a8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-27T01:57:25.257 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:23.639+0000 7f0e22ffd700 1 --2- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f0e18038400 0x7f0e1803a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:25.257 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:24.640+0000 7f0e22ffd700 1 -- 172.21.15.29:0/1354498556 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 7) v1 ==== 44978+0+0 (secure 0 0 0) 0x7f0e2804e100 con 0x7f0e2c105b60 2024-04-27T01:57:25.257 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:24.641+0000 7f0e22ffd700 1 --2- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f0e1803bd20 0x7f0e1803e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:25.257 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:24.641+0000 7f0e22ffd700 1 -- 172.21.15.29:0/1354498556 --> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0e1803afc0 con 0x7f0e1803bd20 2024-04-27T01:57:25.257 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:24.641+0000 7f0e312a7700 1 --2- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f0e1803bd20 0x7f0e1803e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:25.257 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:24.642+0000 7f0e312a7700 1 --2- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f0e1803bd20 0x7f0e1803e100 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c003a10 tx=0x7f0e1c0092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:25.257 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:24.642+0000 7f0e22ffd700 1 -- 172.21.15.29:0/1354498556 <== mgr.14120 v2:172.21.15.29:6800/3873642219 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f0e1803afc0 con 0x7f0e1803bd20 2024-04-27T01:57:25.257 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:24.648+0000 7f0e33d0c700 1 -- 172.21.15.29:0/1354498556 --> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f0e2c19aba0 con 0x7f0e1803bd20 2024-04-27T01:57:25.258 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:24.649+0000 7f0e22ffd700 1 -- 172.21.15.29:0/1354498556 <== mgr.14120 v2:172.21.15.29:6800/3873642219 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f0e2c19aba0 con 0x7f0e1803bd20 2024-04-27T01:57:25.258 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:24.649+0000 7f0e33d0c700 1 -- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f0e1803bd20 msgr2=0x7f0e1803e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:25.258 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:24.649+0000 7f0e33d0c700 1 --2- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f0e1803bd20 0x7f0e1803e100 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c003a10 tx=0x7f0e1c0092b0 comp rx=0 tx=0).stop 2024-04-27T01:57:25.258 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:24.649+0000 7f0e33d0c700 1 -- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e2c105b60 msgr2=0x7f0e2c199860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:25.258 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:24.649+0000 7f0e33d0c700 1 --2- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e2c105b60 0x7f0e2c199860 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0e280119e0 tx=0x7f0e28011ac0 comp rx=0 tx=0).stop 2024-04-27T01:57:25.258 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:24.649+0000 7f0e33d0c700 1 -- 172.21.15.29:0/1354498556 shutdown_connections 2024-04-27T01:57:25.258 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:24.649+0000 7f0e33d0c700 1 --2- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f0e1803bd20 0x7f0e1803e100 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:25.258 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:24.649+0000 7f0e33d0c700 1 --2- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e2c105b60 0x7f0e2c199860 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:25.258 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:24.649+0000 7f0e33d0c700 1 --2- 172.21.15.29:0/1354498556 >> [v2:172.21.15.29:6800/3122572856,v1:172.21.15.29:6801/3122572856] conn(0x7f0e18038400 0x7f0e1803a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:25.258 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:24.649+0000 7f0e33d0c700 1 -- 172.21.15.29:0/1354498556 >> 172.21.15.29:0/1354498556 conn(0x7f0e2c1011b0 msgr2=0x7f0e2c102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:25.258 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:24.649+0000 7f0e33d0c700 1 -- 172.21.15.29:0/1354498556 shutdown_connections 2024-04-27T01:57:25.258 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:24.649+0000 7f0e33d0c700 1 -- 172.21.15.29:0/1354498556 wait complete. 2024-04-27T01:57:25.258 INFO:teuthology.orchestra.run.smithi029.stdout:mgr epoch 5 is available 2024-04-27T01:57:25.258 INFO:teuthology.orchestra.run.smithi029.stdout:Setting orchestrator backend to cephadm... 2024-04-27T01:57:25.911 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:25 smithi029 ceph-mon[26769]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-27T01:57:25.911 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:25 smithi029 ceph-mon[26769]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-27T01:57:25.911 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:25 smithi029 ceph-mon[26769]: [27/Apr/2024:01:57:24] ENGINE Bus STARTING 2024-04-27T01:57:25.911 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:25 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:57:26.514 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.705+0000 7fa461af6700 1 Processor -- start 2024-04-27T01:57:26.514 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.705+0000 7fa461af6700 1 -- start start 2024-04-27T01:57:26.514 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.705+0000 7fa461af6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c104a00 0x7fa45c104e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:26.514 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.706+0000 7fa461af6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c105350 con 0x7fa45c104a00 2024-04-27T01:57:26.514 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.706+0000 7fa45b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c104a00 0x7fa45c104e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:26.514 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.706+0000 7fa45b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c104a00 0x7fa45c104e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50192/0 (socket says 172.21.15.29:50192) 2024-04-27T01:57:26.515 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.706+0000 7fa45b7fe700 1 -- 172.21.15.29:0/385550921 learned_addr learned my addr 172.21.15.29:0/385550921 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:26.515 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.706+0000 7fa45b7fe700 1 -- 172.21.15.29:0/385550921 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa45c105490 con 0x7fa45c104a00 2024-04-27T01:57:26.515 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.706+0000 7fa45b7fe700 1 --2- 172.21.15.29:0/385550921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c104a00 0x7fa45c104e10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa44c0098a0 tx=0x7fa44c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74cdf164a60ff49b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:26.515 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.706+0000 7fa45a7fc700 1 -- 172.21.15.29:0/385550921 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa44c004180 con 0x7fa45c104a00 2024-04-27T01:57:26.515 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.706+0000 7fa45a7fc700 1 -- 172.21.15.29:0/385550921 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa44c0042e0 con 0x7fa45c104a00 2024-04-27T01:57:26.515 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.706+0000 7fa45a7fc700 1 -- 172.21.15.29:0/385550921 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa44c011a50 con 0x7fa45c104a00 2024-04-27T01:57:26.515 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.707+0000 7fa461af6700 1 -- 172.21.15.29:0/385550921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c104a00 msgr2=0x7fa45c104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:26.515 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.707+0000 7fa461af6700 1 --2- 172.21.15.29:0/385550921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c104a00 0x7fa45c104e10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa44c0098a0 tx=0x7fa44c009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:26.515 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.707+0000 7fa461af6700 1 -- 172.21.15.29:0/385550921 shutdown_connections 2024-04-27T01:57:26.515 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.707+0000 7fa461af6700 1 --2- 172.21.15.29:0/385550921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c104a00 0x7fa45c104e10 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:26.515 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.707+0000 7fa461af6700 1 -- 172.21.15.29:0/385550921 >> 172.21.15.29:0/385550921 conn(0x7fa45c100070 msgr2=0x7fa45c102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:26.515 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.707+0000 7fa461af6700 1 -- 172.21.15.29:0/385550921 shutdown_connections 2024-04-27T01:57:26.515 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.707+0000 7fa461af6700 1 -- 172.21.15.29:0/385550921 wait complete. 2024-04-27T01:57:26.516 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.708+0000 7fa461af6700 1 Processor -- start 2024-04-27T01:57:26.516 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.708+0000 7fa461af6700 1 -- start start 2024-04-27T01:57:26.516 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.708+0000 7fa461af6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c104a00 0x7fa45c199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:26.516 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.708+0000 7fa461af6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c199d50 con 0x7fa45c104a00 2024-04-27T01:57:26.516 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.708+0000 7fa45b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c104a00 0x7fa45c199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:26.516 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.708+0000 7fa45b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c104a00 0x7fa45c199810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50202/0 (socket says 172.21.15.29:50202) 2024-04-27T01:57:26.516 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.708+0000 7fa45b7fe700 1 -- 172.21.15.29:0/3330139349 learned_addr learned my addr 172.21.15.29:0/3330139349 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:26.516 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.708+0000 7fa45b7fe700 1 -- 172.21.15.29:0/3330139349 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44c009550 con 0x7fa45c104a00 2024-04-27T01:57:26.516 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.708+0000 7fa45b7fe700 1 --2- 172.21.15.29:0/3330139349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c104a00 0x7fa45c199810 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa44c000c00 tx=0x7fa44c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:26.516 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.709+0000 7fa458ff9700 1 -- 172.21.15.29:0/3330139349 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa44c003b90 con 0x7fa45c104a00 2024-04-27T01:57:26.516 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.709+0000 7fa461af6700 1 -- 172.21.15.29:0/3330139349 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa45c199f50 con 0x7fa45c104a00 2024-04-27T01:57:26.516 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.709+0000 7fa458ff9700 1 -- 172.21.15.29:0/3330139349 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa44c0248b0 con 0x7fa45c104a00 2024-04-27T01:57:26.516 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.709+0000 7fa458ff9700 1 -- 172.21.15.29:0/3330139349 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa44c01a8e0 con 0x7fa45c104a00 2024-04-27T01:57:26.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.709+0000 7fa461af6700 1 -- 172.21.15.29:0/3330139349 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa45c19a370 con 0x7fa45c104a00 2024-04-27T01:57:26.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.710+0000 7fa458ff9700 1 -- 172.21.15.29:0/3330139349 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fa44c01aba0 con 0x7fa45c104a00 2024-04-27T01:57:26.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.710+0000 7fa461af6700 1 -- 172.21.15.29:0/3330139349 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa45c062380 con 0x7fa45c104a00 2024-04-27T01:57:26.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.710+0000 7fa458ff9700 1 --2- 172.21.15.29:0/3330139349 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fa4440383f0 0x7fa44403a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:26.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.710+0000 7fa458ff9700 1 -- 172.21.15.29:0/3330139349 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa44c04c5a0 con 0x7fa45c104a00 2024-04-27T01:57:26.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.710+0000 7fa45affd700 1 --2- 172.21.15.29:0/3330139349 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fa4440383f0 0x7fa44403a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:26.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.710+0000 7fa45affd700 1 --2- 172.21.15.29:0/3330139349 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fa4440383f0 0x7fa44403a8a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa450006fd0 tx=0x7fa450006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:26.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.715+0000 7fa458ff9700 1 -- 172.21.15.29:0/3330139349 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa44c024db0 con 0x7fa45c104a00 2024-04-27T01:57:26.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.929+0000 7fa461af6700 1 -- 172.21.15.29:0/3330139349 --> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fa45c101500 con 0x7fa4440383f0 2024-04-27T01:57:26.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.936+0000 7fa458ff9700 1 -- 172.21.15.29:0/3330139349 <== mgr.14120 v2:172.21.15.29:6800/3873642219 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa45c101500 con 0x7fa4440383f0 2024-04-27T01:57:26.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.938+0000 7fa461af6700 1 -- 172.21.15.29:0/3330139349 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fa4440383f0 msgr2=0x7fa44403a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:26.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.938+0000 7fa461af6700 1 --2- 172.21.15.29:0/3330139349 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fa4440383f0 0x7fa44403a8a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa450006fd0 tx=0x7fa450006e40 comp rx=0 tx=0).stop 2024-04-27T01:57:26.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.938+0000 7fa461af6700 1 -- 172.21.15.29:0/3330139349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c104a00 msgr2=0x7fa45c199810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:26.518 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.938+0000 7fa461af6700 1 --2- 172.21.15.29:0/3330139349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c104a00 0x7fa45c199810 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa44c000c00 tx=0x7fa44c011ab0 comp rx=0 tx=0).stop 2024-04-27T01:57:26.518 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.938+0000 7fa461af6700 1 -- 172.21.15.29:0/3330139349 shutdown_connections 2024-04-27T01:57:26.518 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.938+0000 7fa461af6700 1 --2- 172.21.15.29:0/3330139349 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fa4440383f0 0x7fa44403a8a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:26.518 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.938+0000 7fa461af6700 1 --2- 172.21.15.29:0/3330139349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c104a00 0x7fa45c199810 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:26.518 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.938+0000 7fa461af6700 1 -- 172.21.15.29:0/3330139349 >> 172.21.15.29:0/3330139349 conn(0x7fa45c100070 msgr2=0x7fa45c100df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:26.518 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.939+0000 7fa461af6700 1 -- 172.21.15.29:0/3330139349 shutdown_connections 2024-04-27T01:57:26.518 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:25.939+0000 7fa461af6700 1 -- 172.21.15.29:0/3330139349 wait complete. 2024-04-27T01:57:26.770 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:26 smithi029 ceph-mon[26769]: [27/Apr/2024:01:57:24] ENGINE Serving on https://172.21.15.29:7150 2024-04-27T01:57:26.771 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:26 smithi029 ceph-mon[26769]: [27/Apr/2024:01:57:24] ENGINE Serving on http://172.21.15.29:8765 2024-04-27T01:57:26.771 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:26 smithi029 ceph-mon[26769]: [27/Apr/2024:01:57:24] ENGINE Bus STARTED 2024-04-27T01:57:26.771 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:26 smithi029 ceph-mon[26769]: mgrmap e8: smithi029.ewvdzs(active, since 2s) 2024-04-27T01:57:26.771 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:26 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:26.771 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:26 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:57:27.822 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:27 smithi029 ceph-mon[26769]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:57:27.831 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-27T01:57:27.831 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.024+0000 7fa52934d700 1 Processor -- start 2024-04-27T01:57:27.831 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.024+0000 7fa52934d700 1 -- start start 2024-04-27T01:57:27.831 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.024+0000 7fa52934d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa52407bdf0 0x7fa52407a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:27.831 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.024+0000 7fa52934d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52407a830 con 0x7fa52407bdf0 2024-04-27T01:57:27.831 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.024+0000 7fa522ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa52407bdf0 0x7fa52407a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:27.831 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.024+0000 7fa522ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa52407bdf0 0x7fa52407a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42560/0 (socket says 172.21.15.29:42560) 2024-04-27T01:57:27.831 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.024+0000 7fa522ffd700 1 -- 172.21.15.29:0/443201045 learned_addr learned my addr 172.21.15.29:0/443201045 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:27.831 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.025+0000 7fa522ffd700 1 -- 172.21.15.29:0/443201045 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52407a970 con 0x7fa52407bdf0 2024-04-27T01:57:27.832 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.025+0000 7fa522ffd700 1 --2- 172.21.15.29:0/443201045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa52407bdf0 0x7fa52407a2f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa5140098a0 tx=0x7fa514009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c19781177cddae5c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:27.832 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.025+0000 7fa521ffb700 1 -- 172.21.15.29:0/443201045 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa514004180 con 0x7fa52407bdf0 2024-04-27T01:57:27.832 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.025+0000 7fa521ffb700 1 -- 172.21.15.29:0/443201045 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa5140042e0 con 0x7fa52407bdf0 2024-04-27T01:57:27.832 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.025+0000 7fa521ffb700 1 -- 172.21.15.29:0/443201045 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa514011a50 con 0x7fa52407bdf0 2024-04-27T01:57:27.832 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.026+0000 7fa52934d700 1 -- 172.21.15.29:0/443201045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa52407bdf0 msgr2=0x7fa52407a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:27.832 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.026+0000 7fa52934d700 1 --2- 172.21.15.29:0/443201045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa52407bdf0 0x7fa52407a2f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa5140098a0 tx=0x7fa514009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:27.832 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.026+0000 7fa52934d700 1 -- 172.21.15.29:0/443201045 shutdown_connections 2024-04-27T01:57:27.832 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.026+0000 7fa52934d700 1 --2- 172.21.15.29:0/443201045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa52407bdf0 0x7fa52407a2f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:27.832 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.026+0000 7fa52934d700 1 -- 172.21.15.29:0/443201045 >> 172.21.15.29:0/443201045 conn(0x7fa524101180 msgr2=0x7fa524103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:27.832 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.026+0000 7fa52934d700 1 -- 172.21.15.29:0/443201045 shutdown_connections 2024-04-27T01:57:27.832 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.026+0000 7fa52934d700 1 -- 172.21.15.29:0/443201045 wait complete. 2024-04-27T01:57:27.832 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.026+0000 7fa52934d700 1 Processor -- start 2024-04-27T01:57:27.832 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.027+0000 7fa52934d700 1 -- start start 2024-04-27T01:57:27.833 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.027+0000 7fa52934d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa52407bdf0 0x7fa52419ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:27.833 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.027+0000 7fa52934d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52419bfb0 con 0x7fa52407bdf0 2024-04-27T01:57:27.833 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.027+0000 7fa522ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa52407bdf0 0x7fa52419ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:27.833 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.027+0000 7fa522ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa52407bdf0 0x7fa52419ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42568/0 (socket says 172.21.15.29:42568) 2024-04-27T01:57:27.833 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.027+0000 7fa522ffd700 1 -- 172.21.15.29:0/3085638539 learned_addr learned my addr 172.21.15.29:0/3085638539 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:27.833 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.027+0000 7fa522ffd700 1 -- 172.21.15.29:0/3085638539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa514009550 con 0x7fa52407bdf0 2024-04-27T01:57:27.833 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.027+0000 7fa522ffd700 1 --2- 172.21.15.29:0/3085638539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa52407bdf0 0x7fa52419ba70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa514000c00 tx=0x7fa514011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:27.833 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.027+0000 7fa50bfff700 1 -- 172.21.15.29:0/3085638539 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa514003b90 con 0x7fa52407bdf0 2024-04-27T01:57:27.833 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.027+0000 7fa50bfff700 1 -- 172.21.15.29:0/3085638539 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa5140248b0 con 0x7fa52407bdf0 2024-04-27T01:57:27.833 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.027+0000 7fa52934d700 1 -- 172.21.15.29:0/3085638539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa52419c1b0 con 0x7fa52407bdf0 2024-04-27T01:57:27.833 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.027+0000 7fa50bfff700 1 -- 172.21.15.29:0/3085638539 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa51401a8e0 con 0x7fa52407bdf0 2024-04-27T01:57:27.833 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.027+0000 7fa52934d700 1 -- 172.21.15.29:0/3085638539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa52419c5d0 con 0x7fa52407bdf0 2024-04-27T01:57:27.833 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.028+0000 7fa50bfff700 1 -- 172.21.15.29:0/3085638539 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fa51401aba0 con 0x7fa52407bdf0 2024-04-27T01:57:27.833 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.028+0000 7fa52934d700 1 -- 172.21.15.29:0/3085638539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa52410bde0 con 0x7fa52407bdf0 2024-04-27T01:57:27.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.029+0000 7fa50bfff700 1 --2- 172.21.15.29:0/3085638539 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fa50c0383f0 0x7fa50c03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:27.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.029+0000 7fa50bfff700 1 -- 172.21.15.29:0/3085638539 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa51404c5d0 con 0x7fa52407bdf0 2024-04-27T01:57:27.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.029+0000 7fa5227fc700 1 --2- 172.21.15.29:0/3085638539 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fa50c0383f0 0x7fa50c03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:27.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.029+0000 7fa5227fc700 1 --2- 172.21.15.29:0/3085638539 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fa50c0383f0 0x7fa50c03a8a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa518006fd0 tx=0x7fa518006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:27.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.034+0000 7fa50bfff700 1 -- 172.21.15.29:0/3085638539 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa5140233a0 con 0x7fa52407bdf0 2024-04-27T01:57:27.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.235+0000 7fa52934d700 1 -- 172.21.15.29:0/3085638539 --> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fa52419f430 con 0x7fa50c0383f0 2024-04-27T01:57:27.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.236+0000 7fa50bfff700 1 -- 172.21.15.29:0/3085638539 <== mgr.14120 v2:172.21.15.29:6800/3873642219 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fa52419f430 con 0x7fa50c0383f0 2024-04-27T01:57:27.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.238+0000 7fa52934d700 1 -- 172.21.15.29:0/3085638539 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fa50c0383f0 msgr2=0x7fa50c03a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:27.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.238+0000 7fa52934d700 1 --2- 172.21.15.29:0/3085638539 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fa50c0383f0 0x7fa50c03a8a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa518006fd0 tx=0x7fa518006e40 comp rx=0 tx=0).stop 2024-04-27T01:57:27.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.238+0000 7fa52934d700 1 -- 172.21.15.29:0/3085638539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa52407bdf0 msgr2=0x7fa52419ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:27.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.238+0000 7fa52934d700 1 --2- 172.21.15.29:0/3085638539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa52407bdf0 0x7fa52419ba70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa514000c00 tx=0x7fa514011ab0 comp rx=0 tx=0).stop 2024-04-27T01:57:27.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.238+0000 7fa52934d700 1 -- 172.21.15.29:0/3085638539 shutdown_connections 2024-04-27T01:57:27.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.238+0000 7fa52934d700 1 --2- 172.21.15.29:0/3085638539 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fa50c0383f0 0x7fa50c03a8a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:27.835 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.238+0000 7fa52934d700 1 --2- 172.21.15.29:0/3085638539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa52407bdf0 0x7fa52419ba70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:27.835 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.238+0000 7fa52934d700 1 -- 172.21.15.29:0/3085638539 >> 172.21.15.29:0/3085638539 conn(0x7fa524101180 msgr2=0x7fa524102820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:27.835 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.238+0000 7fa52934d700 1 -- 172.21.15.29:0/3085638539 shutdown_connections 2024-04-27T01:57:27.835 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:27.238+0000 7fa52934d700 1 -- 172.21.15.29:0/3085638539 wait complete. 2024-04-27T01:57:27.835 INFO:teuthology.orchestra.run.smithi029.stdout:Generating ssh key... 2024-04-27T01:57:28.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:28 smithi029 ceph-mon[26769]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:57:28.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:28 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:28.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:28 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:29.252 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.265+0000 7f1b9c369700 1 Processor -- start 2024-04-27T01:57:29.252 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.265+0000 7f1b9c369700 1 -- start start 2024-04-27T01:57:29.252 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.265+0000 7f1b9c369700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94106600 0x7f1b941089e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:29.252 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.265+0000 7f1b9c369700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b94079d70 con 0x7f1b94106600 2024-04-27T01:57:29.252 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.265+0000 7f1b9a105700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94106600 0x7f1b941089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:29.252 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.265+0000 7f1b9a105700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94106600 0x7f1b941089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42580/0 (socket says 172.21.15.29:42580) 2024-04-27T01:57:29.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.265+0000 7f1b9a105700 1 -- 172.21.15.29:0/3498859163 learned_addr learned my addr 172.21.15.29:0/3498859163 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:29.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.266+0000 7f1b9a105700 1 -- 172.21.15.29:0/3498859163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b94079eb0 con 0x7f1b94106600 2024-04-27T01:57:29.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.266+0000 7f1b9a105700 1 --2- 172.21.15.29:0/3498859163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94106600 0x7f1b941089e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1b900098a0 tx=0x7f1b90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb37ee29bd0d235f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:29.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.266+0000 7f1b99103700 1 -- 172.21.15.29:0/3498859163 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b90004180 con 0x7f1b94106600 2024-04-27T01:57:29.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.266+0000 7f1b99103700 1 -- 172.21.15.29:0/3498859163 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1b900042e0 con 0x7f1b94106600 2024-04-27T01:57:29.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.266+0000 7f1b99103700 1 -- 172.21.15.29:0/3498859163 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b90011a50 con 0x7f1b94106600 2024-04-27T01:57:29.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.267+0000 7f1b9c369700 1 -- 172.21.15.29:0/3498859163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94106600 msgr2=0x7f1b941089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:29.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.267+0000 7f1b9c369700 1 --2- 172.21.15.29:0/3498859163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94106600 0x7f1b941089e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1b900098a0 tx=0x7f1b90009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:29.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.267+0000 7f1b9c369700 1 -- 172.21.15.29:0/3498859163 shutdown_connections 2024-04-27T01:57:29.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.267+0000 7f1b9c369700 1 --2- 172.21.15.29:0/3498859163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94106600 0x7f1b941089e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:29.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.267+0000 7f1b9c369700 1 -- 172.21.15.29:0/3498859163 >> 172.21.15.29:0/3498859163 conn(0x7f1b94100080 msgr2=0x7f1b94102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:29.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.267+0000 7f1b9c369700 1 -- 172.21.15.29:0/3498859163 shutdown_connections 2024-04-27T01:57:29.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.267+0000 7f1b9c369700 1 -- 172.21.15.29:0/3498859163 wait complete. 2024-04-27T01:57:29.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.268+0000 7f1b9c369700 1 Processor -- start 2024-04-27T01:57:29.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.268+0000 7f1b9c369700 1 -- start start 2024-04-27T01:57:29.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.268+0000 7f1b9c369700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94106600 0x7f1b94105f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:29.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.268+0000 7f1b9c369700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b941064a0 con 0x7f1b94106600 2024-04-27T01:57:29.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.268+0000 7f1b9a105700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94106600 0x7f1b94105f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:29.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.268+0000 7f1b9a105700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94106600 0x7f1b94105f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42586/0 (socket says 172.21.15.29:42586) 2024-04-27T01:57:29.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.268+0000 7f1b9a105700 1 -- 172.21.15.29:0/1899281840 learned_addr learned my addr 172.21.15.29:0/1899281840 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:29.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.268+0000 7f1b9a105700 1 -- 172.21.15.29:0/1899281840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b90009550 con 0x7f1b94106600 2024-04-27T01:57:29.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.268+0000 7f1b9a105700 1 --2- 172.21.15.29:0/1899281840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94106600 0x7f1b94105f60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1b90006af0 tx=0x7f1b90011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:29.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.268+0000 7f1b8b7fe700 1 -- 172.21.15.29:0/1899281840 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b90003b90 con 0x7f1b94106600 2024-04-27T01:57:29.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.269+0000 7f1b8b7fe700 1 -- 172.21.15.29:0/1899281840 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1b9001a8a0 con 0x7f1b94106600 2024-04-27T01:57:29.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.269+0000 7f1b9c369700 1 -- 172.21.15.29:0/1899281840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b941045b0 con 0x7f1b94106600 2024-04-27T01:57:29.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.269+0000 7f1b8b7fe700 1 -- 172.21.15.29:0/1899281840 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b90019880 con 0x7f1b94106600 2024-04-27T01:57:29.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.269+0000 7f1b9c369700 1 -- 172.21.15.29:0/1899281840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b941049d0 con 0x7f1b94106600 2024-04-27T01:57:29.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.269+0000 7f1b8b7fe700 1 -- 172.21.15.29:0/1899281840 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f1b9001a430 con 0x7f1b94106600 2024-04-27T01:57:29.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.269+0000 7f1b9c369700 1 -- 172.21.15.29:0/1899281840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b940623f0 con 0x7f1b94106600 2024-04-27T01:57:29.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.270+0000 7f1b8b7fe700 1 --2- 172.21.15.29:0/1899281840 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f1b800383b0 0x7f1b8003a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:29.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.270+0000 7f1b8b7fe700 1 -- 172.21.15.29:0/1899281840 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1b9004c760 con 0x7f1b94106600 2024-04-27T01:57:29.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.270+0000 7f1b99904700 1 --2- 172.21.15.29:0/1899281840 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f1b800383b0 0x7f1b8003a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:29.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.270+0000 7f1b99904700 1 --2- 172.21.15.29:0/1899281840 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f1b800383b0 0x7f1b8003a860 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1b84006fd0 tx=0x7f1b84006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:29.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.274+0000 7f1b8b7fe700 1 -- 172.21.15.29:0/1899281840 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1b940623f0 con 0x7f1b94106600 2024-04-27T01:57:29.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.502+0000 7f1b9c369700 1 -- 172.21.15.29:0/1899281840 --> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b94062380 con 0x7f1b800383b0 2024-04-27T01:57:29.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.693+0000 7f1b8b7fe700 1 -- 172.21.15.29:0/1899281840 <== mgr.14120 v2:172.21.15.29:6800/3873642219 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1b94062380 con 0x7f1b800383b0 2024-04-27T01:57:29.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.695+0000 7f1b9c369700 1 -- 172.21.15.29:0/1899281840 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f1b800383b0 msgr2=0x7f1b8003a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:29.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.695+0000 7f1b9c369700 1 --2- 172.21.15.29:0/1899281840 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f1b800383b0 0x7f1b8003a860 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1b84006fd0 tx=0x7f1b84006e40 comp rx=0 tx=0).stop 2024-04-27T01:57:29.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.695+0000 7f1b9c369700 1 -- 172.21.15.29:0/1899281840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94106600 msgr2=0x7f1b94105f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:29.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.695+0000 7f1b9c369700 1 --2- 172.21.15.29:0/1899281840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94106600 0x7f1b94105f60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1b90006af0 tx=0x7f1b90011a90 comp rx=0 tx=0).stop 2024-04-27T01:57:29.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.695+0000 7f1b9c369700 1 -- 172.21.15.29:0/1899281840 shutdown_connections 2024-04-27T01:57:29.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.695+0000 7f1b9c369700 1 --2- 172.21.15.29:0/1899281840 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f1b800383b0 0x7f1b8003a860 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:29.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.695+0000 7f1b9c369700 1 --2- 172.21.15.29:0/1899281840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94106600 0x7f1b94105f60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:29.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.695+0000 7f1b9c369700 1 -- 172.21.15.29:0/1899281840 >> 172.21.15.29:0/1899281840 conn(0x7f1b94100080 msgr2=0x7f1b94101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:29.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.695+0000 7f1b9c369700 1 -- 172.21.15.29:0/1899281840 shutdown_connections 2024-04-27T01:57:29.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:28.695+0000 7f1b9c369700 1 -- 172.21.15.29:0/1899281840 wait complete. 2024-04-27T01:57:29.926 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:29 smithi029 ceph-mon[26769]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:57:29.927 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:29 smithi029 ceph-mon[26769]: Generating ssh key... 2024-04-27T01:57:30.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrvDQxSFEILBV4WWqSNuz9odisfN66Pm2UacN9QGO+79rzFM2SXj61/pttSR176IQtxnTwGJO1FFwelosoe7Us7M6iAW+3n5zwOPAFsoCCDbd/JO+0THx7WD6PUA/5MUG4rlUzaI3WapyNx/6wRotuDanxe6HnNQ/tQXCk2pLo8JJxMc5HX+njVi1zPVwZx55K+AUAKcr9lOQqemUCdIr0zMgHJgL38FhbnvyZBUFr4oT3XnC1641Q77WrDrf/1Crko1JofmnUeqjVnRIh8uc4dtZnzTaUva8qQ7V58AzhBwMURxQbkkc7t7eFCBSgR55Qe0DeZhTq5c+CrVYnT0+A9RqX5ykG5mMroRI4bhKPmx+2X0r8WcoKaz94TB0J1TNMpE83QQJ+9eKnEyxQ487cOiNyaw02rxFMOW6FoKF4Ok/vgd9jN0DLYHerDgb6G08cHQxby5BE03drMYygt08rmxcDQTGev4bkQ3J4F7QBBkNpoLLh3/gZayKyjhOpJnE= ceph-4c587076-0439-11ef-bc93-c7b262605968 2024-04-27T01:57:30.501 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.734+0000 7fce5575b700 1 Processor -- start 2024-04-27T01:57:30.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.734+0000 7fce5575b700 1 -- start start 2024-04-27T01:57:30.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.735+0000 7fce5575b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce5007bdd0 0x7fce5007a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:30.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.735+0000 7fce5575b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5007a810 con 0x7fce5007bdd0 2024-04-27T01:57:30.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.735+0000 7fce4effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce5007bdd0 0x7fce5007a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:30.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.735+0000 7fce4effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce5007bdd0 0x7fce5007a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42600/0 (socket says 172.21.15.29:42600) 2024-04-27T01:57:30.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.735+0000 7fce4effd700 1 -- 172.21.15.29:0/762857379 learned_addr learned my addr 172.21.15.29:0/762857379 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:30.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.735+0000 7fce4effd700 1 -- 172.21.15.29:0/762857379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce5007a950 con 0x7fce5007bdd0 2024-04-27T01:57:30.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.735+0000 7fce4effd700 1 --2- 172.21.15.29:0/762857379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce5007bdd0 0x7fce5007a2d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fce400098a0 tx=0x7fce40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4241a99dc0c7ea96 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:30.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.735+0000 7fce4dffb700 1 -- 172.21.15.29:0/762857379 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce40004180 con 0x7fce5007bdd0 2024-04-27T01:57:30.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.736+0000 7fce4dffb700 1 -- 172.21.15.29:0/762857379 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fce400042e0 con 0x7fce5007bdd0 2024-04-27T01:57:30.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.736+0000 7fce4dffb700 1 -- 172.21.15.29:0/762857379 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce40011a50 con 0x7fce5007bdd0 2024-04-27T01:57:30.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.736+0000 7fce5575b700 1 -- 172.21.15.29:0/762857379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce5007bdd0 msgr2=0x7fce5007a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:30.502 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.736+0000 7fce5575b700 1 --2- 172.21.15.29:0/762857379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce5007bdd0 0x7fce5007a2d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fce400098a0 tx=0x7fce40009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:30.503 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.736+0000 7fce5575b700 1 -- 172.21.15.29:0/762857379 shutdown_connections 2024-04-27T01:57:30.503 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.736+0000 7fce5575b700 1 --2- 172.21.15.29:0/762857379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce5007bdd0 0x7fce5007a2d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:30.503 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.736+0000 7fce5575b700 1 -- 172.21.15.29:0/762857379 >> 172.21.15.29:0/762857379 conn(0x7fce50101180 msgr2=0x7fce50103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:30.503 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.736+0000 7fce5575b700 1 -- 172.21.15.29:0/762857379 shutdown_connections 2024-04-27T01:57:30.503 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.736+0000 7fce5575b700 1 -- 172.21.15.29:0/762857379 wait complete. 2024-04-27T01:57:30.503 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.737+0000 7fce5575b700 1 Processor -- start 2024-04-27T01:57:30.503 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.737+0000 7fce5575b700 1 -- start start 2024-04-27T01:57:30.503 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.737+0000 7fce5575b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce5007bdd0 0x7fce5019ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:30.503 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.737+0000 7fce5575b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5019bfc0 con 0x7fce5007bdd0 2024-04-27T01:57:30.503 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.737+0000 7fce4effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce5007bdd0 0x7fce5019ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:30.503 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.737+0000 7fce4effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce5007bdd0 0x7fce5019ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42614/0 (socket says 172.21.15.29:42614) 2024-04-27T01:57:30.503 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.737+0000 7fce4effd700 1 -- 172.21.15.29:0/245933354 learned_addr learned my addr 172.21.15.29:0/245933354 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:30.503 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.738+0000 7fce4effd700 1 -- 172.21.15.29:0/245933354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce40009550 con 0x7fce5007bdd0 2024-04-27T01:57:30.503 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.738+0000 7fce4effd700 1 --2- 172.21.15.29:0/245933354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce5007bdd0 0x7fce5019ba80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fce40000c00 tx=0x7fce400047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:30.504 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.738+0000 7fce37fff700 1 -- 172.21.15.29:0/245933354 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce40003a00 con 0x7fce5007bdd0 2024-04-27T01:57:30.504 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.738+0000 7fce37fff700 1 -- 172.21.15.29:0/245933354 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fce40003b60 con 0x7fce5007bdd0 2024-04-27T01:57:30.504 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.738+0000 7fce5575b700 1 -- 172.21.15.29:0/245933354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce5019c1c0 con 0x7fce5007bdd0 2024-04-27T01:57:30.504 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.738+0000 7fce37fff700 1 -- 172.21.15.29:0/245933354 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce4001a800 con 0x7fce5007bdd0 2024-04-27T01:57:30.504 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.738+0000 7fce5575b700 1 -- 172.21.15.29:0/245933354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce5019c5e0 con 0x7fce5007bdd0 2024-04-27T01:57:30.504 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.739+0000 7fce37fff700 1 -- 172.21.15.29:0/245933354 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fce4001aa70 con 0x7fce5007bdd0 2024-04-27T01:57:30.504 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.739+0000 7fce5575b700 1 -- 172.21.15.29:0/245933354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce50062380 con 0x7fce5007bdd0 2024-04-27T01:57:30.504 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.739+0000 7fce37fff700 1 --2- 172.21.15.29:0/245933354 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fce380383f0 0x7fce3803a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:30.504 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.739+0000 7fce37fff700 1 -- 172.21.15.29:0/245933354 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fce4004c650 con 0x7fce5007bdd0 2024-04-27T01:57:30.504 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.739+0000 7fce4e7fc700 1 --2- 172.21.15.29:0/245933354 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fce380383f0 0x7fce3803a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:30.504 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.740+0000 7fce4e7fc700 1 --2- 172.21.15.29:0/245933354 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fce380383f0 0x7fce3803a8a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fce44006fd0 tx=0x7fce44006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:30.504 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.744+0000 7fce37fff700 1 -- 172.21.15.29:0/245933354 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fce4001ad20 con 0x7fce5007bdd0 2024-04-27T01:57:30.504 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.945+0000 7fce5575b700 1 -- 172.21.15.29:0/245933354 --> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fce50102670 con 0x7fce380383f0 2024-04-27T01:57:30.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.946+0000 7fce37fff700 1 -- 172.21.15.29:0/245933354 <== mgr.14120 v2:172.21.15.29:6800/3873642219 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fce50102670 con 0x7fce380383f0 2024-04-27T01:57:30.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.947+0000 7fce5575b700 1 -- 172.21.15.29:0/245933354 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fce380383f0 msgr2=0x7fce3803a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:30.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.948+0000 7fce5575b700 1 --2- 172.21.15.29:0/245933354 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fce380383f0 0x7fce3803a8a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fce44006fd0 tx=0x7fce44006e40 comp rx=0 tx=0).stop 2024-04-27T01:57:30.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.948+0000 7fce5575b700 1 -- 172.21.15.29:0/245933354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce5007bdd0 msgr2=0x7fce5019ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:30.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.948+0000 7fce5575b700 1 --2- 172.21.15.29:0/245933354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce5007bdd0 0x7fce5019ba80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fce40000c00 tx=0x7fce400047c0 comp rx=0 tx=0).stop 2024-04-27T01:57:30.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.948+0000 7fce5575b700 1 -- 172.21.15.29:0/245933354 shutdown_connections 2024-04-27T01:57:30.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.948+0000 7fce5575b700 1 --2- 172.21.15.29:0/245933354 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fce380383f0 0x7fce3803a8a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:30.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.948+0000 7fce5575b700 1 --2- 172.21.15.29:0/245933354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce5007bdd0 0x7fce5019ba80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:30.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.948+0000 7fce5575b700 1 -- 172.21.15.29:0/245933354 >> 172.21.15.29:0/245933354 conn(0x7fce50101180 msgr2=0x7fce50101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:30.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.948+0000 7fce5575b700 1 -- 172.21.15.29:0/245933354 shutdown_connections 2024-04-27T01:57:30.505 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:29.948+0000 7fce5575b700 1 -- 172.21.15.29:0/245933354 wait complete. 2024-04-27T01:57:30.505 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-27T01:57:30.505 INFO:teuthology.orchestra.run.smithi029.stdout:Adding key to root@localhost authorized_keys... 2024-04-27T01:57:30.505 INFO:teuthology.orchestra.run.smithi029.stdout:Adding host smithi029... 2024-04-27T01:57:32.051 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:31 smithi029 ceph-mon[26769]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:57:33.029 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:32 smithi029 ceph-mon[26769]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi029", "addr": "172.21.15.29", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:57:33.029 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:32 smithi029 ceph-mon[26769]: Deploying cephadm binary to smithi029 2024-04-27T01:57:34.202 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Added host 'smithi029' with addr '172.21.15.29' 2024-04-27T01:57:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.971+0000 7f29c9e6e700 1 Processor -- start 2024-04-27T01:57:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.971+0000 7f29c9e6e700 1 -- start start 2024-04-27T01:57:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.971+0000 7f29c9e6e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29c4106600 0x7f29c41089e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.971+0000 7f29c9e6e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c4079d70 con 0x7f29c4106600 2024-04-27T01:57:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.971+0000 7f29c37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29c4106600 0x7f29c41089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.971+0000 7f29c37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29c4106600 0x7f29c41089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42628/0 (socket says 172.21.15.29:42628) 2024-04-27T01:57:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.971+0000 7f29c37fe700 1 -- 172.21.15.29:0/1208812764 learned_addr learned my addr 172.21.15.29:0/1208812764 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.972+0000 7f29c37fe700 1 -- 172.21.15.29:0/1208812764 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c4079eb0 con 0x7f29c4106600 2024-04-27T01:57:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.972+0000 7f29c37fe700 1 --2- 172.21.15.29:0/1208812764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29c4106600 0x7f29c41089e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f29b4009e80 tx=0x7f29b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61566900dab747bb server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.972+0000 7f29c27fc700 1 -- 172.21.15.29:0/1208812764 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b400e070 con 0x7f29c4106600 2024-04-27T01:57:34.203 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.972+0000 7f29c27fc700 1 -- 172.21.15.29:0/1208812764 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f29b4003e90 con 0x7f29c4106600 2024-04-27T01:57:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.972+0000 7f29c27fc700 1 -- 172.21.15.29:0/1208812764 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b4004380 con 0x7f29c4106600 2024-04-27T01:57:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.973+0000 7f29c9e6e700 1 -- 172.21.15.29:0/1208812764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29c4106600 msgr2=0x7f29c41089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.973+0000 7f29c9e6e700 1 --2- 172.21.15.29:0/1208812764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29c4106600 0x7f29c41089e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f29b4009e80 tx=0x7f29b400b330 comp rx=0 tx=0).stop 2024-04-27T01:57:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.973+0000 7f29c9e6e700 1 -- 172.21.15.29:0/1208812764 shutdown_connections 2024-04-27T01:57:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.973+0000 7f29c9e6e700 1 --2- 172.21.15.29:0/1208812764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29c4106600 0x7f29c41089e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.973+0000 7f29c9e6e700 1 -- 172.21.15.29:0/1208812764 >> 172.21.15.29:0/1208812764 conn(0x7f29c4100080 msgr2=0x7f29c4102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.973+0000 7f29c9e6e700 1 -- 172.21.15.29:0/1208812764 shutdown_connections 2024-04-27T01:57:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.973+0000 7f29c9e6e700 1 -- 172.21.15.29:0/1208812764 wait complete. 2024-04-27T01:57:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.973+0000 7f29c9e6e700 1 Processor -- start 2024-04-27T01:57:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.974+0000 7f29c9e6e700 1 -- start start 2024-04-27T01:57:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.974+0000 7f29c9e6e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29c4106600 0x7f29c41997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.974+0000 7f29c9e6e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c4199d10 con 0x7f29c4106600 2024-04-27T01:57:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.974+0000 7f29c37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29c4106600 0x7f29c41997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:34.204 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.974+0000 7f29c37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29c4106600 0x7f29c41997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42638/0 (socket says 172.21.15.29:42638) 2024-04-27T01:57:34.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.974+0000 7f29c37fe700 1 -- 172.21.15.29:0/1067677564 learned_addr learned my addr 172.21.15.29:0/1067677564 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:34.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.974+0000 7f29c37fe700 1 -- 172.21.15.29:0/1067677564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b4009b30 con 0x7f29c4106600 2024-04-27T01:57:34.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.974+0000 7f29c37fe700 1 --2- 172.21.15.29:0/1067677564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29c4106600 0x7f29c41997d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f29b400b7b0 tx=0x7f29b4004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:34.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.974+0000 7f29c0ff9700 1 -- 172.21.15.29:0/1067677564 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b400e040 con 0x7f29c4106600 2024-04-27T01:57:34.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.975+0000 7f29c9e6e700 1 -- 172.21.15.29:0/1067677564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c4199f10 con 0x7f29c4106600 2024-04-27T01:57:34.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.975+0000 7f29c0ff9700 1 -- 172.21.15.29:0/1067677564 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f29b4003c70 con 0x7f29c4106600 2024-04-27T01:57:34.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.975+0000 7f29c0ff9700 1 -- 172.21.15.29:0/1067677564 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b401b730 con 0x7f29c4106600 2024-04-27T01:57:34.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.975+0000 7f29c9e6e700 1 -- 172.21.15.29:0/1067677564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c419a390 con 0x7f29c4106600 2024-04-27T01:57:34.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.975+0000 7f29c0ff9700 1 -- 172.21.15.29:0/1067677564 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f29b4019040 con 0x7f29c4106600 2024-04-27T01:57:34.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.975+0000 7f29c9e6e700 1 -- 172.21.15.29:0/1067677564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c40623f0 con 0x7f29c4106600 2024-04-27T01:57:34.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.976+0000 7f29c0ff9700 1 --2- 172.21.15.29:0/1067677564 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f29b00383b0 0x7f29b003a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:34.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.976+0000 7f29c0ff9700 1 -- 172.21.15.29:0/1067677564 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f29b404c040 con 0x7f29c4106600 2024-04-27T01:57:34.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.976+0000 7f29c2ffd700 1 --2- 172.21.15.29:0/1067677564 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f29b00383b0 0x7f29b003a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:34.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.976+0000 7f29c2ffd700 1 --2- 172.21.15.29:0/1067677564 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f29b00383b0 0x7f29b003a860 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f29ac006fd0 tx=0x7f29ac006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:34.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:30.980+0000 7f29c0ff9700 1 -- 172.21.15.29:0/1067677564 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f29c40623f0 con 0x7f29c4106600 2024-04-27T01:57:34.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:31.184+0000 7f29c9e6e700 1 -- 172.21.15.29:0/1067677564 --> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi029", "addr": "172.21.15.29", "target": ["mon-mgr", ""]}) v1 -- 0x7f29c4062380 con 0x7f29b00383b0 2024-04-27T01:57:34.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:33.571+0000 7f29c0ff9700 1 -- 172.21.15.29:0/1067677564 <== mgr.14120 v2:172.21.15.29:6800/3873642219 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f29c4062380 con 0x7f29b00383b0 2024-04-27T01:57:34.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:33.573+0000 7f29c9e6e700 1 -- 172.21.15.29:0/1067677564 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f29b00383b0 msgr2=0x7f29b003a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:34.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:33.573+0000 7f29c9e6e700 1 --2- 172.21.15.29:0/1067677564 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f29b00383b0 0x7f29b003a860 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f29ac006fd0 tx=0x7f29ac006e40 comp rx=0 tx=0).stop 2024-04-27T01:57:34.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:33.573+0000 7f29c9e6e700 1 -- 172.21.15.29:0/1067677564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29c4106600 msgr2=0x7f29c41997d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:34.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:33.573+0000 7f29c9e6e700 1 --2- 172.21.15.29:0/1067677564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29c4106600 0x7f29c41997d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f29b400b7b0 tx=0x7f29b4004430 comp rx=0 tx=0).stop 2024-04-27T01:57:34.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:33.573+0000 7f29c9e6e700 1 -- 172.21.15.29:0/1067677564 shutdown_connections 2024-04-27T01:57:34.207 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:33.573+0000 7f29c9e6e700 1 --2- 172.21.15.29:0/1067677564 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f29b00383b0 0x7f29b003a860 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:34.207 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:33.573+0000 7f29c9e6e700 1 --2- 172.21.15.29:0/1067677564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29c4106600 0x7f29c41997d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:34.207 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:33.573+0000 7f29c9e6e700 1 -- 172.21.15.29:0/1067677564 >> 172.21.15.29:0/1067677564 conn(0x7f29c4100080 msgr2=0x7f29c4100e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:34.207 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:33.574+0000 7f29c9e6e700 1 -- 172.21.15.29:0/1067677564 shutdown_connections 2024-04-27T01:57:34.207 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:33.574+0000 7f29c9e6e700 1 -- 172.21.15.29:0/1067677564 wait complete. 2024-04-27T01:57:34.207 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying mon service with default placement... 2024-04-27T01:57:34.876 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:34 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:34.876 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:34 smithi029 ceph-mon[26769]: Added host smithi029 2024-04-27T01:57:34.876 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:34 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:57:35.867 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-27T01:57:35.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.683+0000 7f5813b3f700 1 Processor -- start 2024-04-27T01:57:35.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.683+0000 7f5813b3f700 1 -- start start 2024-04-27T01:57:35.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.683+0000 7f5813b3f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f580c106670 0x7f580c108a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:35.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.683+0000 7f5813b3f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580c0744b0 con 0x7f580c106670 2024-04-27T01:57:35.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.684+0000 7f58118db700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f580c106670 0x7f580c108a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:35.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.684+0000 7f58118db700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f580c106670 0x7f580c108a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42650/0 (socket says 172.21.15.29:42650) 2024-04-27T01:57:35.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.684+0000 7f58118db700 1 -- 172.21.15.29:0/4051047743 learned_addr learned my addr 172.21.15.29:0/4051047743 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:35.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.684+0000 7f58118db700 1 -- 172.21.15.29:0/4051047743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580c0745f0 con 0x7f580c106670 2024-04-27T01:57:35.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.684+0000 7f58118db700 1 --2- 172.21.15.29:0/4051047743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f580c106670 0x7f580c108a50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f58080098a0 tx=0x7f5808009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dcba0eb90360ae8 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:35.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.684+0000 7f58108d9700 1 -- 172.21.15.29:0/4051047743 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5808004180 con 0x7f580c106670 2024-04-27T01:57:35.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.684+0000 7f58108d9700 1 -- 172.21.15.29:0/4051047743 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f58080042e0 con 0x7f580c106670 2024-04-27T01:57:35.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.684+0000 7f58108d9700 1 -- 172.21.15.29:0/4051047743 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5808011a50 con 0x7f580c106670 2024-04-27T01:57:35.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.685+0000 7f5813b3f700 1 -- 172.21.15.29:0/4051047743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f580c106670 msgr2=0x7f580c108a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:35.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.685+0000 7f5813b3f700 1 --2- 172.21.15.29:0/4051047743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f580c106670 0x7f580c108a50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f58080098a0 tx=0x7f5808009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:35.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.685+0000 7f5813b3f700 1 -- 172.21.15.29:0/4051047743 shutdown_connections 2024-04-27T01:57:35.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.685+0000 7f5813b3f700 1 --2- 172.21.15.29:0/4051047743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f580c106670 0x7f580c108a50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:35.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.685+0000 7f5813b3f700 1 -- 172.21.15.29:0/4051047743 >> 172.21.15.29:0/4051047743 conn(0x7f580c100070 msgr2=0x7f580c102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:35.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.685+0000 7f5813b3f700 1 -- 172.21.15.29:0/4051047743 shutdown_connections 2024-04-27T01:57:35.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.685+0000 7f5813b3f700 1 -- 172.21.15.29:0/4051047743 wait complete. 2024-04-27T01:57:35.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.686+0000 7f5813b3f700 1 Processor -- start 2024-04-27T01:57:35.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.686+0000 7f5813b3f700 1 -- start start 2024-04-27T01:57:35.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.686+0000 7f5813b3f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f580c106670 0x7f580c199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:35.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.686+0000 7f5813b3f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580c199d90 con 0x7f580c106670 2024-04-27T01:57:35.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.686+0000 7f58118db700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f580c106670 0x7f580c199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:35.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.686+0000 7f58118db700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f580c106670 0x7f580c199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42652/0 (socket says 172.21.15.29:42652) 2024-04-27T01:57:35.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.686+0000 7f58118db700 1 -- 172.21.15.29:0/860658586 learned_addr learned my addr 172.21.15.29:0/860658586 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:35.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.686+0000 7f58118db700 1 -- 172.21.15.29:0/860658586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5808009550 con 0x7f580c106670 2024-04-27T01:57:35.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.687+0000 7f58118db700 1 --2- 172.21.15.29:0/860658586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f580c106670 0x7f580c199850 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5808000c00 tx=0x7f5808011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:35.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.687+0000 7f5802ffd700 1 -- 172.21.15.29:0/860658586 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5808003b90 con 0x7f580c106670 2024-04-27T01:57:35.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.687+0000 7f5802ffd700 1 -- 172.21.15.29:0/860658586 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f58080248b0 con 0x7f580c106670 2024-04-27T01:57:35.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.687+0000 7f5802ffd700 1 -- 172.21.15.29:0/860658586 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f580801a8e0 con 0x7f580c106670 2024-04-27T01:57:35.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.687+0000 7f5813b3f700 1 -- 172.21.15.29:0/860658586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f580c199f90 con 0x7f580c106670 2024-04-27T01:57:35.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.687+0000 7f5813b3f700 1 -- 172.21.15.29:0/860658586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f580c19a3b0 con 0x7f580c106670 2024-04-27T01:57:35.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.688+0000 7f5802ffd700 1 -- 172.21.15.29:0/860658586 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f5808024a20 con 0x7f580c106670 2024-04-27T01:57:35.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.688+0000 7f5802ffd700 1 --2- 172.21.15.29:0/860658586 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f57f8038440 0x7f57f803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:35.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.688+0000 7f5802ffd700 1 -- 172.21.15.29:0/860658586 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f580804c390 con 0x7f580c106670 2024-04-27T01:57:35.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.688+0000 7f5813b3f700 1 -- 172.21.15.29:0/860658586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f580c062380 con 0x7f580c106670 2024-04-27T01:57:35.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.688+0000 7f58110da700 1 --2- 172.21.15.29:0/860658586 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f57f8038440 0x7f57f803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:35.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.689+0000 7f58110da700 1 --2- 172.21.15.29:0/860658586 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f57f8038440 0x7f57f803a8f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f57fc006fd0 tx=0x7f57fc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:35.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.693+0000 7f5802ffd700 1 -- 172.21.15.29:0/860658586 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f5808018db0 con 0x7f580c106670 2024-04-27T01:57:35.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.908+0000 7f5813b3f700 1 -- 172.21.15.29:0/860658586 --> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f580c102380 con 0x7f57f8038440 2024-04-27T01:57:35.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.913+0000 7f5802ffd700 1 -- 172.21.15.29:0/860658586 <== mgr.14120 v2:172.21.15.29:6800/3873642219 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f580c102380 con 0x7f57f8038440 2024-04-27T01:57:35.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.915+0000 7f5813b3f700 1 -- 172.21.15.29:0/860658586 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f57f8038440 msgr2=0x7f57f803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:35.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.915+0000 7f5813b3f700 1 --2- 172.21.15.29:0/860658586 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f57f8038440 0x7f57f803a8f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f57fc006fd0 tx=0x7f57fc006e40 comp rx=0 tx=0).stop 2024-04-27T01:57:35.872 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.915+0000 7f5813b3f700 1 -- 172.21.15.29:0/860658586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f580c106670 msgr2=0x7f580c199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:35.872 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.915+0000 7f5813b3f700 1 --2- 172.21.15.29:0/860658586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f580c106670 0x7f580c199850 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5808000c00 tx=0x7f5808011ab0 comp rx=0 tx=0).stop 2024-04-27T01:57:35.872 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.915+0000 7f5813b3f700 1 -- 172.21.15.29:0/860658586 shutdown_connections 2024-04-27T01:57:35.872 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.915+0000 7f5813b3f700 1 --2- 172.21.15.29:0/860658586 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f57f8038440 0x7f57f803a8f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:35.872 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.915+0000 7f5813b3f700 1 --2- 172.21.15.29:0/860658586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f580c106670 0x7f580c199850 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:35.872 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.915+0000 7f5813b3f700 1 -- 172.21.15.29:0/860658586 >> 172.21.15.29:0/860658586 conn(0x7f580c100070 msgr2=0x7f580c101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:35.872 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.915+0000 7f5813b3f700 1 -- 172.21.15.29:0/860658586 shutdown_connections 2024-04-27T01:57:35.872 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:34.915+0000 7f5813b3f700 1 -- 172.21.15.29:0/860658586 wait complete. 2024-04-27T01:57:35.872 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying mgr service with default placement... 2024-04-27T01:57:36.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:35 smithi029 ceph-mon[26769]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:57:36.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:35 smithi029 ceph-mon[26769]: Saving service mon spec with placement count:5 2024-04-27T01:57:36.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:35 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:37.645 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:37 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:37.645 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:37 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:37.645 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:37 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:37.648 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-27T01:57:37.648 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.773+0000 7f740d890700 1 Processor -- start 2024-04-27T01:57:37.649 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.773+0000 7f740d890700 1 -- start start 2024-04-27T01:57:37.649 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.773+0000 7f740d890700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f740807be10 0x7f740807a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:37.653 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.773+0000 7f740d890700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f740807a850 con 0x7f740807be10 2024-04-27T01:57:37.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.773+0000 7f7406ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f740807be10 0x7f740807a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:37.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.773+0000 7f7406ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f740807be10 0x7f740807a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44812/0 (socket says 172.21.15.29:44812) 2024-04-27T01:57:37.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.773+0000 7f7406ffd700 1 -- 172.21.15.29:0/3478690970 learned_addr learned my addr 172.21.15.29:0/3478690970 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:37.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.773+0000 7f7406ffd700 1 -- 172.21.15.29:0/3478690970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f740807a990 con 0x7f740807be10 2024-04-27T01:57:37.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.773+0000 7f7406ffd700 1 --2- 172.21.15.29:0/3478690970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f740807be10 0x7f740807a310 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f73f8009e80 tx=0x7f73f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75428770f08bba30 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:37.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.774+0000 7f7405ffb700 1 -- 172.21.15.29:0/3478690970 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73f800e070 con 0x7f740807be10 2024-04-27T01:57:37.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.774+0000 7f7405ffb700 1 -- 172.21.15.29:0/3478690970 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f73f800b8c0 con 0x7f740807be10 2024-04-27T01:57:37.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.774+0000 7f7405ffb700 1 -- 172.21.15.29:0/3478690970 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73f80044d0 con 0x7f740807be10 2024-04-27T01:57:37.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.774+0000 7f740d890700 1 -- 172.21.15.29:0/3478690970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f740807be10 msgr2=0x7f740807a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:37.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.774+0000 7f740d890700 1 --2- 172.21.15.29:0/3478690970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f740807be10 0x7f740807a310 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f73f8009e80 tx=0x7f73f800b330 comp rx=0 tx=0).stop 2024-04-27T01:57:37.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.774+0000 7f740d890700 1 -- 172.21.15.29:0/3478690970 shutdown_connections 2024-04-27T01:57:37.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.774+0000 7f740d890700 1 --2- 172.21.15.29:0/3478690970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f740807be10 0x7f740807a310 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:37.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.774+0000 7f740d890700 1 -- 172.21.15.29:0/3478690970 >> 172.21.15.29:0/3478690970 conn(0x7f7408101180 msgr2=0x7f7408103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:37.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.774+0000 7f740d890700 1 -- 172.21.15.29:0/3478690970 shutdown_connections 2024-04-27T01:57:37.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.774+0000 7f740d890700 1 -- 172.21.15.29:0/3478690970 wait complete. 2024-04-27T01:57:37.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.775+0000 7f740d890700 1 Processor -- start 2024-04-27T01:57:37.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.775+0000 7f740d890700 1 -- start start 2024-04-27T01:57:37.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.775+0000 7f740d890700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f740807be10 0x7f740819ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:37.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.775+0000 7f740d890700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f740819bfc0 con 0x7f740807be10 2024-04-27T01:57:37.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.775+0000 7f7406ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f740807be10 0x7f740819ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:37.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.775+0000 7f7406ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f740807be10 0x7f740819ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44828/0 (socket says 172.21.15.29:44828) 2024-04-27T01:57:37.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.775+0000 7f7406ffd700 1 -- 172.21.15.29:0/1786759574 learned_addr learned my addr 172.21.15.29:0/1786759574 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:37.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.776+0000 7f7406ffd700 1 -- 172.21.15.29:0/1786759574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f8009b30 con 0x7f740807be10 2024-04-27T01:57:37.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.776+0000 7f7406ffd700 1 --2- 172.21.15.29:0/1786759574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f740807be10 0x7f740819ba80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f73f800bf20 tx=0x7f73f80095b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:37.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.776+0000 7f740c88e700 1 -- 172.21.15.29:0/1786759574 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73f800e040 con 0x7f740807be10 2024-04-27T01:57:37.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.776+0000 7f740c88e700 1 -- 172.21.15.29:0/1786759574 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f73f8004690 con 0x7f740807be10 2024-04-27T01:57:37.656 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.776+0000 7f740d890700 1 -- 172.21.15.29:0/1786759574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f740819c1c0 con 0x7f740807be10 2024-04-27T01:57:37.656 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.776+0000 7f740c88e700 1 -- 172.21.15.29:0/1786759574 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73f8012650 con 0x7f740807be10 2024-04-27T01:57:37.656 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.776+0000 7f740d890700 1 -- 172.21.15.29:0/1786759574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f740819c640 con 0x7f740807be10 2024-04-27T01:57:37.656 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.777+0000 7f740c88e700 1 -- 172.21.15.29:0/1786759574 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f73f8019040 con 0x7f740807be10 2024-04-27T01:57:37.656 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.777+0000 7f740d890700 1 -- 172.21.15.29:0/1786759574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7408062380 con 0x7f740807be10 2024-04-27T01:57:37.656 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.777+0000 7f740c88e700 1 --2- 172.21.15.29:0/1786759574 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f73f4038440 0x7f73f403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:37.656 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.777+0000 7f740c88e700 1 -- 172.21.15.29:0/1786759574 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f73f804bd50 con 0x7f740807be10 2024-04-27T01:57:37.656 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.777+0000 7f74067fc700 1 --2- 172.21.15.29:0/1786759574 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f73f4038440 0x7f73f403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:37.656 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.778+0000 7f74067fc700 1 --2- 172.21.15.29:0/1786759574 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f73f4038440 0x7f73f403a8f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f73f0006fd0 tx=0x7f73f0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:37.656 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.782+0000 7f740c88e700 1 -- 172.21.15.29:0/1786759574 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f73f801b610 con 0x7f740807be10 2024-04-27T01:57:37.656 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:36.997+0000 7f740d890700 1 -- 172.21.15.29:0/1786759574 --> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f7408102630 con 0x7f73f4038440 2024-04-27T01:57:37.656 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:37.002+0000 7f740c88e700 1 -- 172.21.15.29:0/1786759574 <== mgr.14120 v2:172.21.15.29:6800/3873642219 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f7408102630 con 0x7f73f4038440 2024-04-27T01:57:37.656 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:37.004+0000 7f740d890700 1 -- 172.21.15.29:0/1786759574 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f73f4038440 msgr2=0x7f73f403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:37.656 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:37.004+0000 7f740d890700 1 --2- 172.21.15.29:0/1786759574 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f73f4038440 0x7f73f403a8f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f73f0006fd0 tx=0x7f73f0006e40 comp rx=0 tx=0).stop 2024-04-27T01:57:37.657 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:37.004+0000 7f740d890700 1 -- 172.21.15.29:0/1786759574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f740807be10 msgr2=0x7f740819ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:37.657 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:37.004+0000 7f740d890700 1 --2- 172.21.15.29:0/1786759574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f740807be10 0x7f740819ba80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f73f800bf20 tx=0x7f73f80095b0 comp rx=0 tx=0).stop 2024-04-27T01:57:37.657 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:37.004+0000 7f740d890700 1 -- 172.21.15.29:0/1786759574 shutdown_connections 2024-04-27T01:57:37.657 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:37.004+0000 7f740d890700 1 --2- 172.21.15.29:0/1786759574 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f73f4038440 0x7f73f403a8f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:37.657 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:37.004+0000 7f740d890700 1 --2- 172.21.15.29:0/1786759574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f740807be10 0x7f740819ba80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:37.657 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:37.004+0000 7f740d890700 1 -- 172.21.15.29:0/1786759574 >> 172.21.15.29:0/1786759574 conn(0x7f7408101180 msgr2=0x7f7408101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:37.657 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:37.005+0000 7f740d890700 1 -- 172.21.15.29:0/1786759574 shutdown_connections 2024-04-27T01:57:37.657 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:37.005+0000 7f740d890700 1 -- 172.21.15.29:0/1786759574 wait complete. 2024-04-27T01:57:37.657 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying crash service with default placement... 2024-04-27T01:57:38.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:38 smithi029 ceph-mon[26769]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:57:38.702 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:38 smithi029 ceph-mon[26769]: Saving service mgr spec with placement count:2 2024-04-27T01:57:39.084 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-04-27T01:57:39.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.132+0000 7f6eef08b700 1 Processor -- start 2024-04-27T01:57:39.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.132+0000 7f6eef08b700 1 -- start start 2024-04-27T01:57:39.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.133+0000 7f6eef08b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee8105b10 0x7f6ee8105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:39.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.133+0000 7f6eef08b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee8106460 con 0x7f6ee8105b10 2024-04-27T01:57:39.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.133+0000 7f6eece27700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee8105b10 0x7f6ee8105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:39.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.133+0000 7f6eece27700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee8105b10 0x7f6ee8105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44838/0 (socket says 172.21.15.29:44838) 2024-04-27T01:57:39.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.133+0000 7f6eece27700 1 -- 172.21.15.29:0/3941484116 learned_addr learned my addr 172.21.15.29:0/3941484116 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:39.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.133+0000 7f6eece27700 1 -- 172.21.15.29:0/3941484116 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee81065a0 con 0x7f6ee8105b10 2024-04-27T01:57:39.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.133+0000 7f6eece27700 1 --2- 172.21.15.29:0/3941484116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee8105b10 0x7f6ee8105f20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6edc0098a0 tx=0x7f6edc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc583d1398dc434 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:39.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.133+0000 7f6ee77fe700 1 -- 172.21.15.29:0/3941484116 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6edc004180 con 0x7f6ee8105b10 2024-04-27T01:57:39.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.133+0000 7f6ee77fe700 1 -- 172.21.15.29:0/3941484116 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6edc0042e0 con 0x7f6ee8105b10 2024-04-27T01:57:39.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.133+0000 7f6ee77fe700 1 -- 172.21.15.29:0/3941484116 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6edc011af0 con 0x7f6ee8105b10 2024-04-27T01:57:39.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.134+0000 7f6eef08b700 1 -- 172.21.15.29:0/3941484116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee8105b10 msgr2=0x7f6ee8105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:39.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.134+0000 7f6eef08b700 1 --2- 172.21.15.29:0/3941484116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee8105b10 0x7f6ee8105f20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6edc0098a0 tx=0x7f6edc009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:39.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.134+0000 7f6eef08b700 1 -- 172.21.15.29:0/3941484116 shutdown_connections 2024-04-27T01:57:39.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.134+0000 7f6eef08b700 1 --2- 172.21.15.29:0/3941484116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee8105b10 0x7f6ee8105f20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:39.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.134+0000 7f6eef08b700 1 -- 172.21.15.29:0/3941484116 >> 172.21.15.29:0/3941484116 conn(0x7f6ee8101180 msgr2=0x7f6ee8103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:39.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.134+0000 7f6eef08b700 1 -- 172.21.15.29:0/3941484116 shutdown_connections 2024-04-27T01:57:39.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.134+0000 7f6eef08b700 1 -- 172.21.15.29:0/3941484116 wait complete. 2024-04-27T01:57:39.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.135+0000 7f6eef08b700 1 Processor -- start 2024-04-27T01:57:39.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.135+0000 7f6eef08b700 1 -- start start 2024-04-27T01:57:39.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.135+0000 7f6eef08b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee8105b10 0x7f6ee8199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:39.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.135+0000 7f6eef08b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee8199d90 con 0x7f6ee8105b10 2024-04-27T01:57:39.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.135+0000 7f6eece27700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee8105b10 0x7f6ee8199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:39.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.135+0000 7f6eece27700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee8105b10 0x7f6ee8199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44846/0 (socket says 172.21.15.29:44846) 2024-04-27T01:57:39.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.135+0000 7f6eece27700 1 -- 172.21.15.29:0/2426591572 learned_addr learned my addr 172.21.15.29:0/2426591572 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:39.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.135+0000 7f6eece27700 1 -- 172.21.15.29:0/2426591572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6edc009550 con 0x7f6ee8105b10 2024-04-27T01:57:39.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.135+0000 7f6eece27700 1 --2- 172.21.15.29:0/2426591572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee8105b10 0x7f6ee8199850 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6edc011ad0 tx=0x7f6edc011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:39.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.135+0000 7f6ee5ffb700 1 -- 172.21.15.29:0/2426591572 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6edc003ab0 con 0x7f6ee8105b10 2024-04-27T01:57:39.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.135+0000 7f6ee5ffb700 1 -- 172.21.15.29:0/2426591572 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6edc003c10 con 0x7f6ee8105b10 2024-04-27T01:57:39.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.135+0000 7f6ee5ffb700 1 -- 172.21.15.29:0/2426591572 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6edc0198e0 con 0x7f6ee8105b10 2024-04-27T01:57:39.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.135+0000 7f6eef08b700 1 -- 172.21.15.29:0/2426591572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee8199f90 con 0x7f6ee8105b10 2024-04-27T01:57:39.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.136+0000 7f6eef08b700 1 -- 172.21.15.29:0/2426591572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee819a3b0 con 0x7f6ee8105b10 2024-04-27T01:57:39.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.136+0000 7f6ee5ffb700 1 -- 172.21.15.29:0/2426591572 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f6edc019b50 con 0x7f6ee8105b10 2024-04-27T01:57:39.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.137+0000 7f6eef08b700 1 -- 172.21.15.29:0/2426591572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee8062380 con 0x7f6ee8105b10 2024-04-27T01:57:39.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.137+0000 7f6ee5ffb700 1 --2- 172.21.15.29:0/2426591572 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f6ed8038440 0x7f6ed803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:39.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.137+0000 7f6ee5ffb700 1 -- 172.21.15.29:0/2426591572 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6edc04c120 con 0x7f6ee8105b10 2024-04-27T01:57:39.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.137+0000 7f6ee7fff700 1 --2- 172.21.15.29:0/2426591572 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f6ed8038440 0x7f6ed803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:39.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.137+0000 7f6ee7fff700 1 --2- 172.21.15.29:0/2426591572 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f6ed8038440 0x7f6ed803a8f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4006fd0 tx=0x7f6ed4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:39.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.142+0000 7f6ee5ffb700 1 -- 172.21.15.29:0/2426591572 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6edc01a7b0 con 0x7f6ee8105b10 2024-04-27T01:57:39.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.358+0000 7f6eef08b700 1 -- 172.21.15.29:0/2426591572 --> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ee8102610 con 0x7f6ed8038440 2024-04-27T01:57:39.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.362+0000 7f6ee5ffb700 1 -- 172.21.15.29:0/2426591572 <== mgr.14120 v2:172.21.15.29:6800/3873642219 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f6ee8102610 con 0x7f6ed8038440 2024-04-27T01:57:39.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.364+0000 7f6eef08b700 1 -- 172.21.15.29:0/2426591572 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f6ed8038440 msgr2=0x7f6ed803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:39.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.364+0000 7f6eef08b700 1 --2- 172.21.15.29:0/2426591572 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f6ed8038440 0x7f6ed803a8f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4006fd0 tx=0x7f6ed4006e40 comp rx=0 tx=0).stop 2024-04-27T01:57:39.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.364+0000 7f6eef08b700 1 -- 172.21.15.29:0/2426591572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee8105b10 msgr2=0x7f6ee8199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:39.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.364+0000 7f6eef08b700 1 --2- 172.21.15.29:0/2426591572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee8105b10 0x7f6ee8199850 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6edc011ad0 tx=0x7f6edc011bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:39.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.365+0000 7f6eef08b700 1 -- 172.21.15.29:0/2426591572 shutdown_connections 2024-04-27T01:57:39.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.365+0000 7f6eef08b700 1 --2- 172.21.15.29:0/2426591572 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f6ed8038440 0x7f6ed803a8f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:39.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.365+0000 7f6eef08b700 1 --2- 172.21.15.29:0/2426591572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee8105b10 0x7f6ee8199850 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:39.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.365+0000 7f6eef08b700 1 -- 172.21.15.29:0/2426591572 >> 172.21.15.29:0/2426591572 conn(0x7f6ee8101180 msgr2=0x7f6ee8101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:39.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.365+0000 7f6eef08b700 1 -- 172.21.15.29:0/2426591572 shutdown_connections 2024-04-27T01:57:39.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:38.365+0000 7f6eef08b700 1 -- 172.21.15.29:0/2426591572 wait complete. 2024-04-27T01:57:39.088 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying ceph-exporter service with default placement... 2024-04-27T01:57:39.647 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:39 smithi029 ceph-mon[26769]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:57:39.648 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:39 smithi029 ceph-mon[26769]: Saving service crash spec with placement * 2024-04-27T01:57:39.648 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:39 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:40.549 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-04-27T01:57:40.549 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.568+0000 7fc774677700 1 Processor -- start 2024-04-27T01:57:40.549 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.568+0000 7fc774677700 1 -- start start 2024-04-27T01:57:40.549 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.568+0000 7fc774677700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc76c07bdf0 0x7fc76c07a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:40.549 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.568+0000 7fc774677700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c07a830 con 0x7fc76c07bdf0 2024-04-27T01:57:40.550 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.568+0000 7fc772413700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc76c07bdf0 0x7fc76c07a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:40.550 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.568+0000 7fc772413700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc76c07bdf0 0x7fc76c07a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44862/0 (socket says 172.21.15.29:44862) 2024-04-27T01:57:40.550 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.568+0000 7fc772413700 1 -- 172.21.15.29:0/4289164702 learned_addr learned my addr 172.21.15.29:0/4289164702 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:40.550 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.568+0000 7fc772413700 1 -- 172.21.15.29:0/4289164702 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc76c07a970 con 0x7fc76c07bdf0 2024-04-27T01:57:40.550 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.569+0000 7fc772413700 1 --2- 172.21.15.29:0/4289164702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc76c07bdf0 0x7fc76c07a2f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc7680098a0 tx=0x7fc768009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a58f7846462614 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:40.550 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.569+0000 7fc771411700 1 -- 172.21.15.29:0/4289164702 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc768004180 con 0x7fc76c07bdf0 2024-04-27T01:57:40.550 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.569+0000 7fc771411700 1 -- 172.21.15.29:0/4289164702 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc7680042e0 con 0x7fc76c07bdf0 2024-04-27T01:57:40.550 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.569+0000 7fc771411700 1 -- 172.21.15.29:0/4289164702 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc768011af0 con 0x7fc76c07bdf0 2024-04-27T01:57:40.550 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.569+0000 7fc774677700 1 -- 172.21.15.29:0/4289164702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc76c07bdf0 msgr2=0x7fc76c07a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:40.550 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.569+0000 7fc774677700 1 --2- 172.21.15.29:0/4289164702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc76c07bdf0 0x7fc76c07a2f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc7680098a0 tx=0x7fc768009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:40.550 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.570+0000 7fc774677700 1 -- 172.21.15.29:0/4289164702 shutdown_connections 2024-04-27T01:57:40.550 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.570+0000 7fc774677700 1 --2- 172.21.15.29:0/4289164702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc76c07bdf0 0x7fc76c07a2f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:40.550 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.570+0000 7fc774677700 1 -- 172.21.15.29:0/4289164702 >> 172.21.15.29:0/4289164702 conn(0x7fc76c101180 msgr2=0x7fc76c103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:40.551 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.570+0000 7fc774677700 1 -- 172.21.15.29:0/4289164702 shutdown_connections 2024-04-27T01:57:40.551 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.570+0000 7fc774677700 1 -- 172.21.15.29:0/4289164702 wait complete. 2024-04-27T01:57:40.551 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.570+0000 7fc774677700 1 Processor -- start 2024-04-27T01:57:40.551 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.570+0000 7fc774677700 1 -- start start 2024-04-27T01:57:40.551 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.570+0000 7fc774677700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc76c07bdf0 0x7fc76c19baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:40.551 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.570+0000 7fc774677700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c19c030 con 0x7fc76c07bdf0 2024-04-27T01:57:40.551 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.571+0000 7fc772413700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc76c07bdf0 0x7fc76c19baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:40.551 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.571+0000 7fc772413700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc76c07bdf0 0x7fc76c19baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44876/0 (socket says 172.21.15.29:44876) 2024-04-27T01:57:40.551 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.571+0000 7fc772413700 1 -- 172.21.15.29:0/720389951 learned_addr learned my addr 172.21.15.29:0/720389951 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:40.551 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.571+0000 7fc772413700 1 -- 172.21.15.29:0/720389951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc768009550 con 0x7fc76c07bdf0 2024-04-27T01:57:40.551 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.571+0000 7fc772413700 1 --2- 172.21.15.29:0/720389951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc76c07bdf0 0x7fc76c19baf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc768006ba0 tx=0x7fc768011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:40.551 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.571+0000 7fc7637fe700 1 -- 172.21.15.29:0/720389951 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc768003b00 con 0x7fc76c07bdf0 2024-04-27T01:57:40.551 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.571+0000 7fc7637fe700 1 -- 172.21.15.29:0/720389951 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc768003c60 con 0x7fc76c07bdf0 2024-04-27T01:57:40.552 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.571+0000 7fc7637fe700 1 -- 172.21.15.29:0/720389951 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc768019820 con 0x7fc76c07bdf0 2024-04-27T01:57:40.552 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.571+0000 7fc774677700 1 -- 172.21.15.29:0/720389951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc76c19c230 con 0x7fc76c07bdf0 2024-04-27T01:57:40.552 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.571+0000 7fc774677700 1 -- 172.21.15.29:0/720389951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc76c19c650 con 0x7fc76c07bdf0 2024-04-27T01:57:40.552 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.572+0000 7fc7637fe700 1 -- 172.21.15.29:0/720389951 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fc76801ac90 con 0x7fc76c07bdf0 2024-04-27T01:57:40.552 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.572+0000 7fc774677700 1 -- 172.21.15.29:0/720389951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc76c062380 con 0x7fc76c07bdf0 2024-04-27T01:57:40.552 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.572+0000 7fc7637fe700 1 --2- 172.21.15.29:0/720389951 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fc7580383f0 0x7fc75803a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:40.552 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.572+0000 7fc7637fe700 1 -- 172.21.15.29:0/720389951 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc76804c420 con 0x7fc76c07bdf0 2024-04-27T01:57:40.552 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.572+0000 7fc771c12700 1 --2- 172.21.15.29:0/720389951 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fc7580383f0 0x7fc75803a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:40.552 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.573+0000 7fc771c12700 1 --2- 172.21.15.29:0/720389951 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fc7580383f0 0x7fc75803a8a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc75c006fd0 tx=0x7fc75c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:40.552 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.577+0000 7fc7637fe700 1 -- 172.21.15.29:0/720389951 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc768018760 con 0x7fc76c07bdf0 2024-04-27T01:57:40.552 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.790+0000 7fc774677700 1 -- 172.21.15.29:0/720389951 --> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fc76c103440 con 0x7fc7580383f0 2024-04-27T01:57:40.552 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.794+0000 7fc7637fe700 1 -- 172.21.15.29:0/720389951 <== mgr.14120 v2:172.21.15.29:6800/3873642219 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fc76c103440 con 0x7fc7580383f0 2024-04-27T01:57:40.552 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.796+0000 7fc774677700 1 -- 172.21.15.29:0/720389951 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fc7580383f0 msgr2=0x7fc75803a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:40.553 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.796+0000 7fc774677700 1 --2- 172.21.15.29:0/720389951 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fc7580383f0 0x7fc75803a8a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc75c006fd0 tx=0x7fc75c006e40 comp rx=0 tx=0).stop 2024-04-27T01:57:40.553 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.796+0000 7fc774677700 1 -- 172.21.15.29:0/720389951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc76c07bdf0 msgr2=0x7fc76c19baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:40.553 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.796+0000 7fc774677700 1 --2- 172.21.15.29:0/720389951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc76c07bdf0 0x7fc76c19baf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc768006ba0 tx=0x7fc768011b50 comp rx=0 tx=0).stop 2024-04-27T01:57:40.553 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.796+0000 7fc774677700 1 -- 172.21.15.29:0/720389951 shutdown_connections 2024-04-27T01:57:40.553 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.796+0000 7fc774677700 1 --2- 172.21.15.29:0/720389951 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fc7580383f0 0x7fc75803a8a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:40.553 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.796+0000 7fc774677700 1 --2- 172.21.15.29:0/720389951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc76c07bdf0 0x7fc76c19baf0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:40.553 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.796+0000 7fc774677700 1 -- 172.21.15.29:0/720389951 >> 172.21.15.29:0/720389951 conn(0x7fc76c101180 msgr2=0x7fc76c102820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:40.553 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.797+0000 7fc774677700 1 -- 172.21.15.29:0/720389951 shutdown_connections 2024-04-27T01:57:40.553 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:39.797+0000 7fc774677700 1 -- 172.21.15.29:0/720389951 wait complete. 2024-04-27T01:57:40.553 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying prometheus service with default placement... 2024-04-27T01:57:40.793 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:40 smithi029 ceph-mon[26769]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:57:40.793 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:40 smithi029 ceph-mon[26769]: Saving service ceph-exporter spec with placement * 2024-04-27T01:57:40.793 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:40 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:40.793 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:40 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:41.882 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-04-27T01:57:41.882 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.030+0000 7fa15bb5b700 1 Processor -- start 2024-04-27T01:57:41.882 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.030+0000 7fa15bb5b700 1 -- start start 2024-04-27T01:57:41.882 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.030+0000 7fa15bb5b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa154105b10 0x7fa154105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:41.882 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.030+0000 7fa15bb5b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa154106460 con 0x7fa154105b10 2024-04-27T01:57:41.882 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.031+0000 7fa1598f7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa154105b10 0x7fa154105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:41.882 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.031+0000 7fa1598f7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa154105b10 0x7fa154105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44878/0 (socket says 172.21.15.29:44878) 2024-04-27T01:57:41.882 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.031+0000 7fa1598f7700 1 -- 172.21.15.29:0/3697149172 learned_addr learned my addr 172.21.15.29:0/3697149172 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:41.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.031+0000 7fa1598f7700 1 -- 172.21.15.29:0/3697149172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1541065a0 con 0x7fa154105b10 2024-04-27T01:57:41.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.031+0000 7fa1598f7700 1 --2- 172.21.15.29:0/3697149172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa154105b10 0x7fa154105f20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa1500098a0 tx=0x7fa150009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=964ad2783b56e60a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:41.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.031+0000 7fa1588f5700 1 -- 172.21.15.29:0/3697149172 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa150004180 con 0x7fa154105b10 2024-04-27T01:57:41.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.032+0000 7fa1588f5700 1 -- 172.21.15.29:0/3697149172 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa1500042e0 con 0x7fa154105b10 2024-04-27T01:57:41.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.032+0000 7fa1588f5700 1 -- 172.21.15.29:0/3697149172 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa150011af0 con 0x7fa154105b10 2024-04-27T01:57:41.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.032+0000 7fa15bb5b700 1 -- 172.21.15.29:0/3697149172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa154105b10 msgr2=0x7fa154105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:41.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.032+0000 7fa15bb5b700 1 --2- 172.21.15.29:0/3697149172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa154105b10 0x7fa154105f20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa1500098a0 tx=0x7fa150009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:41.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.032+0000 7fa15bb5b700 1 -- 172.21.15.29:0/3697149172 shutdown_connections 2024-04-27T01:57:41.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.032+0000 7fa15bb5b700 1 --2- 172.21.15.29:0/3697149172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa154105b10 0x7fa154105f20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:41.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.032+0000 7fa15bb5b700 1 -- 172.21.15.29:0/3697149172 >> 172.21.15.29:0/3697149172 conn(0x7fa154101180 msgr2=0x7fa154103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:41.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.032+0000 7fa15bb5b700 1 -- 172.21.15.29:0/3697149172 shutdown_connections 2024-04-27T01:57:41.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.032+0000 7fa15bb5b700 1 -- 172.21.15.29:0/3697149172 wait complete. 2024-04-27T01:57:41.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.033+0000 7fa15bb5b700 1 Processor -- start 2024-04-27T01:57:41.883 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.033+0000 7fa15bb5b700 1 -- start start 2024-04-27T01:57:41.884 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.033+0000 7fa15bb5b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa154105b10 0x7fa154199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:41.884 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.033+0000 7fa15bb5b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa154199dd0 con 0x7fa154105b10 2024-04-27T01:57:41.884 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.033+0000 7fa1598f7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa154105b10 0x7fa154199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:41.884 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.033+0000 7fa1598f7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa154105b10 0x7fa154199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44888/0 (socket says 172.21.15.29:44888) 2024-04-27T01:57:41.884 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.033+0000 7fa1598f7700 1 -- 172.21.15.29:0/2625245437 learned_addr learned my addr 172.21.15.29:0/2625245437 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:41.884 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.034+0000 7fa1598f7700 1 -- 172.21.15.29:0/2625245437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa150009550 con 0x7fa154105b10 2024-04-27T01:57:41.884 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.034+0000 7fa1598f7700 1 --2- 172.21.15.29:0/2625245437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa154105b10 0x7fa154199890 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa150006ba0 tx=0x7fa150011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:41.884 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.034+0000 7fa146ffd700 1 -- 172.21.15.29:0/2625245437 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa150003b00 con 0x7fa154105b10 2024-04-27T01:57:41.884 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.034+0000 7fa146ffd700 1 -- 172.21.15.29:0/2625245437 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa15001a8e0 con 0x7fa154105b10 2024-04-27T01:57:41.884 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.034+0000 7fa15bb5b700 1 -- 172.21.15.29:0/2625245437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa154199fd0 con 0x7fa154105b10 2024-04-27T01:57:41.884 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.034+0000 7fa146ffd700 1 -- 172.21.15.29:0/2625245437 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1500199c0 con 0x7fa154105b10 2024-04-27T01:57:41.884 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.034+0000 7fa15bb5b700 1 -- 172.21.15.29:0/2625245437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa15419a3f0 con 0x7fa154105b10 2024-04-27T01:57:41.884 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.035+0000 7fa146ffd700 1 -- 172.21.15.29:0/2625245437 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fa150019c30 con 0x7fa154105b10 2024-04-27T01:57:41.884 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.035+0000 7fa15bb5b700 1 -- 172.21.15.29:0/2625245437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa154062380 con 0x7fa154105b10 2024-04-27T01:57:41.885 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.035+0000 7fa146ffd700 1 --2- 172.21.15.29:0/2625245437 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fa1400383f0 0x7fa14003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:41.885 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.035+0000 7fa146ffd700 1 -- 172.21.15.29:0/2625245437 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa15004c770 con 0x7fa154105b10 2024-04-27T01:57:41.885 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.036+0000 7fa1590f6700 1 --2- 172.21.15.29:0/2625245437 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fa1400383f0 0x7fa14003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:41.885 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.036+0000 7fa1590f6700 1 --2- 172.21.15.29:0/2625245437 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fa1400383f0 0x7fa14003a8a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa148006fd0 tx=0x7fa148006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:41.885 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.040+0000 7fa146ffd700 1 -- 172.21.15.29:0/2625245437 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa15001a430 con 0x7fa154105b10 2024-04-27T01:57:41.885 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.260+0000 7fa15bb5b700 1 -- 172.21.15.29:0/2625245437 --> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fa154102670 con 0x7fa1400383f0 2024-04-27T01:57:41.885 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.265+0000 7fa146ffd700 1 -- 172.21.15.29:0/2625245437 <== mgr.14120 v2:172.21.15.29:6800/3873642219 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fa154102670 con 0x7fa1400383f0 2024-04-27T01:57:41.885 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.268+0000 7fa15bb5b700 1 -- 172.21.15.29:0/2625245437 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fa1400383f0 msgr2=0x7fa14003a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:41.885 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.268+0000 7fa15bb5b700 1 --2- 172.21.15.29:0/2625245437 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fa1400383f0 0x7fa14003a8a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa148006fd0 tx=0x7fa148006e40 comp rx=0 tx=0).stop 2024-04-27T01:57:41.885 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.268+0000 7fa15bb5b700 1 -- 172.21.15.29:0/2625245437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa154105b10 msgr2=0x7fa154199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:41.885 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.268+0000 7fa15bb5b700 1 --2- 172.21.15.29:0/2625245437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa154105b10 0x7fa154199890 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa150006ba0 tx=0x7fa150011b50 comp rx=0 tx=0).stop 2024-04-27T01:57:41.885 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.268+0000 7fa15bb5b700 1 -- 172.21.15.29:0/2625245437 shutdown_connections 2024-04-27T01:57:41.885 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.268+0000 7fa15bb5b700 1 --2- 172.21.15.29:0/2625245437 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fa1400383f0 0x7fa14003a8a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:41.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.268+0000 7fa15bb5b700 1 --2- 172.21.15.29:0/2625245437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa154105b10 0x7fa154199890 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:41.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.268+0000 7fa15bb5b700 1 -- 172.21.15.29:0/2625245437 >> 172.21.15.29:0/2625245437 conn(0x7fa154101180 msgr2=0x7fa154101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:41.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.268+0000 7fa15bb5b700 1 -- 172.21.15.29:0/2625245437 shutdown_connections 2024-04-27T01:57:41.886 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:41.268+0000 7fa15bb5b700 1 -- 172.21.15.29:0/2625245437 wait complete. 2024-04-27T01:57:41.886 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying grafana service with default placement... 2024-04-27T01:57:42.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:42 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:42.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:42 smithi029 ceph-mon[26769]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:57:42.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:42 smithi029 ceph-mon[26769]: Saving service prometheus spec with placement count:1 2024-04-27T01:57:42.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:42 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:43.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:43 smithi029 ceph-mon[26769]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:57:43.702 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:43 smithi029 ceph-mon[26769]: Saving service grafana spec with placement count:1 2024-04-27T01:57:43.702 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:43 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:43.730 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-04-27T01:57:43.730 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.319+0000 7f612266c700 1 Processor -- start 2024-04-27T01:57:43.731 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.319+0000 7f612266c700 1 -- start start 2024-04-27T01:57:43.731 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.319+0000 7f612266c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f611c106610 0x7f611c1089f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:43.731 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.319+0000 7f612266c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c074510 con 0x7f611c106610 2024-04-27T01:57:43.731 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.319+0000 7f611bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f611c106610 0x7f611c1089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:43.731 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.319+0000 7f611bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f611c106610 0x7f611c1089f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44896/0 (socket says 172.21.15.29:44896) 2024-04-27T01:57:43.731 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.319+0000 7f611bfff700 1 -- 172.21.15.29:0/472173357 learned_addr learned my addr 172.21.15.29:0/472173357 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:43.731 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.320+0000 7f611bfff700 1 -- 172.21.15.29:0/472173357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f611c074650 con 0x7f611c106610 2024-04-27T01:57:43.731 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.320+0000 7f611bfff700 1 --2- 172.21.15.29:0/472173357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f611c106610 0x7f611c1089f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f610c0098a0 tx=0x7f610c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f96839766d954d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:43.731 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.320+0000 7f611affd700 1 -- 172.21.15.29:0/472173357 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f610c004180 con 0x7f611c106610 2024-04-27T01:57:43.731 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.320+0000 7f611affd700 1 -- 172.21.15.29:0/472173357 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f610c0042e0 con 0x7f611c106610 2024-04-27T01:57:43.731 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.320+0000 7f611affd700 1 -- 172.21.15.29:0/472173357 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f610c011af0 con 0x7f611c106610 2024-04-27T01:57:43.731 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.321+0000 7f612266c700 1 -- 172.21.15.29:0/472173357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f611c106610 msgr2=0x7f611c1089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:43.731 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.321+0000 7f612266c700 1 --2- 172.21.15.29:0/472173357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f611c106610 0x7f611c1089f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f610c0098a0 tx=0x7f610c009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:43.732 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.321+0000 7f612266c700 1 -- 172.21.15.29:0/472173357 shutdown_connections 2024-04-27T01:57:43.732 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.321+0000 7f612266c700 1 --2- 172.21.15.29:0/472173357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f611c106610 0x7f611c1089f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:43.732 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.321+0000 7f612266c700 1 -- 172.21.15.29:0/472173357 >> 172.21.15.29:0/472173357 conn(0x7f611c100070 msgr2=0x7f611c102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:43.732 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.321+0000 7f612266c700 1 -- 172.21.15.29:0/472173357 shutdown_connections 2024-04-27T01:57:43.732 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.321+0000 7f612266c700 1 -- 172.21.15.29:0/472173357 wait complete. 2024-04-27T01:57:43.732 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.321+0000 7f612266c700 1 Processor -- start 2024-04-27T01:57:43.732 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.321+0000 7f612266c700 1 -- start start 2024-04-27T01:57:43.732 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.322+0000 7f612266c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f611c106610 0x7f611c199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:43.732 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.322+0000 7f612266c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c199dd0 con 0x7f611c106610 2024-04-27T01:57:43.732 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.322+0000 7f611bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f611c106610 0x7f611c199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:43.732 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.322+0000 7f611bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f611c106610 0x7f611c199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44904/0 (socket says 172.21.15.29:44904) 2024-04-27T01:57:43.732 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.322+0000 7f611bfff700 1 -- 172.21.15.29:0/4229299264 learned_addr learned my addr 172.21.15.29:0/4229299264 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:43.732 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.322+0000 7f611bfff700 1 -- 172.21.15.29:0/4229299264 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610c009550 con 0x7f611c106610 2024-04-27T01:57:43.733 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.322+0000 7f611bfff700 1 --2- 172.21.15.29:0/4229299264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f611c106610 0x7f611c199890 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f610c006ba0 tx=0x7f610c011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:43.733 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.322+0000 7f61197fa700 1 -- 172.21.15.29:0/4229299264 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f610c003b00 con 0x7f611c106610 2024-04-27T01:57:43.733 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.322+0000 7f61197fa700 1 -- 172.21.15.29:0/4229299264 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f610c01a8e0 con 0x7f611c106610 2024-04-27T01:57:43.733 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.322+0000 7f612266c700 1 -- 172.21.15.29:0/4229299264 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f611c199fd0 con 0x7f611c106610 2024-04-27T01:57:43.733 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.322+0000 7f61197fa700 1 -- 172.21.15.29:0/4229299264 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f610c019990 con 0x7f611c106610 2024-04-27T01:57:43.733 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.323+0000 7f612266c700 1 -- 172.21.15.29:0/4229299264 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f611c19a3f0 con 0x7f611c106610 2024-04-27T01:57:43.733 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.323+0000 7f61197fa700 1 -- 172.21.15.29:0/4229299264 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f610c019c00 con 0x7f611c106610 2024-04-27T01:57:43.733 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.323+0000 7f612266c700 1 -- 172.21.15.29:0/4229299264 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f611c109bf0 con 0x7f611c106610 2024-04-27T01:57:43.733 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.324+0000 7f61197fa700 1 --2- 172.21.15.29:0/4229299264 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f61040383f0 0x7f610403a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:43.733 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.324+0000 7f61197fa700 1 -- 172.21.15.29:0/4229299264 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f610c04c840 con 0x7f611c106610 2024-04-27T01:57:43.733 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.324+0000 7f611b7fe700 1 --2- 172.21.15.29:0/4229299264 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f61040383f0 0x7f610403a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:43.733 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.324+0000 7f611b7fe700 1 --2- 172.21.15.29:0/4229299264 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f61040383f0 0x7f610403a8a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6110006fd0 tx=0x7f6110006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:43.733 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.328+0000 7f61197fa700 1 -- 172.21.15.29:0/4229299264 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f610c02d440 con 0x7f611c106610 2024-04-27T01:57:43.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.551+0000 7f612266c700 1 -- 172.21.15.29:0/4229299264 --> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f611c19d250 con 0x7f61040383f0 2024-04-27T01:57:43.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.555+0000 7f61197fa700 1 -- 172.21.15.29:0/4229299264 <== mgr.14120 v2:172.21.15.29:6800/3873642219 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f611c19d250 con 0x7f61040383f0 2024-04-27T01:57:43.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.557+0000 7f612266c700 1 -- 172.21.15.29:0/4229299264 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f61040383f0 msgr2=0x7f610403a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:43.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.557+0000 7f612266c700 1 --2- 172.21.15.29:0/4229299264 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f61040383f0 0x7f610403a8a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6110006fd0 tx=0x7f6110006e40 comp rx=0 tx=0).stop 2024-04-27T01:57:43.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.557+0000 7f612266c700 1 -- 172.21.15.29:0/4229299264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f611c106610 msgr2=0x7f611c199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:43.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.557+0000 7f612266c700 1 --2- 172.21.15.29:0/4229299264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f611c106610 0x7f611c199890 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f610c006ba0 tx=0x7f610c011b50 comp rx=0 tx=0).stop 2024-04-27T01:57:43.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.557+0000 7f612266c700 1 -- 172.21.15.29:0/4229299264 shutdown_connections 2024-04-27T01:57:43.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.557+0000 7f612266c700 1 --2- 172.21.15.29:0/4229299264 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f61040383f0 0x7f610403a8a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:43.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.557+0000 7f612266c700 1 --2- 172.21.15.29:0/4229299264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f611c106610 0x7f611c199890 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:43.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.557+0000 7f612266c700 1 -- 172.21.15.29:0/4229299264 >> 172.21.15.29:0/4229299264 conn(0x7f611c100070 msgr2=0x7f611c101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:43.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.557+0000 7f612266c700 1 -- 172.21.15.29:0/4229299264 shutdown_connections 2024-04-27T01:57:43.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:42.557+0000 7f612266c700 1 -- 172.21.15.29:0/4229299264 wait complete. 2024-04-27T01:57:43.734 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying node-exporter service with default placement... 2024-04-27T01:57:44.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:44 smithi029 ceph-mon[26769]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:57:44.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:44 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:45.375 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-04-27T01:57:45.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.209+0000 7f39a07af700 1 Processor -- start 2024-04-27T01:57:45.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.209+0000 7f39a07af700 1 -- start start 2024-04-27T01:57:45.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.209+0000 7f39a07af700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3998106670 0x7f3998108a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:45.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.209+0000 7f39a07af700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39980744b0 con 0x7f3998106670 2024-04-27T01:57:45.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.209+0000 7f399e54b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3998106670 0x7f3998108a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:45.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.209+0000 7f399e54b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3998106670 0x7f3998108a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44912/0 (socket says 172.21.15.29:44912) 2024-04-27T01:57:45.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.209+0000 7f399e54b700 1 -- 172.21.15.29:0/1865063462 learned_addr learned my addr 172.21.15.29:0/1865063462 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:45.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.209+0000 7f399e54b700 1 -- 172.21.15.29:0/1865063462 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39980745f0 con 0x7f3998106670 2024-04-27T01:57:45.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.209+0000 7f399e54b700 1 --2- 172.21.15.29:0/1865063462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3998106670 0x7f3998108a50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f39940098a0 tx=0x7f3994009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=244e90603297844a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:45.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.209+0000 7f399d549700 1 -- 172.21.15.29:0/1865063462 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3994004180 con 0x7f3998106670 2024-04-27T01:57:45.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.209+0000 7f399d549700 1 -- 172.21.15.29:0/1865063462 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f39940042e0 con 0x7f3998106670 2024-04-27T01:57:45.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.209+0000 7f399d549700 1 -- 172.21.15.29:0/1865063462 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3994011af0 con 0x7f3998106670 2024-04-27T01:57:45.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.210+0000 7f39a07af700 1 -- 172.21.15.29:0/1865063462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3998106670 msgr2=0x7f3998108a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:45.377 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.210+0000 7f39a07af700 1 --2- 172.21.15.29:0/1865063462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3998106670 0x7f3998108a50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f39940098a0 tx=0x7f3994009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:45.377 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.210+0000 7f39a07af700 1 -- 172.21.15.29:0/1865063462 shutdown_connections 2024-04-27T01:57:45.377 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.210+0000 7f39a07af700 1 --2- 172.21.15.29:0/1865063462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3998106670 0x7f3998108a50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:45.377 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.210+0000 7f39a07af700 1 -- 172.21.15.29:0/1865063462 >> 172.21.15.29:0/1865063462 conn(0x7f3998100070 msgr2=0x7f3998102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:45.377 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.210+0000 7f39a07af700 1 -- 172.21.15.29:0/1865063462 shutdown_connections 2024-04-27T01:57:45.377 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.210+0000 7f39a07af700 1 -- 172.21.15.29:0/1865063462 wait complete. 2024-04-27T01:57:45.377 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.211+0000 7f39a07af700 1 Processor -- start 2024-04-27T01:57:45.377 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.211+0000 7f39a07af700 1 -- start start 2024-04-27T01:57:45.377 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.211+0000 7f39a07af700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3998106670 0x7f3998199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:45.377 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.211+0000 7f39a07af700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3998199dd0 con 0x7f3998106670 2024-04-27T01:57:45.377 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.211+0000 7f399e54b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3998106670 0x7f3998199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:45.377 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.211+0000 7f399e54b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3998106670 0x7f3998199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44924/0 (socket says 172.21.15.29:44924) 2024-04-27T01:57:45.377 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.211+0000 7f399e54b700 1 -- 172.21.15.29:0/1256411470 learned_addr learned my addr 172.21.15.29:0/1256411470 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:45.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.211+0000 7f399e54b700 1 -- 172.21.15.29:0/1256411470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3994009550 con 0x7f3998106670 2024-04-27T01:57:45.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.212+0000 7f399e54b700 1 --2- 172.21.15.29:0/1256411470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3998106670 0x7f3998199890 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3994000c00 tx=0x7f3994011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:45.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.212+0000 7f398b7fe700 1 -- 172.21.15.29:0/1256411470 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3994003a70 con 0x7f3998106670 2024-04-27T01:57:45.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.212+0000 7f39a07af700 1 -- 172.21.15.29:0/1256411470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3998199fd0 con 0x7f3998106670 2024-04-27T01:57:45.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.212+0000 7f398b7fe700 1 -- 172.21.15.29:0/1256411470 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3994003bd0 con 0x7f3998106670 2024-04-27T01:57:45.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.212+0000 7f398b7fe700 1 -- 172.21.15.29:0/1256411470 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39940238c0 con 0x7f3998106670 2024-04-27T01:57:45.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.212+0000 7f39a07af700 1 -- 172.21.15.29:0/1256411470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f399819a3f0 con 0x7f3998106670 2024-04-27T01:57:45.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.213+0000 7f398b7fe700 1 -- 172.21.15.29:0/1256411470 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f3994023b80 con 0x7f3998106670 2024-04-27T01:57:45.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.213+0000 7f39a07af700 1 -- 172.21.15.29:0/1256411470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3998109bf0 con 0x7f3998106670 2024-04-27T01:57:45.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.213+0000 7f398b7fe700 1 --2- 172.21.15.29:0/1256411470 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f39840383f0 0x7f398403a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:45.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.213+0000 7f398b7fe700 1 -- 172.21.15.29:0/1256411470 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f399404c1f0 con 0x7f3998106670 2024-04-27T01:57:45.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.213+0000 7f399dd4a700 1 --2- 172.21.15.29:0/1256411470 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f39840383f0 0x7f398403a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:45.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.214+0000 7f399dd4a700 1 --2- 172.21.15.29:0/1256411470 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f39840383f0 0x7f398403a8a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f398c006fd0 tx=0x7f398c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:45.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.218+0000 7f398b7fe700 1 -- 172.21.15.29:0/1256411470 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f39940193c0 con 0x7f3998106670 2024-04-27T01:57:45.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.430+0000 7f39a07af700 1 -- 172.21.15.29:0/1256411470 --> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f3998102380 con 0x7f39840383f0 2024-04-27T01:57:45.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.435+0000 7f398b7fe700 1 -- 172.21.15.29:0/1256411470 <== mgr.14120 v2:172.21.15.29:6800/3873642219 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f3998102380 con 0x7f39840383f0 2024-04-27T01:57:45.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.437+0000 7f39a07af700 1 -- 172.21.15.29:0/1256411470 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f39840383f0 msgr2=0x7f398403a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:45.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.437+0000 7f39a07af700 1 --2- 172.21.15.29:0/1256411470 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f39840383f0 0x7f398403a8a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f398c006fd0 tx=0x7f398c006e40 comp rx=0 tx=0).stop 2024-04-27T01:57:45.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.437+0000 7f39a07af700 1 -- 172.21.15.29:0/1256411470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3998106670 msgr2=0x7f3998199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:45.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.437+0000 7f39a07af700 1 --2- 172.21.15.29:0/1256411470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3998106670 0x7f3998199890 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3994000c00 tx=0x7f3994011b50 comp rx=0 tx=0).stop 2024-04-27T01:57:45.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.437+0000 7f39a07af700 1 -- 172.21.15.29:0/1256411470 shutdown_connections 2024-04-27T01:57:45.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.437+0000 7f39a07af700 1 --2- 172.21.15.29:0/1256411470 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f39840383f0 0x7f398403a8a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:45.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.437+0000 7f39a07af700 1 --2- 172.21.15.29:0/1256411470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3998106670 0x7f3998199890 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:45.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.437+0000 7f39a07af700 1 -- 172.21.15.29:0/1256411470 >> 172.21.15.29:0/1256411470 conn(0x7f3998100070 msgr2=0x7f3998101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:45.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.437+0000 7f39a07af700 1 -- 172.21.15.29:0/1256411470 shutdown_connections 2024-04-27T01:57:45.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:44.437+0000 7f39a07af700 1 -- 172.21.15.29:0/1256411470 wait complete. 2024-04-27T01:57:45.380 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying alertmanager service with default placement... 2024-04-27T01:57:45.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:45 smithi029 ceph-mon[26769]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:57:46.011 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:45 smithi029 ceph-mon[26769]: Saving service node-exporter spec with placement * 2024-04-27T01:57:46.689 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-04-27T01:57:46.689 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.865+0000 7f6e7f468700 1 Processor -- start 2024-04-27T01:57:46.689 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.866+0000 7f6e7f468700 1 -- start start 2024-04-27T01:57:46.689 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.866+0000 7f6e7f468700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103e00 0x7f6e78104210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:46.689 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.866+0000 7f6e7f468700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e78104750 con 0x7f6e78103e00 2024-04-27T01:57:46.689 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.866+0000 7f6e7d204700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103e00 0x7f6e78104210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:46.689 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.866+0000 7f6e7d204700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103e00 0x7f6e78104210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44934/0 (socket says 172.21.15.29:44934) 2024-04-27T01:57:46.689 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.866+0000 7f6e7d204700 1 -- 172.21.15.29:0/1947411938 learned_addr learned my addr 172.21.15.29:0/1947411938 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:46.689 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.866+0000 7f6e7d204700 1 -- 172.21.15.29:0/1947411938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e78104890 con 0x7f6e78103e00 2024-04-27T01:57:46.690 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.867+0000 7f6e7d204700 1 --2- 172.21.15.29:0/1947411938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103e00 0x7f6e78104210 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6e680098a0 tx=0x7f6e68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17cd6e3800c8de41 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:46.690 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.867+0000 7f6e6ffff700 1 -- 172.21.15.29:0/1947411938 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e68004180 con 0x7f6e78103e00 2024-04-27T01:57:46.690 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.867+0000 7f6e6ffff700 1 -- 172.21.15.29:0/1947411938 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6e680042e0 con 0x7f6e78103e00 2024-04-27T01:57:46.690 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.867+0000 7f6e6ffff700 1 -- 172.21.15.29:0/1947411938 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e68011af0 con 0x7f6e78103e00 2024-04-27T01:57:46.690 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.867+0000 7f6e7f468700 1 -- 172.21.15.29:0/1947411938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103e00 msgr2=0x7f6e78104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:46.690 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.867+0000 7f6e7f468700 1 --2- 172.21.15.29:0/1947411938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103e00 0x7f6e78104210 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6e680098a0 tx=0x7f6e68009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:46.690 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.868+0000 7f6e7f468700 1 -- 172.21.15.29:0/1947411938 shutdown_connections 2024-04-27T01:57:46.690 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.868+0000 7f6e7f468700 1 --2- 172.21.15.29:0/1947411938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103e00 0x7f6e78104210 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:46.690 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.868+0000 7f6e7f468700 1 -- 172.21.15.29:0/1947411938 >> 172.21.15.29:0/1947411938 conn(0x7f6e780ff7e0 msgr2=0x7f6e78101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:46.690 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.868+0000 7f6e7f468700 1 -- 172.21.15.29:0/1947411938 shutdown_connections 2024-04-27T01:57:46.690 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.868+0000 7f6e7f468700 1 -- 172.21.15.29:0/1947411938 wait complete. 2024-04-27T01:57:46.690 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.868+0000 7f6e7f468700 1 Processor -- start 2024-04-27T01:57:46.690 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.868+0000 7f6e7f468700 1 -- start start 2024-04-27T01:57:46.691 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.869+0000 7f6e7f468700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103e00 0x7f6e78106790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:46.691 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.869+0000 7f6e7f468700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e78106cd0 con 0x7f6e78103e00 2024-04-27T01:57:46.691 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.869+0000 7f6e7d204700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103e00 0x7f6e78106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:46.691 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.869+0000 7f6e7d204700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103e00 0x7f6e78106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44940/0 (socket says 172.21.15.29:44940) 2024-04-27T01:57:46.691 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.869+0000 7f6e7d204700 1 -- 172.21.15.29:0/1641262684 learned_addr learned my addr 172.21.15.29:0/1641262684 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:46.691 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.869+0000 7f6e7d204700 1 -- 172.21.15.29:0/1641262684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e68009550 con 0x7f6e78103e00 2024-04-27T01:57:46.691 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.869+0000 7f6e7d204700 1 --2- 172.21.15.29:0/1641262684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103e00 0x7f6e78106790 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6e68006ba0 tx=0x7f6e68011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:46.691 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.869+0000 7f6e6e7fc700 1 -- 172.21.15.29:0/1641262684 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e68003b00 con 0x7f6e78103e00 2024-04-27T01:57:46.691 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.869+0000 7f6e6e7fc700 1 -- 172.21.15.29:0/1641262684 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6e6801a8e0 con 0x7f6e78103e00 2024-04-27T01:57:46.691 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.869+0000 7f6e7f468700 1 -- 172.21.15.29:0/1641262684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e78104de0 con 0x7f6e78103e00 2024-04-27T01:57:46.691 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.869+0000 7f6e6e7fc700 1 -- 172.21.15.29:0/1641262684 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e68019990 con 0x7f6e78103e00 2024-04-27T01:57:46.691 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.870+0000 7f6e7f468700 1 -- 172.21.15.29:0/1641262684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e78105200 con 0x7f6e78103e00 2024-04-27T01:57:46.691 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.870+0000 7f6e6e7fc700 1 -- 172.21.15.29:0/1641262684 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f6e68019c50 con 0x7f6e78103e00 2024-04-27T01:57:46.692 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.870+0000 7f6e7f468700 1 -- 172.21.15.29:0/1641262684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e78062380 con 0x7f6e78103e00 2024-04-27T01:57:46.692 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.871+0000 7f6e6e7fc700 1 --2- 172.21.15.29:0/1641262684 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f6e640383a0 0x7f6e6403a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:46.692 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.871+0000 7f6e6e7fc700 1 -- 172.21.15.29:0/1641262684 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6e6804c800 con 0x7f6e78103e00 2024-04-27T01:57:46.692 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.871+0000 7f6e7ca03700 1 --2- 172.21.15.29:0/1641262684 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f6e640383a0 0x7f6e6403a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:46.692 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.871+0000 7f6e7ca03700 1 --2- 172.21.15.29:0/1641262684 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f6e640383a0 0x7f6e6403a850 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6e74006fd0 tx=0x7f6e74006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:46.692 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:45.876+0000 7f6e6e7fc700 1 -- 172.21.15.29:0/1641262684 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6e6802d440 con 0x7f6e78103e00 2024-04-27T01:57:46.692 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:46.090+0000 7f6e7f468700 1 -- 172.21.15.29:0/1641262684 --> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e78100cf0 con 0x7f6e640383a0 2024-04-27T01:57:46.692 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:46.094+0000 7f6e6e7fc700 1 -- 172.21.15.29:0/1641262684 <== mgr.14120 v2:172.21.15.29:6800/3873642219 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f6e78100cf0 con 0x7f6e640383a0 2024-04-27T01:57:46.692 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:46.096+0000 7f6e7f468700 1 -- 172.21.15.29:0/1641262684 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f6e640383a0 msgr2=0x7f6e6403a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:46.692 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:46.096+0000 7f6e7f468700 1 --2- 172.21.15.29:0/1641262684 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f6e640383a0 0x7f6e6403a850 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6e74006fd0 tx=0x7f6e74006e40 comp rx=0 tx=0).stop 2024-04-27T01:57:46.692 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:46.096+0000 7f6e7f468700 1 -- 172.21.15.29:0/1641262684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103e00 msgr2=0x7f6e78106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:46.692 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:46.096+0000 7f6e7f468700 1 --2- 172.21.15.29:0/1641262684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103e00 0x7f6e78106790 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6e68006ba0 tx=0x7f6e68011b50 comp rx=0 tx=0).stop 2024-04-27T01:57:46.692 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:46.096+0000 7f6e7f468700 1 -- 172.21.15.29:0/1641262684 shutdown_connections 2024-04-27T01:57:46.692 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:46.096+0000 7f6e7f468700 1 --2- 172.21.15.29:0/1641262684 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f6e640383a0 0x7f6e6403a850 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:46.693 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:46.096+0000 7f6e7f468700 1 --2- 172.21.15.29:0/1641262684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103e00 0x7f6e78106790 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:46.693 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:46.096+0000 7f6e7f468700 1 -- 172.21.15.29:0/1641262684 >> 172.21.15.29:0/1641262684 conn(0x7f6e780ff7e0 msgr2=0x7f6e781005e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:46.693 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:46.097+0000 7f6e7f468700 1 -- 172.21.15.29:0/1641262684 shutdown_connections 2024-04-27T01:57:46.693 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:46.097+0000 7f6e7f468700 1 -- 172.21.15.29:0/1641262684 wait complete. 2024-04-27T01:57:46.936 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:46 smithi029 ceph-mon[26769]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:57:46.936 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:46 smithi029 ceph-mon[26769]: from='mgr.14120 172.21.15.29:0/707109264' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:57:47.945 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:47 smithi029 ceph-mon[26769]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:57:47.945 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:47 smithi029 ceph-mon[26769]: Saving service alertmanager spec with placement count:1 2024-04-27T01:57:47.945 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:47 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/8289713' entity='client.admin' 2024-04-27T01:57:47.955 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.171+0000 7f47c1030700 1 Processor -- start 2024-04-27T01:57:47.955 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.171+0000 7f47c1030700 1 -- start start 2024-04-27T01:57:47.955 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.172+0000 7f47c1030700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47bc105b10 0x7f47bc105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:47.955 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.172+0000 7f47c1030700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc106460 con 0x7f47bc105b10 2024-04-27T01:57:47.955 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.172+0000 7f47bad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47bc105b10 0x7f47bc105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:47.955 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.172+0000 7f47bad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47bc105b10 0x7f47bc105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36772/0 (socket says 172.21.15.29:36772) 2024-04-27T01:57:47.955 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.172+0000 7f47bad9d700 1 -- 172.21.15.29:0/1592724785 learned_addr learned my addr 172.21.15.29:0/1592724785 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:47.955 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.172+0000 7f47bad9d700 1 -- 172.21.15.29:0/1592724785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47bc1065a0 con 0x7f47bc105b10 2024-04-27T01:57:47.955 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.172+0000 7f47bad9d700 1 --2- 172.21.15.29:0/1592724785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47bc105b10 0x7f47bc105f20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f47ac0098a0 tx=0x7f47ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b270a82a3aa12003 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:47.955 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.172+0000 7f47b9d9b700 1 -- 172.21.15.29:0/1592724785 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47ac004180 con 0x7f47bc105b10 2024-04-27T01:57:47.955 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.172+0000 7f47b9d9b700 1 -- 172.21.15.29:0/1592724785 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f47ac0042e0 con 0x7f47bc105b10 2024-04-27T01:57:47.955 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.172+0000 7f47b9d9b700 1 -- 172.21.15.29:0/1592724785 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47ac011af0 con 0x7f47bc105b10 2024-04-27T01:57:47.956 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.173+0000 7f47c1030700 1 -- 172.21.15.29:0/1592724785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47bc105b10 msgr2=0x7f47bc105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:47.956 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.173+0000 7f47c1030700 1 --2- 172.21.15.29:0/1592724785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47bc105b10 0x7f47bc105f20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f47ac0098a0 tx=0x7f47ac009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:47.956 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.173+0000 7f47c1030700 1 -- 172.21.15.29:0/1592724785 shutdown_connections 2024-04-27T01:57:47.956 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.173+0000 7f47c1030700 1 --2- 172.21.15.29:0/1592724785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47bc105b10 0x7f47bc105f20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:47.956 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.173+0000 7f47c1030700 1 -- 172.21.15.29:0/1592724785 >> 172.21.15.29:0/1592724785 conn(0x7f47bc101180 msgr2=0x7f47bc103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:47.956 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.173+0000 7f47c1030700 1 -- 172.21.15.29:0/1592724785 shutdown_connections 2024-04-27T01:57:47.956 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.173+0000 7f47c1030700 1 -- 172.21.15.29:0/1592724785 wait complete. 2024-04-27T01:57:47.956 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.174+0000 7f47c1030700 1 Processor -- start 2024-04-27T01:57:47.956 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.174+0000 7f47c1030700 1 -- start start 2024-04-27T01:57:47.956 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.174+0000 7f47c1030700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47bc105b10 0x7f47bc199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:47.956 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.174+0000 7f47c1030700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc199dd0 con 0x7f47bc105b10 2024-04-27T01:57:47.956 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.174+0000 7f47bad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47bc105b10 0x7f47bc199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:47.956 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.174+0000 7f47bad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47bc105b10 0x7f47bc199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36786/0 (socket says 172.21.15.29:36786) 2024-04-27T01:57:47.957 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.174+0000 7f47bad9d700 1 -- 172.21.15.29:0/8289713 learned_addr learned my addr 172.21.15.29:0/8289713 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:47.957 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.175+0000 7f47bad9d700 1 -- 172.21.15.29:0/8289713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ac009550 con 0x7f47bc105b10 2024-04-27T01:57:47.957 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.175+0000 7f47bad9d700 1 --2- 172.21.15.29:0/8289713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47bc105b10 0x7f47bc199890 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f47ac006ba0 tx=0x7f47ac011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:47.957 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.175+0000 7f47a3fff700 1 -- 172.21.15.29:0/8289713 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47ac003b00 con 0x7f47bc105b10 2024-04-27T01:57:47.957 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.175+0000 7f47a3fff700 1 -- 172.21.15.29:0/8289713 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f47ac01a8e0 con 0x7f47bc105b10 2024-04-27T01:57:47.957 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.175+0000 7f47c1030700 1 -- 172.21.15.29:0/8289713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47bc199fd0 con 0x7f47bc105b10 2024-04-27T01:57:47.958 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.175+0000 7f47a3fff700 1 -- 172.21.15.29:0/8289713 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47ac0199c0 con 0x7f47bc105b10 2024-04-27T01:57:47.958 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.175+0000 7f47c1030700 1 -- 172.21.15.29:0/8289713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47bc19a3f0 con 0x7f47bc105b10 2024-04-27T01:57:47.958 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.176+0000 7f47a3fff700 1 -- 172.21.15.29:0/8289713 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f47ac019c80 con 0x7f47bc105b10 2024-04-27T01:57:47.958 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.176+0000 7f47a3fff700 1 --2- 172.21.15.29:0/8289713 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f47a40383a0 0x7f47a403a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:47.958 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.176+0000 7f47a3fff700 1 -- 172.21.15.29:0/8289713 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f47ac04c610 con 0x7f47bc105b10 2024-04-27T01:57:47.958 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.176+0000 7f47ba59c700 1 --2- 172.21.15.29:0/8289713 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f47a40383a0 0x7f47a403a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:47.958 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.176+0000 7f47c1030700 1 -- 172.21.15.29:0/8289713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47bc062380 con 0x7f47bc105b10 2024-04-27T01:57:47.958 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.177+0000 7f47ba59c700 1 --2- 172.21.15.29:0/8289713 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f47a40383a0 0x7f47a403a850 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f47b0006fd0 tx=0x7f47b0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:47.958 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.181+0000 7f47a3fff700 1 -- 172.21.15.29:0/8289713 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f47ac02d440 con 0x7f47bc105b10 2024-04-27T01:57:47.958 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.384+0000 7f47c1030700 1 -- 172.21.15.29:0/8289713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f47bc19d420 con 0x7f47bc105b10 2024-04-27T01:57:47.958 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.387+0000 7f47a3fff700 1 -- 172.21.15.29:0/8289713 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f47ac02b340 con 0x7f47bc105b10 2024-04-27T01:57:47.958 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.389+0000 7f47c1030700 1 -- 172.21.15.29:0/8289713 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f47a40383a0 msgr2=0x7f47a403a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:47.959 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.389+0000 7f47c1030700 1 --2- 172.21.15.29:0/8289713 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f47a40383a0 0x7f47a403a850 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f47b0006fd0 tx=0x7f47b0006e40 comp rx=0 tx=0).stop 2024-04-27T01:57:47.959 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.389+0000 7f47c1030700 1 -- 172.21.15.29:0/8289713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47bc105b10 msgr2=0x7f47bc199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:47.959 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.389+0000 7f47c1030700 1 --2- 172.21.15.29:0/8289713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47bc105b10 0x7f47bc199890 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f47ac006ba0 tx=0x7f47ac011b50 comp rx=0 tx=0).stop 2024-04-27T01:57:47.959 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.389+0000 7f47c1030700 1 -- 172.21.15.29:0/8289713 shutdown_connections 2024-04-27T01:57:47.959 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.389+0000 7f47c1030700 1 --2- 172.21.15.29:0/8289713 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f47a40383a0 0x7f47a403a850 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:47.959 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.389+0000 7f47c1030700 1 --2- 172.21.15.29:0/8289713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47bc105b10 0x7f47bc199890 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:47.959 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.389+0000 7f47c1030700 1 -- 172.21.15.29:0/8289713 >> 172.21.15.29:0/8289713 conn(0x7f47bc101180 msgr2=0x7f47bc101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:47.959 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.389+0000 7f47c1030700 1 -- 172.21.15.29:0/8289713 shutdown_connections 2024-04-27T01:57:47.959 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:47.389+0000 7f47c1030700 1 -- 172.21.15.29:0/8289713 wait complete. 2024-04-27T01:57:49.083 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:48 smithi029 ceph-mon[26769]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:57:49.083 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:48 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/3010021419' entity='client.admin' 2024-04-27T01:57:49.325 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.492+0000 7fe233a2f700 1 Processor -- start 2024-04-27T01:57:49.326 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.492+0000 7fe233a2f700 1 -- start start 2024-04-27T01:57:49.326 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.492+0000 7fe233a2f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe22c07be30 0x7fe22c07a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:49.326 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.492+0000 7fe233a2f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c07a870 con 0x7fe22c07be30 2024-04-27T01:57:49.326 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.493+0000 7fe2317cb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe22c07be30 0x7fe22c07a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:49.326 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.493+0000 7fe2317cb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe22c07be30 0x7fe22c07a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36790/0 (socket says 172.21.15.29:36790) 2024-04-27T01:57:49.326 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.493+0000 7fe2317cb700 1 -- 172.21.15.29:0/250945185 learned_addr learned my addr 172.21.15.29:0/250945185 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:49.326 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.493+0000 7fe2317cb700 1 -- 172.21.15.29:0/250945185 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22c07a9b0 con 0x7fe22c07be30 2024-04-27T01:57:49.326 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.493+0000 7fe2317cb700 1 --2- 172.21.15.29:0/250945185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe22c07be30 0x7fe22c07a330 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe2280098a0 tx=0x7fe228009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad3a9c8b110f200f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:49.326 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.493+0000 7fe223fff700 1 -- 172.21.15.29:0/250945185 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe228004180 con 0x7fe22c07be30 2024-04-27T01:57:49.326 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.493+0000 7fe223fff700 1 -- 172.21.15.29:0/250945185 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe2280042e0 con 0x7fe22c07be30 2024-04-27T01:57:49.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.493+0000 7fe223fff700 1 -- 172.21.15.29:0/250945185 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe228011af0 con 0x7fe22c07be30 2024-04-27T01:57:49.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.494+0000 7fe233a2f700 1 -- 172.21.15.29:0/250945185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe22c07be30 msgr2=0x7fe22c07a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:49.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.494+0000 7fe233a2f700 1 --2- 172.21.15.29:0/250945185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe22c07be30 0x7fe22c07a330 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe2280098a0 tx=0x7fe228009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:49.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.494+0000 7fe233a2f700 1 -- 172.21.15.29:0/250945185 shutdown_connections 2024-04-27T01:57:49.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.494+0000 7fe233a2f700 1 --2- 172.21.15.29:0/250945185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe22c07be30 0x7fe22c07a330 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:49.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.494+0000 7fe233a2f700 1 -- 172.21.15.29:0/250945185 >> 172.21.15.29:0/250945185 conn(0x7fe22c101180 msgr2=0x7fe22c103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:49.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.494+0000 7fe233a2f700 1 -- 172.21.15.29:0/250945185 shutdown_connections 2024-04-27T01:57:49.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.494+0000 7fe233a2f700 1 -- 172.21.15.29:0/250945185 wait complete. 2024-04-27T01:57:49.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.495+0000 7fe233a2f700 1 Processor -- start 2024-04-27T01:57:49.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.495+0000 7fe233a2f700 1 -- start start 2024-04-27T01:57:49.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.495+0000 7fe233a2f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe22c07be30 0x7fe22c19bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:49.327 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.495+0000 7fe233a2f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c19c020 con 0x7fe22c07be30 2024-04-27T01:57:49.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.495+0000 7fe2317cb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe22c07be30 0x7fe22c19bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:49.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.495+0000 7fe2317cb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe22c07be30 0x7fe22c19bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36806/0 (socket says 172.21.15.29:36806) 2024-04-27T01:57:49.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.495+0000 7fe2317cb700 1 -- 172.21.15.29:0/3010021419 learned_addr learned my addr 172.21.15.29:0/3010021419 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:49.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.495+0000 7fe2317cb700 1 -- 172.21.15.29:0/3010021419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe228009550 con 0x7fe22c07be30 2024-04-27T01:57:49.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.495+0000 7fe2317cb700 1 --2- 172.21.15.29:0/3010021419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe22c07be30 0x7fe22c19bae0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe228006ba0 tx=0x7fe228011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:49.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.495+0000 7fe2227fc700 1 -- 172.21.15.29:0/3010021419 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe228003b00 con 0x7fe22c07be30 2024-04-27T01:57:49.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.496+0000 7fe2227fc700 1 -- 172.21.15.29:0/3010021419 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe22801a8e0 con 0x7fe22c07be30 2024-04-27T01:57:49.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.496+0000 7fe233a2f700 1 -- 172.21.15.29:0/3010021419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe22c19c220 con 0x7fe22c07be30 2024-04-27T01:57:49.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.496+0000 7fe2227fc700 1 -- 172.21.15.29:0/3010021419 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe228019990 con 0x7fe22c07be30 2024-04-27T01:57:49.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.496+0000 7fe233a2f700 1 -- 172.21.15.29:0/3010021419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe22c19c640 con 0x7fe22c07be30 2024-04-27T01:57:49.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.497+0000 7fe2227fc700 1 -- 172.21.15.29:0/3010021419 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fe228019c00 con 0x7fe22c07be30 2024-04-27T01:57:49.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.497+0000 7fe233a2f700 1 -- 172.21.15.29:0/3010021419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe22c10be00 con 0x7fe22c07be30 2024-04-27T01:57:49.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.497+0000 7fe2227fc700 1 --2- 172.21.15.29:0/3010021419 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fe2180383f0 0x7fe21803a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:49.328 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.497+0000 7fe2227fc700 1 -- 172.21.15.29:0/3010021419 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe22804c570 con 0x7fe22c07be30 2024-04-27T01:57:49.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.497+0000 7fe230fca700 1 --2- 172.21.15.29:0/3010021419 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fe2180383f0 0x7fe21803a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:49.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.497+0000 7fe230fca700 1 --2- 172.21.15.29:0/3010021419 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fe2180383f0 0x7fe21803a8a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe21c006fd0 tx=0x7fe21c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:49.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.502+0000 7fe2227fc700 1 -- 172.21.15.29:0/3010021419 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe22801a430 con 0x7fe22c07be30 2024-04-27T01:57:49.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.702+0000 7fe233a2f700 1 -- 172.21.15.29:0/3010021419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fe22c10bff0 con 0x7fe22c07be30 2024-04-27T01:57:49.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.705+0000 7fe2227fc700 1 -- 172.21.15.29:0/3010021419 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fe228018b40 con 0x7fe22c07be30 2024-04-27T01:57:49.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.707+0000 7fe233a2f700 1 -- 172.21.15.29:0/3010021419 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fe2180383f0 msgr2=0x7fe21803a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:49.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.707+0000 7fe233a2f700 1 --2- 172.21.15.29:0/3010021419 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fe2180383f0 0x7fe21803a8a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe21c006fd0 tx=0x7fe21c006e40 comp rx=0 tx=0).stop 2024-04-27T01:57:49.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.707+0000 7fe233a2f700 1 -- 172.21.15.29:0/3010021419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe22c07be30 msgr2=0x7fe22c19bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:49.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.707+0000 7fe233a2f700 1 --2- 172.21.15.29:0/3010021419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe22c07be30 0x7fe22c19bae0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe228006ba0 tx=0x7fe228011b50 comp rx=0 tx=0).stop 2024-04-27T01:57:49.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.707+0000 7fe233a2f700 1 -- 172.21.15.29:0/3010021419 shutdown_connections 2024-04-27T01:57:49.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.707+0000 7fe233a2f700 1 --2- 172.21.15.29:0/3010021419 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7fe2180383f0 0x7fe21803a8a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:49.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.707+0000 7fe233a2f700 1 --2- 172.21.15.29:0/3010021419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe22c07be30 0x7fe22c19bae0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:49.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.707+0000 7fe233a2f700 1 -- 172.21.15.29:0/3010021419 >> 172.21.15.29:0/3010021419 conn(0x7fe22c101180 msgr2=0x7fe22c102880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:49.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.707+0000 7fe233a2f700 1 -- 172.21.15.29:0/3010021419 shutdown_connections 2024-04-27T01:57:49.330 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:48.707+0000 7fe233a2f700 1 -- 172.21.15.29:0/3010021419 wait complete. 2024-04-27T01:57:49.330 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling the dashboard module... 2024-04-27T01:57:51.130 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:50 smithi029 ceph-mon[26769]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:57:51.130 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:50 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/1769910878' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-27T01:57:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.771+0000 7f70dfc5e700 1 Processor -- start 2024-04-27T01:57:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.772+0000 7f70dfc5e700 1 -- start start 2024-04-27T01:57:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.772+0000 7f70dfc5e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8105b10 0x7f70d8105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.772+0000 7f70dfc5e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d8106460 con 0x7f70d8105b10 2024-04-27T01:57:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.772+0000 7f70dd9fa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8105b10 0x7f70d8105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.772+0000 7f70dd9fa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8105b10 0x7f70d8105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36818/0 (socket says 172.21.15.29:36818) 2024-04-27T01:57:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.772+0000 7f70dd9fa700 1 -- 172.21.15.29:0/3652080187 learned_addr learned my addr 172.21.15.29:0/3652080187 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.772+0000 7f70dd9fa700 1 -- 172.21.15.29:0/3652080187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d81065a0 con 0x7f70d8105b10 2024-04-27T01:57:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.772+0000 7f70dd9fa700 1 --2- 172.21.15.29:0/3652080187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8105b10 0x7f70d8105f20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f70d40098a0 tx=0x7f70d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=181b599f76c4e0fc server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.773+0000 7f70dc9f8700 1 -- 172.21.15.29:0/3652080187 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70d4004180 con 0x7f70d8105b10 2024-04-27T01:57:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.773+0000 7f70dc9f8700 1 -- 172.21.15.29:0/3652080187 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f70d40042e0 con 0x7f70d8105b10 2024-04-27T01:57:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.773+0000 7f70dc9f8700 1 -- 172.21.15.29:0/3652080187 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70d4011af0 con 0x7f70d8105b10 2024-04-27T01:57:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.773+0000 7f70dfc5e700 1 -- 172.21.15.29:0/3652080187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8105b10 msgr2=0x7f70d8105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.773+0000 7f70dfc5e700 1 --2- 172.21.15.29:0/3652080187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8105b10 0x7f70d8105f20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f70d40098a0 tx=0x7f70d4009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.773+0000 7f70dfc5e700 1 -- 172.21.15.29:0/3652080187 shutdown_connections 2024-04-27T01:57:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.773+0000 7f70dfc5e700 1 --2- 172.21.15.29:0/3652080187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8105b10 0x7f70d8105f20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.773+0000 7f70dfc5e700 1 -- 172.21.15.29:0/3652080187 >> 172.21.15.29:0/3652080187 conn(0x7f70d8101180 msgr2=0x7f70d8103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.773+0000 7f70dfc5e700 1 -- 172.21.15.29:0/3652080187 shutdown_connections 2024-04-27T01:57:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.773+0000 7f70dfc5e700 1 -- 172.21.15.29:0/3652080187 wait complete. 2024-04-27T01:57:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.774+0000 7f70dfc5e700 1 Processor -- start 2024-04-27T01:57:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.774+0000 7f70dfc5e700 1 -- start start 2024-04-27T01:57:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.774+0000 7f70dfc5e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8105b10 0x7f70d8199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.774+0000 7f70dfc5e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d8199dd0 con 0x7f70d8105b10 2024-04-27T01:57:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.774+0000 7f70dd9fa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8105b10 0x7f70d8199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.774+0000 7f70dd9fa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8105b10 0x7f70d8199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36828/0 (socket says 172.21.15.29:36828) 2024-04-27T01:57:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.774+0000 7f70dd9fa700 1 -- 172.21.15.29:0/1769910878 learned_addr learned my addr 172.21.15.29:0/1769910878 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.775+0000 7f70dd9fa700 1 -- 172.21.15.29:0/1769910878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d4009550 con 0x7f70d8105b10 2024-04-27T01:57:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.775+0000 7f70dd9fa700 1 --2- 172.21.15.29:0/1769910878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8105b10 0x7f70d8199890 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f70d4011ad0 tx=0x7f70d4011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.775+0000 7f70caffd700 1 -- 172.21.15.29:0/1769910878 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70d4003ab0 con 0x7f70d8105b10 2024-04-27T01:57:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.775+0000 7f70caffd700 1 -- 172.21.15.29:0/1769910878 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f70d4003c10 con 0x7f70d8105b10 2024-04-27T01:57:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.775+0000 7f70dfc5e700 1 -- 172.21.15.29:0/1769910878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70d8199fd0 con 0x7f70d8105b10 2024-04-27T01:57:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.775+0000 7f70caffd700 1 -- 172.21.15.29:0/1769910878 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70d40198e0 con 0x7f70d8105b10 2024-04-27T01:57:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.775+0000 7f70dfc5e700 1 -- 172.21.15.29:0/1769910878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70d819a3f0 con 0x7f70d8105b10 2024-04-27T01:57:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.776+0000 7f70caffd700 1 -- 172.21.15.29:0/1769910878 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f70d401a430 con 0x7f70d8105b10 2024-04-27T01:57:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.776+0000 7f70dfc5e700 1 -- 172.21.15.29:0/1769910878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70d8062380 con 0x7f70d8105b10 2024-04-27T01:57:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.776+0000 7f70caffd700 1 --2- 172.21.15.29:0/1769910878 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f70c4038440 0x7f70c403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.776+0000 7f70caffd700 1 -- 172.21.15.29:0/1769910878 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f70d404c140 con 0x7f70d8105b10 2024-04-27T01:57:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.776+0000 7f70dd1f9700 1 --2- 172.21.15.29:0/1769910878 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f70c4038440 0x7f70c403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.777+0000 7f70dd1f9700 1 --2- 172.21.15.29:0/1769910878 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f70c4038440 0x7f70c403a8f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f70cc006fd0 tx=0x7f70cc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:51.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:49.781+0000 7f70caffd700 1 -- 172.21.15.29:0/1769910878 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f70d4018b00 con 0x7f70d8105b10 2024-04-27T01:57:51.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:50.039+0000 7f70dfc5e700 1 -- 172.21.15.29:0/1769910878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f70d819d590 con 0x7f70d8105b10 2024-04-27T01:57:51.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:50.785+0000 7f70caffd700 1 -- 172.21.15.29:0/1769910878 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 9) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f70d4019b00 con 0x7f70d8105b10 2024-04-27T01:57:51.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:50.787+0000 7f70caffd700 1 -- 172.21.15.29:0/1769910878 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f70d401ad10 con 0x7f70d8105b10 2024-04-27T01:57:51.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:50.791+0000 7f70dfc5e700 1 -- 172.21.15.29:0/1769910878 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f70c4038440 msgr2=0x7f70c403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:51.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:50.791+0000 7f70dfc5e700 1 --2- 172.21.15.29:0/1769910878 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f70c4038440 0x7f70c403a8f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f70cc006fd0 tx=0x7f70cc006e40 comp rx=0 tx=0).stop 2024-04-27T01:57:51.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:50.791+0000 7f70dfc5e700 1 -- 172.21.15.29:0/1769910878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8105b10 msgr2=0x7f70d8199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:51.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:50.791+0000 7f70dfc5e700 1 --2- 172.21.15.29:0/1769910878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8105b10 0x7f70d8199890 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f70d4011ad0 tx=0x7f70d4011bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:51.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:50.791+0000 7f70dfc5e700 1 -- 172.21.15.29:0/1769910878 shutdown_connections 2024-04-27T01:57:51.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:50.791+0000 7f70dfc5e700 1 --2- 172.21.15.29:0/1769910878 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f70c4038440 0x7f70c403a8f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:51.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:50.791+0000 7f70dfc5e700 1 --2- 172.21.15.29:0/1769910878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70d8105b10 0x7f70d8199890 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:51.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:50.791+0000 7f70dfc5e700 1 -- 172.21.15.29:0/1769910878 >> 172.21.15.29:0/1769910878 conn(0x7f70d8101180 msgr2=0x7f70d8101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:51.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:50.791+0000 7f70dfc5e700 1 -- 172.21.15.29:0/1769910878 shutdown_connections 2024-04-27T01:57:51.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:50.791+0000 7f70dfc5e700 1 -- 172.21.15.29:0/1769910878 wait complete. 2024-04-27T01:57:52.097 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:51 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/1769910878' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-27T01:57:52.098 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:51 smithi029 ceph-mon[26769]: mgrmap e9: smithi029.ewvdzs(active, since 27s) 2024-04-27T01:57:52.804 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:52 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/516589786' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-27T01:57:52.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-04-27T01:57:52.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-27T01:57:52.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-27T01:57:52.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "active_name": "smithi029.ewvdzs", 2024-04-27T01:57:52.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-27T01:57:52.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-04-27T01:57:52.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.910+0000 7f23f30a1700 1 Processor -- start 2024-04-27T01:57:52.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.910+0000 7f23f30a1700 1 -- start start 2024-04-27T01:57:52.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.910+0000 7f23f30a1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec1049f0 0x7f23ec104e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:52.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.910+0000 7f23f30a1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec105340 con 0x7f23ec1049f0 2024-04-27T01:57:52.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.911+0000 7f23f0e3d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec1049f0 0x7f23ec104e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:52.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.911+0000 7f23f0e3d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec1049f0 0x7f23ec104e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36852/0 (socket says 172.21.15.29:36852) 2024-04-27T01:57:52.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.911+0000 7f23f0e3d700 1 -- 172.21.15.29:0/3595743797 learned_addr learned my addr 172.21.15.29:0/3595743797 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:52.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.911+0000 7f23f0e3d700 1 -- 172.21.15.29:0/3595743797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ec105480 con 0x7f23ec1049f0 2024-04-27T01:57:52.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.911+0000 7f23f0e3d700 1 --2- 172.21.15.29:0/3595743797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec1049f0 0x7f23ec104e00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f23e00098a0 tx=0x7f23e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65cb0f67d5961b13 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:52.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.911+0000 7f23eb7fe700 1 -- 172.21.15.29:0/3595743797 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23e0004180 con 0x7f23ec1049f0 2024-04-27T01:57:52.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.911+0000 7f23eb7fe700 1 -- 172.21.15.29:0/3595743797 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f23e00042e0 con 0x7f23ec1049f0 2024-04-27T01:57:52.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.912+0000 7f23eb7fe700 1 -- 172.21.15.29:0/3595743797 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23e0011af0 con 0x7f23ec1049f0 2024-04-27T01:57:52.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.912+0000 7f23f30a1700 1 -- 172.21.15.29:0/3595743797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec1049f0 msgr2=0x7f23ec104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:52.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.912+0000 7f23f30a1700 1 --2- 172.21.15.29:0/3595743797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec1049f0 0x7f23ec104e00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f23e00098a0 tx=0x7f23e0009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:52.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.912+0000 7f23f30a1700 1 -- 172.21.15.29:0/3595743797 shutdown_connections 2024-04-27T01:57:52.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.912+0000 7f23f30a1700 1 --2- 172.21.15.29:0/3595743797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec1049f0 0x7f23ec104e00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:52.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.912+0000 7f23f30a1700 1 -- 172.21.15.29:0/3595743797 >> 172.21.15.29:0/3595743797 conn(0x7f23ec100080 msgr2=0x7f23ec102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:52.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.912+0000 7f23f30a1700 1 -- 172.21.15.29:0/3595743797 shutdown_connections 2024-04-27T01:57:52.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.912+0000 7f23f30a1700 1 -- 172.21.15.29:0/3595743797 wait complete. 2024-04-27T01:57:52.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.913+0000 7f23f30a1700 1 Processor -- start 2024-04-27T01:57:52.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.913+0000 7f23f30a1700 1 -- start start 2024-04-27T01:57:52.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.913+0000 7f23f30a1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec1049f0 0x7f23ec18fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:52.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.913+0000 7f23f30a1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec190110 con 0x7f23ec1049f0 2024-04-27T01:57:52.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.913+0000 7f23f0e3d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec1049f0 0x7f23ec18fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:52.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.913+0000 7f23f0e3d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec1049f0 0x7f23ec18fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36856/0 (socket says 172.21.15.29:36856) 2024-04-27T01:57:52.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.913+0000 7f23f0e3d700 1 -- 172.21.15.29:0/516589786 learned_addr learned my addr 172.21.15.29:0/516589786 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:52.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.914+0000 7f23f0e3d700 1 -- 172.21.15.29:0/516589786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e0009550 con 0x7f23ec1049f0 2024-04-27T01:57:52.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.914+0000 7f23f0e3d700 1 --2- 172.21.15.29:0/516589786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec1049f0 0x7f23ec18fbd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f23e0006ba0 tx=0x7f23e0011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:52.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.914+0000 7f23e9ffb700 1 -- 172.21.15.29:0/516589786 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23e0003b00 con 0x7f23ec1049f0 2024-04-27T01:57:52.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.914+0000 7f23e9ffb700 1 -- 172.21.15.29:0/516589786 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f23e001a8e0 con 0x7f23ec1049f0 2024-04-27T01:57:52.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.914+0000 7f23f30a1700 1 -- 172.21.15.29:0/516589786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23ec18e220 con 0x7f23ec1049f0 2024-04-27T01:57:52.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.914+0000 7f23e9ffb700 1 -- 172.21.15.29:0/516589786 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23e00199c0 con 0x7f23ec1049f0 2024-04-27T01:57:52.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.914+0000 7f23f30a1700 1 -- 172.21.15.29:0/516589786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23ec18e640 con 0x7f23ec1049f0 2024-04-27T01:57:52.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.915+0000 7f23e9ffb700 1 -- 172.21.15.29:0/516589786 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f23e0019c30 con 0x7f23ec1049f0 2024-04-27T01:57:52.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.915+0000 7f23e9ffb700 1 --2- 172.21.15.29:0/516589786 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f23dc038440 0x7f23dc03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:52.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.915+0000 7f23e9ffb700 1 -- 172.21.15.29:0/516589786 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f23e004c600 con 0x7f23ec1049f0 2024-04-27T01:57:52.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.915+0000 7f23ebfff700 1 -- 172.21.15.29:0/516589786 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f23dc038440 msgr2=0x7f23dc03a8f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3873642219 2024-04-27T01:57:52.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.915+0000 7f23ebfff700 1 --2- 172.21.15.29:0/516589786 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f23dc038440 0x7f23dc03a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-27T01:57:52.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.916+0000 7f23f30a1700 1 -- 172.21.15.29:0/516589786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23ec062380 con 0x7f23ec1049f0 2024-04-27T01:57:52.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:51.920+0000 7f23e9ffb700 1 -- 172.21.15.29:0/516589786 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f23e0048080 con 0x7f23ec1049f0 2024-04-27T01:57:52.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:52.116+0000 7f23ebfff700 1 -- 172.21.15.29:0/516589786 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f23dc038440 msgr2=0x7f23dc03a8f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3873642219 2024-04-27T01:57:52.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:52.116+0000 7f23ebfff700 1 --2- 172.21.15.29:0/516589786 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f23dc038440 0x7f23dc03a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-27T01:57:52.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:52.248+0000 7f23f30a1700 1 -- 172.21.15.29:0/516589786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f23ec1068a0 con 0x7f23ec1049f0 2024-04-27T01:57:52.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:52.248+0000 7f23e9ffb700 1 -- 172.21.15.29:0/516589786 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f23e002d440 con 0x7f23ec1049f0 2024-04-27T01:57:52.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:52.252+0000 7f23f30a1700 1 -- 172.21.15.29:0/516589786 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f23dc038440 msgr2=0x7f23dc03a8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-27T01:57:52.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:52.252+0000 7f23f30a1700 1 --2- 172.21.15.29:0/516589786 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f23dc038440 0x7f23dc03a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:52.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:52.252+0000 7f23f30a1700 1 -- 172.21.15.29:0/516589786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec1049f0 msgr2=0x7f23ec18fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:52.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:52.252+0000 7f23f30a1700 1 --2- 172.21.15.29:0/516589786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec1049f0 0x7f23ec18fbd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f23e0006ba0 tx=0x7f23e0011b50 comp rx=0 tx=0).stop 2024-04-27T01:57:52.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:52.252+0000 7f23f30a1700 1 -- 172.21.15.29:0/516589786 shutdown_connections 2024-04-27T01:57:52.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:52.252+0000 7f23f30a1700 1 --2- 172.21.15.29:0/516589786 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f23dc038440 0x7f23dc03a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:52.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:52.252+0000 7f23f30a1700 1 --2- 172.21.15.29:0/516589786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec1049f0 0x7f23ec18fbd0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:52.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:52.252+0000 7f23f30a1700 1 -- 172.21.15.29:0/516589786 >> 172.21.15.29:0/516589786 conn(0x7f23ec100080 msgr2=0x7f23ec079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:52.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:52.252+0000 7f23f30a1700 1 -- 172.21.15.29:0/516589786 shutdown_connections 2024-04-27T01:57:52.819 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:52.252+0000 7f23f30a1700 1 -- 172.21.15.29:0/516589786 wait complete. 2024-04-27T01:57:52.819 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for the mgr to restart... 2024-04-27T01:57:52.819 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr epoch 9... 2024-04-27T01:57:58.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:57 smithi029 ceph-mon[26769]: Active manager daemon smithi029.ewvdzs restarted 2024-04-27T01:57:58.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:57 smithi029 ceph-mon[26769]: Activating manager daemon smithi029.ewvdzs 2024-04-27T01:57:58.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:57 smithi029 ceph-mon[26769]: osdmap e3: 0 total, 0 up, 0 in 2024-04-27T01:57:58.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:57 smithi029 ceph-mon[26769]: mgrmap e10: smithi029.ewvdzs(active, starting, since 0.00346885s) 2024-04-27T01:57:58.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:57 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-04-27T01:57:58.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:57 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mgr metadata", "who": "smithi029.ewvdzs", "id": "smithi029.ewvdzs"}]: dispatch 2024-04-27T01:57:58.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:57 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T01:57:58.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:57 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T01:57:58.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:57 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T01:57:58.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:57 smithi029 ceph-mon[26769]: Manager daemon smithi029.ewvdzs is now available 2024-04-27T01:57:58.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:57 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:57:58.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:57 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:57:59.021 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:58 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.ewvdzs/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:57:59.021 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:58 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.ewvdzs/trash_purge_schedule"}]: dispatch 2024-04-27T01:57:59.021 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:58 smithi029 ceph-mon[26769]: mgrmap e11: smithi029.ewvdzs(active, since 1.00595s) 2024-04-27T01:57:59.646 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-04-27T01:57:59.646 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-27T01:57:59.646 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-27T01:57:59.646 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-04-27T01:57:59.646 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.295+0000 7f60b7e19700 1 Processor -- start 2024-04-27T01:57:59.646 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.295+0000 7f60b7e19700 1 -- start start 2024-04-27T01:57:59.647 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.296+0000 7f60b7e19700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0106640 0x7f60b0108a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:59.647 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.296+0000 7f60b7e19700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b0079d70 con 0x7f60b0106640 2024-04-27T01:57:59.647 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.296+0000 7f60b5bb5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0106640 0x7f60b0108a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:59.647 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.296+0000 7f60b5bb5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0106640 0x7f60b0108a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36868/0 (socket says 172.21.15.29:36868) 2024-04-27T01:57:59.647 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.296+0000 7f60b5bb5700 1 -- 172.21.15.29:0/3048160604 learned_addr learned my addr 172.21.15.29:0/3048160604 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:59.647 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.296+0000 7f60b5bb5700 1 -- 172.21.15.29:0/3048160604 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b0079eb0 con 0x7f60b0106640 2024-04-27T01:57:59.647 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.296+0000 7f60b5bb5700 1 --2- 172.21.15.29:0/3048160604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0106640 0x7f60b0108a20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f60ac0098a0 tx=0x7f60ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c27dade4885e0d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:59.647 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.296+0000 7f60b4bb3700 1 -- 172.21.15.29:0/3048160604 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60ac004180 con 0x7f60b0106640 2024-04-27T01:57:59.648 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.297+0000 7f60b4bb3700 1 -- 172.21.15.29:0/3048160604 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f60ac0042e0 con 0x7f60b0106640 2024-04-27T01:57:59.648 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.297+0000 7f60b4bb3700 1 -- 172.21.15.29:0/3048160604 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60ac011af0 con 0x7f60b0106640 2024-04-27T01:57:59.648 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.297+0000 7f60b7e19700 1 -- 172.21.15.29:0/3048160604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0106640 msgr2=0x7f60b0108a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:59.648 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.297+0000 7f60b7e19700 1 --2- 172.21.15.29:0/3048160604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0106640 0x7f60b0108a20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f60ac0098a0 tx=0x7f60ac009bb0 comp rx=0 tx=0).stop 2024-04-27T01:57:59.648 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.298+0000 7f60b7e19700 1 -- 172.21.15.29:0/3048160604 shutdown_connections 2024-04-27T01:57:59.648 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.298+0000 7f60b7e19700 1 --2- 172.21.15.29:0/3048160604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0106640 0x7f60b0108a20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:59.648 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.298+0000 7f60b7e19700 1 -- 172.21.15.29:0/3048160604 >> 172.21.15.29:0/3048160604 conn(0x7f60b0100060 msgr2=0x7f60b01024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:59.648 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.298+0000 7f60b7e19700 1 -- 172.21.15.29:0/3048160604 shutdown_connections 2024-04-27T01:57:59.648 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.298+0000 7f60b7e19700 1 -- 172.21.15.29:0/3048160604 wait complete. 2024-04-27T01:57:59.648 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.298+0000 7f60b7e19700 1 Processor -- start 2024-04-27T01:57:59.648 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.298+0000 7f60b7e19700 1 -- start start 2024-04-27T01:57:59.648 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.298+0000 7f60b7e19700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0106640 0x7f60b0199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:59.648 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.298+0000 7f60b7e19700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b0199e50 con 0x7f60b0106640 2024-04-27T01:57:59.648 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.299+0000 7f60b5bb5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0106640 0x7f60b0199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:59.649 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.299+0000 7f60b5bb5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0106640 0x7f60b0199910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36878/0 (socket says 172.21.15.29:36878) 2024-04-27T01:57:59.649 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.299+0000 7f60b5bb5700 1 -- 172.21.15.29:0/3766675914 learned_addr learned my addr 172.21.15.29:0/3766675914 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:57:59.649 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.299+0000 7f60b5bb5700 1 -- 172.21.15.29:0/3766675914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac009550 con 0x7f60b0106640 2024-04-27T01:57:59.649 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.299+0000 7f60b5bb5700 1 --2- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0106640 0x7f60b0199910 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f60ac011670 tx=0x7f60ac011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:59.649 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.299+0000 7f60a6ffd700 1 -- 172.21.15.29:0/3766675914 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60ac0039c0 con 0x7f60b0106640 2024-04-27T01:57:59.649 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.299+0000 7f60a6ffd700 1 -- 172.21.15.29:0/3766675914 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f60ac003b20 con 0x7f60b0106640 2024-04-27T01:57:59.649 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.299+0000 7f60a6ffd700 1 -- 172.21.15.29:0/3766675914 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60ac0247f0 con 0x7f60b0106640 2024-04-27T01:57:59.649 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.299+0000 7f60b7e19700 1 -- 172.21.15.29:0/3766675914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b019a050 con 0x7f60b0106640 2024-04-27T01:57:59.649 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.299+0000 7f60b7e19700 1 -- 172.21.15.29:0/3766675914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b019a470 con 0x7f60b0106640 2024-04-27T01:57:59.649 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.300+0000 7f60a6ffd700 1 -- 172.21.15.29:0/3766675914 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f60ac024a60 con 0x7f60b0106640 2024-04-27T01:57:59.649 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.300+0000 7f60a6ffd700 1 --2- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f609c038450 0x7f609c03a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:59.649 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.300+0000 7f60b7e19700 1 -- 172.21.15.29:0/3766675914 --> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f60b0062380 con 0x7f609c038450 2024-04-27T01:57:59.649 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.300+0000 7f60a6ffd700 1 -- 172.21.15.29:0/3766675914 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f60ac04c5b0 con 0x7f60b0106640 2024-04-27T01:57:59.649 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.300+0000 7f60b53b4700 1 -- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f609c038450 msgr2=0x7f609c03a900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3873642219 2024-04-27T01:57:59.650 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.300+0000 7f60b53b4700 1 --2- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f609c038450 0x7f609c03a900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-27T01:57:59.650 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.501+0000 7f60b53b4700 1 -- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f609c038450 msgr2=0x7f609c03a900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3873642219 2024-04-27T01:57:59.650 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.501+0000 7f60b53b4700 1 --2- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f609c038450 0x7f609c03a900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-27T01:57:59.650 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.902+0000 7f60b53b4700 1 -- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f609c038450 msgr2=0x7f609c03a900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3873642219 2024-04-27T01:57:59.650 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:53.902+0000 7f60b53b4700 1 --2- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f609c038450 0x7f609c03a900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-27T01:57:59.650 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:54.703+0000 7f60b53b4700 1 -- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f609c038450 msgr2=0x7f609c03a900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3873642219 2024-04-27T01:57:59.650 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:54.703+0000 7f60b53b4700 1 --2- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f609c038450 0x7f609c03a900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-27T01:57:59.650 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:56.304+0000 7f60b53b4700 1 -- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f609c038450 msgr2=0x7f609c03a900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3873642219 2024-04-27T01:57:59.650 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:56.304+0000 7f60b53b4700 1 --2- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f609c038450 0x7f609c03a900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-27T01:57:59.650 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:57.789+0000 7f60a6ffd700 1 -- 172.21.15.29:0/3766675914 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mgrmap(e 10) v1 ==== 44864+0+0 (secure 0 0 0) 0x7f60ac01ac70 con 0x7f60b0106640 2024-04-27T01:57:59.650 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:57.789+0000 7f60a6ffd700 1 -- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f609c038450 msgr2=0x7f609c03a900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-27T01:57:59.650 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:57.789+0000 7f60a6ffd700 1 --2- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f609c038450 0x7f609c03a900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:59.650 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:58.791+0000 7f60a6ffd700 1 -- 172.21.15.29:0/3766675914 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 11) v1 ==== 44991+0+0 (secure 0 0 0) 0x7f60ac04de60 con 0x7f60b0106640 2024-04-27T01:57:59.650 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:58.791+0000 7f60a6ffd700 1 --2- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f609c03bd50 0x7f609c03e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:57:59.651 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:58.791+0000 7f60a6ffd700 1 -- 172.21.15.29:0/3766675914 --> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f60b0062380 con 0x7f609c03bd50 2024-04-27T01:57:59.651 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:58.792+0000 7f60b53b4700 1 --2- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f609c03bd50 0x7f609c03e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:57:59.651 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:58.792+0000 7f60b53b4700 1 --2- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f609c03bd50 0x7f609c03e130 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f60a0003a10 tx=0x7f60a0009270 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:57:59.651 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:58.793+0000 7f60a6ffd700 1 -- 172.21.15.29:0/3766675914 <== mgr.14164 v2:172.21.15.29:6800/2469926221 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f60b0062380 con 0x7f609c03bd50 2024-04-27T01:57:59.651 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:58.799+0000 7f60b7e19700 1 -- 172.21.15.29:0/3766675914 --> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f60b019ac50 con 0x7f609c03bd50 2024-04-27T01:57:59.651 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:58.800+0000 7f60a6ffd700 1 -- 172.21.15.29:0/3766675914 <== mgr.14164 v2:172.21.15.29:6800/2469926221 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f60b019ac50 con 0x7f609c03bd50 2024-04-27T01:57:59.651 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:58.800+0000 7f60b7e19700 1 -- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f609c03bd50 msgr2=0x7f609c03e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:59.651 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:58.800+0000 7f60b7e19700 1 --2- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f609c03bd50 0x7f609c03e130 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f60a0003a10 tx=0x7f60a0009270 comp rx=0 tx=0).stop 2024-04-27T01:57:59.651 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:58.800+0000 7f60b7e19700 1 -- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0106640 msgr2=0x7f60b0199910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:57:59.651 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:58.800+0000 7f60b7e19700 1 --2- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0106640 0x7f60b0199910 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f60ac011670 tx=0x7f60ac011aa0 comp rx=0 tx=0).stop 2024-04-27T01:57:59.651 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:58.800+0000 7f60b7e19700 1 -- 172.21.15.29:0/3766675914 shutdown_connections 2024-04-27T01:57:59.651 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:58.800+0000 7f60b7e19700 1 --2- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f609c03bd50 0x7f609c03e130 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:59.651 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:58.800+0000 7f60b7e19700 1 --2- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0106640 0x7f60b0199910 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:59.651 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:58.800+0000 7f60b7e19700 1 --2- 172.21.15.29:0/3766675914 >> [v2:172.21.15.29:6800/3873642219,v1:172.21.15.29:6801/3873642219] conn(0x7f609c038450 0x7f609c03a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:57:59.652 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:58.800+0000 7f60b7e19700 1 -- 172.21.15.29:0/3766675914 >> 172.21.15.29:0/3766675914 conn(0x7f60b0100060 msgr2=0x7f60b01017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:57:59.652 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:58.800+0000 7f60b7e19700 1 -- 172.21.15.29:0/3766675914 shutdown_connections 2024-04-27T01:57:59.652 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:57:58.800+0000 7f60b7e19700 1 -- 172.21.15.29:0/3766675914 wait complete. 2024-04-27T01:57:59.652 INFO:teuthology.orchestra.run.smithi029.stdout:mgr epoch 9 is available 2024-04-27T01:57:59.652 INFO:teuthology.orchestra.run.smithi029.stdout:Generating a dashboard self-signed certificate... 2024-04-27T01:57:59.897 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:59 smithi029 ceph-mon[26769]: [27/Apr/2024:01:57:58] ENGINE Bus STARTING 2024-04-27T01:57:59.897 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:59 smithi029 ceph-mon[26769]: [27/Apr/2024:01:57:58] ENGINE Serving on http://172.21.15.29:8765 2024-04-27T01:57:59.897 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:59 smithi029 ceph-mon[26769]: [27/Apr/2024:01:57:58] ENGINE Serving on https://172.21.15.29:7150 2024-04-27T01:57:59.898 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:59 smithi029 ceph-mon[26769]: [27/Apr/2024:01:57:58] ENGINE Bus STARTED 2024-04-27T01:57:59.898 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:59 smithi029 ceph-mon[26769]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-27T01:57:59.898 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:59 smithi029 ceph-mon[26769]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-27T01:57:59.898 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:57:59 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:01.051 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-27T01:58:01.051 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.149+0000 7f28c28df700 1 Processor -- start 2024-04-27T01:58:01.052 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.149+0000 7f28c28df700 1 -- start start 2024-04-27T01:58:01.052 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.149+0000 7f28c28df700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28bc1049f0 0x7f28bc104e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:01.052 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.149+0000 7f28c28df700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc105340 con 0x7f28bc1049f0 2024-04-27T01:58:01.052 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.150+0000 7f28bbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28bc1049f0 0x7f28bc104e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:01.052 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.150+0000 7f28bbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28bc1049f0 0x7f28bc104e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47736/0 (socket says 172.21.15.29:47736) 2024-04-27T01:58:01.052 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.150+0000 7f28bbfff700 1 -- 172.21.15.29:0/1716356854 learned_addr learned my addr 172.21.15.29:0/1716356854 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:58:01.052 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.150+0000 7f28bbfff700 1 -- 172.21.15.29:0/1716356854 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28bc105480 con 0x7f28bc1049f0 2024-04-27T01:58:01.052 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.150+0000 7f28bbfff700 1 --2- 172.21.15.29:0/1716356854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28bc1049f0 0x7f28bc104e00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f28ac0098a0 tx=0x7f28ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0c513f93233979c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:01.052 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.150+0000 7f28baffd700 1 -- 172.21.15.29:0/1716356854 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28ac004180 con 0x7f28bc1049f0 2024-04-27T01:58:01.052 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.150+0000 7f28baffd700 1 -- 172.21.15.29:0/1716356854 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f28ac0042e0 con 0x7f28bc1049f0 2024-04-27T01:58:01.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.150+0000 7f28baffd700 1 -- 172.21.15.29:0/1716356854 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28ac011af0 con 0x7f28bc1049f0 2024-04-27T01:58:01.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.151+0000 7f28c28df700 1 -- 172.21.15.29:0/1716356854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28bc1049f0 msgr2=0x7f28bc104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:01.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.151+0000 7f28c28df700 1 --2- 172.21.15.29:0/1716356854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28bc1049f0 0x7f28bc104e00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f28ac0098a0 tx=0x7f28ac009bb0 comp rx=0 tx=0).stop 2024-04-27T01:58:01.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.151+0000 7f28c28df700 1 -- 172.21.15.29:0/1716356854 shutdown_connections 2024-04-27T01:58:01.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.151+0000 7f28c28df700 1 --2- 172.21.15.29:0/1716356854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28bc1049f0 0x7f28bc104e00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:01.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.151+0000 7f28c28df700 1 -- 172.21.15.29:0/1716356854 >> 172.21.15.29:0/1716356854 conn(0x7f28bc100080 msgr2=0x7f28bc102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:01.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.151+0000 7f28c28df700 1 -- 172.21.15.29:0/1716356854 shutdown_connections 2024-04-27T01:58:01.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.151+0000 7f28c28df700 1 -- 172.21.15.29:0/1716356854 wait complete. 2024-04-27T01:58:01.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.152+0000 7f28c28df700 1 Processor -- start 2024-04-27T01:58:01.053 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.152+0000 7f28c28df700 1 -- start start 2024-04-27T01:58:01.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.153+0000 7f28c28df700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28bc1049f0 0x7f28bc1998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:01.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.153+0000 7f28c28df700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc199de0 con 0x7f28bc1049f0 2024-04-27T01:58:01.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.153+0000 7f28bbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28bc1049f0 0x7f28bc1998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:01.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.153+0000 7f28bbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28bc1049f0 0x7f28bc1998a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47746/0 (socket says 172.21.15.29:47746) 2024-04-27T01:58:01.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.153+0000 7f28bbfff700 1 -- 172.21.15.29:0/2276935130 learned_addr learned my addr 172.21.15.29:0/2276935130 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:58:01.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.153+0000 7f28bbfff700 1 -- 172.21.15.29:0/2276935130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ac009550 con 0x7f28bc1049f0 2024-04-27T01:58:01.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.153+0000 7f28bbfff700 1 --2- 172.21.15.29:0/2276935130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28bc1049f0 0x7f28bc1998a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f28ac006ba0 tx=0x7f28ac011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:01.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.153+0000 7f28b97fa700 1 -- 172.21.15.29:0/2276935130 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28ac003b00 con 0x7f28bc1049f0 2024-04-27T01:58:01.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.153+0000 7f28c28df700 1 -- 172.21.15.29:0/2276935130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28bc199fe0 con 0x7f28bc1049f0 2024-04-27T01:58:01.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.154+0000 7f28b97fa700 1 -- 172.21.15.29:0/2276935130 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f28ac01a8e0 con 0x7f28bc1049f0 2024-04-27T01:58:01.054 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.154+0000 7f28c28df700 1 -- 172.21.15.29:0/2276935130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28bc19a400 con 0x7f28bc1049f0 2024-04-27T01:58:01.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.154+0000 7f28b97fa700 1 -- 172.21.15.29:0/2276935130 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28ac0199c0 con 0x7f28bc1049f0 2024-04-27T01:58:01.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.155+0000 7f28b97fa700 1 -- 172.21.15.29:0/2276935130 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 44991+0+0 (secure 0 0 0) 0x7f28ac019c80 con 0x7f28bc1049f0 2024-04-27T01:58:01.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.155+0000 7f28b97fa700 1 --2- 172.21.15.29:0/2276935130 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f28a40382c0 0x7f28a403a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:01.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.155+0000 7f28b97fa700 1 -- 172.21.15.29:0/2276935130 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f28ac01aa50 con 0x7f28bc1049f0 2024-04-27T01:58:01.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.155+0000 7f28bb7fe700 1 --2- 172.21.15.29:0/2276935130 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f28a40382c0 0x7f28a403a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:01.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.155+0000 7f28c28df700 1 -- 172.21.15.29:0/2276935130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28bc062380 con 0x7f28bc1049f0 2024-04-27T01:58:01.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.155+0000 7f28bb7fe700 1 --2- 172.21.15.29:0/2276935130 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f28a40382c0 0x7f28a403a770 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f28b0006fd0 tx=0x7f28b0006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:01.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.160+0000 7f28b97fa700 1 -- 172.21.15.29:0/2276935130 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f28ac048020 con 0x7f28bc1049f0 2024-04-27T01:58:01.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.369+0000 7f28c28df700 1 -- 172.21.15.29:0/2276935130 --> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f28bc101500 con 0x7f28a40382c0 2024-04-27T01:58:01.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.428+0000 7f28b97fa700 1 -- 172.21.15.29:0/2276935130 <== mgr.14164 v2:172.21.15.29:6800/2469926221 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f28bc101500 con 0x7f28a40382c0 2024-04-27T01:58:01.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.430+0000 7f28c28df700 1 -- 172.21.15.29:0/2276935130 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f28a40382c0 msgr2=0x7f28a403a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:01.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.430+0000 7f28c28df700 1 --2- 172.21.15.29:0/2276935130 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f28a40382c0 0x7f28a403a770 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f28b0006fd0 tx=0x7f28b0006e40 comp rx=0 tx=0).stop 2024-04-27T01:58:01.055 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.430+0000 7f28c28df700 1 -- 172.21.15.29:0/2276935130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28bc1049f0 msgr2=0x7f28bc1998a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:01.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.430+0000 7f28c28df700 1 --2- 172.21.15.29:0/2276935130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28bc1049f0 0x7f28bc1998a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f28ac006ba0 tx=0x7f28ac011b50 comp rx=0 tx=0).stop 2024-04-27T01:58:01.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.430+0000 7f28c28df700 1 -- 172.21.15.29:0/2276935130 shutdown_connections 2024-04-27T01:58:01.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.430+0000 7f28c28df700 1 --2- 172.21.15.29:0/2276935130 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f28a40382c0 0x7f28a403a770 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:01.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.430+0000 7f28c28df700 1 --2- 172.21.15.29:0/2276935130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28bc1049f0 0x7f28bc1998a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:01.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.430+0000 7f28c28df700 1 -- 172.21.15.29:0/2276935130 >> 172.21.15.29:0/2276935130 conn(0x7f28bc100080 msgr2=0x7f28bc100df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:01.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.430+0000 7f28c28df700 1 -- 172.21.15.29:0/2276935130 shutdown_connections 2024-04-27T01:58:01.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:00.430+0000 7f28c28df700 1 -- 172.21.15.29:0/2276935130 wait complete. 2024-04-27T01:58:01.056 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial admin user... 2024-04-27T01:58:01.629 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:01 smithi029 ceph-mon[26769]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:58:01.629 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:01 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:01.629 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:01 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:01.629 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:01 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:01.630 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:01 smithi029 ceph-mon[26769]: mgrmap e12: smithi029.ewvdzs(active, since 2s) 2024-04-27T01:58:02.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:02 smithi029 ceph-mon[26769]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:58:02.702 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:02 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:02.733 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$D8S9AliiqoQTa/39iJl7WOGaayjTAqsMpXYEQUJE7qwrcU3rMhuMu", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714183082, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-27T01:58:02.733 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.517+0000 7f1237f60700 1 Processor -- start 2024-04-27T01:58:02.733 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.517+0000 7f1237f60700 1 -- start start 2024-04-27T01:58:02.733 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.517+0000 7f1237f60700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1230105b10 0x7f1230105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:02.733 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.517+0000 7f1237f60700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1230106460 con 0x7f1230105b10 2024-04-27T01:58:02.733 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.517+0000 7f1235cfc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1230105b10 0x7f1230105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:02.733 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.517+0000 7f1235cfc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1230105b10 0x7f1230105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47748/0 (socket says 172.21.15.29:47748) 2024-04-27T01:58:02.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.517+0000 7f1235cfc700 1 -- 172.21.15.29:0/205351520 learned_addr learned my addr 172.21.15.29:0/205351520 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:58:02.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.518+0000 7f1235cfc700 1 -- 172.21.15.29:0/205351520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12301065a0 con 0x7f1230105b10 2024-04-27T01:58:02.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.518+0000 7f1235cfc700 1 --2- 172.21.15.29:0/205351520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1230105b10 0x7f1230105f20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f122c0098a0 tx=0x7f122c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b50a36860edd1ba0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:02.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.518+0000 7f1234cfa700 1 -- 172.21.15.29:0/205351520 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f122c004180 con 0x7f1230105b10 2024-04-27T01:58:02.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.518+0000 7f1234cfa700 1 -- 172.21.15.29:0/205351520 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f122c0042e0 con 0x7f1230105b10 2024-04-27T01:58:02.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.518+0000 7f1234cfa700 1 -- 172.21.15.29:0/205351520 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f122c011af0 con 0x7f1230105b10 2024-04-27T01:58:02.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.519+0000 7f1237f60700 1 -- 172.21.15.29:0/205351520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1230105b10 msgr2=0x7f1230105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:02.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.519+0000 7f1237f60700 1 --2- 172.21.15.29:0/205351520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1230105b10 0x7f1230105f20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f122c0098a0 tx=0x7f122c009bb0 comp rx=0 tx=0).stop 2024-04-27T01:58:02.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.519+0000 7f1237f60700 1 -- 172.21.15.29:0/205351520 shutdown_connections 2024-04-27T01:58:02.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.519+0000 7f1237f60700 1 --2- 172.21.15.29:0/205351520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1230105b10 0x7f1230105f20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:02.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.519+0000 7f1237f60700 1 -- 172.21.15.29:0/205351520 >> 172.21.15.29:0/205351520 conn(0x7f1230101180 msgr2=0x7f1230103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:02.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.519+0000 7f1237f60700 1 -- 172.21.15.29:0/205351520 shutdown_connections 2024-04-27T01:58:02.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.519+0000 7f1237f60700 1 -- 172.21.15.29:0/205351520 wait complete. 2024-04-27T01:58:02.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.519+0000 7f1237f60700 1 Processor -- start 2024-04-27T01:58:02.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.519+0000 7f1237f60700 1 -- start start 2024-04-27T01:58:02.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.520+0000 7f1237f60700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1230105b10 0x7f1230199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:02.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.520+0000 7f1237f60700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1230199dd0 con 0x7f1230105b10 2024-04-27T01:58:02.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.520+0000 7f1235cfc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1230105b10 0x7f1230199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:02.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.520+0000 7f1235cfc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1230105b10 0x7f1230199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47752/0 (socket says 172.21.15.29:47752) 2024-04-27T01:58:02.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.520+0000 7f1235cfc700 1 -- 172.21.15.29:0/1454349993 learned_addr learned my addr 172.21.15.29:0/1454349993 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:58:02.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.520+0000 7f1235cfc700 1 -- 172.21.15.29:0/1454349993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f122c009550 con 0x7f1230105b10 2024-04-27T01:58:02.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.520+0000 7f1235cfc700 1 --2- 172.21.15.29:0/1454349993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1230105b10 0x7f1230199890 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f122c011ad0 tx=0x7f122c011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:02.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.521+0000 7f1226ffd700 1 -- 172.21.15.29:0/1454349993 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f122c003ab0 con 0x7f1230105b10 2024-04-27T01:58:02.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.521+0000 7f1226ffd700 1 -- 172.21.15.29:0/1454349993 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f122c003c10 con 0x7f1230105b10 2024-04-27T01:58:02.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.521+0000 7f1226ffd700 1 -- 172.21.15.29:0/1454349993 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f122c0198e0 con 0x7f1230105b10 2024-04-27T01:58:02.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.521+0000 7f1237f60700 1 -- 172.21.15.29:0/1454349993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1230199fd0 con 0x7f1230105b10 2024-04-27T01:58:02.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.521+0000 7f1237f60700 1 -- 172.21.15.29:0/1454349993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f123019a3f0 con 0x7f1230105b10 2024-04-27T01:58:02.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.522+0000 7f1226ffd700 1 -- 172.21.15.29:0/1454349993 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f122c019b50 con 0x7f1230105b10 2024-04-27T01:58:02.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.522+0000 7f1237f60700 1 -- 172.21.15.29:0/1454349993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1230062380 con 0x7f1230105b10 2024-04-27T01:58:02.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.522+0000 7f1226ffd700 1 --2- 172.21.15.29:0/1454349993 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f121c03c8a0 0x7f121c03ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:02.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.522+0000 7f1226ffd700 1 -- 172.21.15.29:0/1454349993 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f122c04c240 con 0x7f1230105b10 2024-04-27T01:58:02.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.522+0000 7f12354fb700 1 --2- 172.21.15.29:0/1454349993 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f121c03c8a0 0x7f121c03ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:02.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.523+0000 7f12354fb700 1 --2- 172.21.15.29:0/1454349993 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f121c03c8a0 0x7f121c03ed50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1220006fd0 tx=0x7f1220006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:02.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.527+0000 7f1226ffd700 1 -- 172.21.15.29:0/1454349993 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f122c01a430 con 0x7f1230105b10 2024-04-27T01:58:02.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:01.759+0000 7f1237f60700 1 -- 172.21.15.29:0/1454349993 --> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f12300621a0 con 0x7f121c03c8a0 2024-04-27T01:58:02.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:02.005+0000 7f1226ffd700 1 -- 172.21.15.29:0/1454349993 <== mgr.14164 v2:172.21.15.29:6800/2469926221 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f12300621a0 con 0x7f121c03c8a0 2024-04-27T01:58:02.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:02.007+0000 7f1237f60700 1 -- 172.21.15.29:0/1454349993 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f121c03c8a0 msgr2=0x7f121c03ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:02.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:02.007+0000 7f1237f60700 1 --2- 172.21.15.29:0/1454349993 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f121c03c8a0 0x7f121c03ed50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1220006fd0 tx=0x7f1220006e40 comp rx=0 tx=0).stop 2024-04-27T01:58:02.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:02.007+0000 7f1237f60700 1 -- 172.21.15.29:0/1454349993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1230105b10 msgr2=0x7f1230199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:02.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:02.007+0000 7f1237f60700 1 --2- 172.21.15.29:0/1454349993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1230105b10 0x7f1230199890 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f122c011ad0 tx=0x7f122c011bb0 comp rx=0 tx=0).stop 2024-04-27T01:58:02.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:02.008+0000 7f1237f60700 1 -- 172.21.15.29:0/1454349993 shutdown_connections 2024-04-27T01:58:02.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:02.008+0000 7f1237f60700 1 --2- 172.21.15.29:0/1454349993 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f121c03c8a0 0x7f121c03ed50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:02.737 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:02.008+0000 7f1237f60700 1 --2- 172.21.15.29:0/1454349993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1230105b10 0x7f1230199890 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:02.737 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:02.008+0000 7f1237f60700 1 -- 172.21.15.29:0/1454349993 >> 172.21.15.29:0/1454349993 conn(0x7f1230101180 msgr2=0x7f1230101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:02.737 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:02.008+0000 7f1237f60700 1 -- 172.21.15.29:0/1454349993 shutdown_connections 2024-04-27T01:58:02.737 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:02.008+0000 7f1237f60700 1 -- 172.21.15.29:0/1454349993 wait complete. 2024-04-27T01:58:02.737 INFO:teuthology.orchestra.run.smithi029.stdout:Fetching dashboard port number... 2024-04-27T01:58:03.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:03 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/3381250426' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-27T01:58:04.272 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 8443 2024-04-27T01:58:04.272 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.211+0000 7f8e940f9700 1 Processor -- start 2024-04-27T01:58:04.272 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.212+0000 7f8e940f9700 1 -- start start 2024-04-27T01:58:04.272 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.212+0000 7f8e940f9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e8c105b10 0x7f8e8c105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:04.272 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.212+0000 7f8e940f9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c106460 con 0x7f8e8c105b10 2024-04-27T01:58:04.272 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.212+0000 7f8e91e95700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e8c105b10 0x7f8e8c105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:04.272 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.212+0000 7f8e91e95700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e8c105b10 0x7f8e8c105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47754/0 (socket says 172.21.15.29:47754) 2024-04-27T01:58:04.273 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.212+0000 7f8e91e95700 1 -- 172.21.15.29:0/124168792 learned_addr learned my addr 172.21.15.29:0/124168792 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:58:04.273 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.212+0000 7f8e91e95700 1 -- 172.21.15.29:0/124168792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8c1065a0 con 0x7f8e8c105b10 2024-04-27T01:58:04.273 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.212+0000 7f8e91e95700 1 --2- 172.21.15.29:0/124168792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e8c105b10 0x7f8e8c105f20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8e880098a0 tx=0x7f8e88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e12e205db1ec83c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:04.273 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.213+0000 7f8e90e93700 1 -- 172.21.15.29:0/124168792 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e88004180 con 0x7f8e8c105b10 2024-04-27T01:58:04.273 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.213+0000 7f8e90e93700 1 -- 172.21.15.29:0/124168792 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8e880042e0 con 0x7f8e8c105b10 2024-04-27T01:58:04.273 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.213+0000 7f8e90e93700 1 -- 172.21.15.29:0/124168792 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e88011af0 con 0x7f8e8c105b10 2024-04-27T01:58:04.273 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.213+0000 7f8e940f9700 1 -- 172.21.15.29:0/124168792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e8c105b10 msgr2=0x7f8e8c105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:04.273 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.213+0000 7f8e940f9700 1 --2- 172.21.15.29:0/124168792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e8c105b10 0x7f8e8c105f20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8e880098a0 tx=0x7f8e88009bb0 comp rx=0 tx=0).stop 2024-04-27T01:58:04.273 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.213+0000 7f8e940f9700 1 -- 172.21.15.29:0/124168792 shutdown_connections 2024-04-27T01:58:04.273 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.213+0000 7f8e940f9700 1 --2- 172.21.15.29:0/124168792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e8c105b10 0x7f8e8c105f20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:04.273 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.213+0000 7f8e940f9700 1 -- 172.21.15.29:0/124168792 >> 172.21.15.29:0/124168792 conn(0x7f8e8c101180 msgr2=0x7f8e8c103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:04.273 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.213+0000 7f8e940f9700 1 -- 172.21.15.29:0/124168792 shutdown_connections 2024-04-27T01:58:04.273 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.213+0000 7f8e940f9700 1 -- 172.21.15.29:0/124168792 wait complete. 2024-04-27T01:58:04.274 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.214+0000 7f8e940f9700 1 Processor -- start 2024-04-27T01:58:04.274 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.214+0000 7f8e940f9700 1 -- start start 2024-04-27T01:58:04.274 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.214+0000 7f8e940f9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e8c105b10 0x7f8e8c1998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:04.274 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.214+0000 7f8e940f9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c199de0 con 0x7f8e8c105b10 2024-04-27T01:58:04.274 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.214+0000 7f8e91e95700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e8c105b10 0x7f8e8c1998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:04.274 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.214+0000 7f8e91e95700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e8c105b10 0x7f8e8c1998a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47768/0 (socket says 172.21.15.29:47768) 2024-04-27T01:58:04.274 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.214+0000 7f8e91e95700 1 -- 172.21.15.29:0/3381250426 learned_addr learned my addr 172.21.15.29:0/3381250426 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:58:04.274 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.214+0000 7f8e91e95700 1 -- 172.21.15.29:0/3381250426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e88009550 con 0x7f8e8c105b10 2024-04-27T01:58:04.274 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.215+0000 7f8e91e95700 1 --2- 172.21.15.29:0/3381250426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e8c105b10 0x7f8e8c1998a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8e88006ba0 tx=0x7f8e88011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:04.274 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.215+0000 7f8e82ffd700 1 -- 172.21.15.29:0/3381250426 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e88003b00 con 0x7f8e8c105b10 2024-04-27T01:58:04.275 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.215+0000 7f8e82ffd700 1 -- 172.21.15.29:0/3381250426 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8e8801a8e0 con 0x7f8e8c105b10 2024-04-27T01:58:04.275 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.215+0000 7f8e940f9700 1 -- 172.21.15.29:0/3381250426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e8c199fe0 con 0x7f8e8c105b10 2024-04-27T01:58:04.275 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.215+0000 7f8e82ffd700 1 -- 172.21.15.29:0/3381250426 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e880199c0 con 0x7f8e8c105b10 2024-04-27T01:58:04.275 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.215+0000 7f8e940f9700 1 -- 172.21.15.29:0/3381250426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e8c19a400 con 0x7f8e8c105b10 2024-04-27T01:58:04.275 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.216+0000 7f8e82ffd700 1 -- 172.21.15.29:0/3381250426 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f8e88019c80 con 0x7f8e8c105b10 2024-04-27T01:58:04.275 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.216+0000 7f8e940f9700 1 -- 172.21.15.29:0/3381250426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e8c062380 con 0x7f8e8c105b10 2024-04-27T01:58:04.275 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.216+0000 7f8e82ffd700 1 --2- 172.21.15.29:0/3381250426 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f8e780383f0 0x7f8e7803a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:04.275 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.216+0000 7f8e82ffd700 1 -- 172.21.15.29:0/3381250426 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8e8804c750 con 0x7f8e8c105b10 2024-04-27T01:58:04.275 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.216+0000 7f8e91694700 1 --2- 172.21.15.29:0/3381250426 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f8e780383f0 0x7f8e7803a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:04.275 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.217+0000 7f8e91694700 1 --2- 172.21.15.29:0/3381250426 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f8e780383f0 0x7f8e7803a8a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c006fd0 tx=0x7f8e7c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:04.275 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.221+0000 7f8e82ffd700 1 -- 172.21.15.29:0/3381250426 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8e88018b40 con 0x7f8e8c105b10 2024-04-27T01:58:04.275 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.427+0000 7f8e940f9700 1 -- 172.21.15.29:0/3381250426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f8e8c19d260 con 0x7f8e8c105b10 2024-04-27T01:58:04.276 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.427+0000 7f8e82ffd700 1 -- 172.21.15.29:0/3381250426 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f8e88018b40 con 0x7f8e8c105b10 2024-04-27T01:58:04.276 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.429+0000 7f8e940f9700 1 -- 172.21.15.29:0/3381250426 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f8e780383f0 msgr2=0x7f8e7803a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:04.276 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.429+0000 7f8e940f9700 1 --2- 172.21.15.29:0/3381250426 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f8e780383f0 0x7f8e7803a8a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c006fd0 tx=0x7f8e7c006e40 comp rx=0 tx=0).stop 2024-04-27T01:58:04.276 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.429+0000 7f8e940f9700 1 -- 172.21.15.29:0/3381250426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e8c105b10 msgr2=0x7f8e8c1998a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:04.276 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.429+0000 7f8e940f9700 1 --2- 172.21.15.29:0/3381250426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e8c105b10 0x7f8e8c1998a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8e88006ba0 tx=0x7f8e88011b50 comp rx=0 tx=0).stop 2024-04-27T01:58:04.276 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.429+0000 7f8e940f9700 1 -- 172.21.15.29:0/3381250426 shutdown_connections 2024-04-27T01:58:04.276 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.429+0000 7f8e940f9700 1 --2- 172.21.15.29:0/3381250426 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f8e780383f0 0x7f8e7803a8a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:04.276 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.429+0000 7f8e940f9700 1 --2- 172.21.15.29:0/3381250426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e8c105b10 0x7f8e8c1998a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:04.276 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.429+0000 7f8e940f9700 1 -- 172.21.15.29:0/3381250426 >> 172.21.15.29:0/3381250426 conn(0x7f8e8c101180 msgr2=0x7f8e8c1027c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:04.276 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.430+0000 7f8e940f9700 1 -- 172.21.15.29:0/3381250426 shutdown_connections 2024-04-27T01:58:04.276 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:03.430+0000 7f8e940f9700 1 -- 172.21.15.29:0/3381250426 wait complete. 2024-04-27T01:58:04.280 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-27T01:58:04.280 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-04-27T01:58:04.288 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-27T01:58:04.288 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-04-27T01:58:04.288 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-04-27T01:58:04.288 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-27T01:58:04.290 INFO:teuthology.orchestra.run.smithi029.stdout:Ceph Dashboard is now available at: 2024-04-27T01:58:04.290 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:58:04.290 INFO:teuthology.orchestra.run.smithi029.stdout: URL: https://smithi029:8443/ 2024-04-27T01:58:04.290 INFO:teuthology.orchestra.run.smithi029.stdout: User: admin 2024-04-27T01:58:04.290 INFO:teuthology.orchestra.run.smithi029.stdout: Password: wcpjbl20tz 2024-04-27T01:58:04.290 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:58:04.290 INFO:teuthology.orchestra.run.smithi029.stdout:Saving cluster configuration to /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/config directory 2024-04-27T01:58:04.291 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling autotune for osd_memory_target 2024-04-27T01:58:05.501 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:05 smithi029 ceph-mon[26769]: mgrmap e13: smithi029.ewvdzs(active, since 6s) 2024-04-27T01:58:05.509 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.722+0000 7f1c6ab33700 1 Processor -- start 2024-04-27T01:58:05.509 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.722+0000 7f1c6ab33700 1 -- start start 2024-04-27T01:58:05.509 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.722+0000 7f1c6ab33700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c64104a00 0x7f1c64104e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:05.509 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.722+0000 7f1c6ab33700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c64105350 con 0x7f1c64104a00 2024-04-27T01:58:05.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.723+0000 7f1c688cf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c64104a00 0x7f1c64104e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:05.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.723+0000 7f1c688cf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c64104a00 0x7f1c64104e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47770/0 (socket says 172.21.15.29:47770) 2024-04-27T01:58:05.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.723+0000 7f1c688cf700 1 -- 172.21.15.29:0/3462736931 learned_addr learned my addr 172.21.15.29:0/3462736931 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:58:05.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.723+0000 7f1c688cf700 1 -- 172.21.15.29:0/3462736931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c64105490 con 0x7f1c64104a00 2024-04-27T01:58:05.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.723+0000 7f1c688cf700 1 --2- 172.21.15.29:0/3462736931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c64104a00 0x7f1c64104e10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c009e80 tx=0x7f1c5c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91760730f1ce87ea server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:05.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.723+0000 7f1c637fe700 1 -- 172.21.15.29:0/3462736931 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c5c00e070 con 0x7f1c64104a00 2024-04-27T01:58:05.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.723+0000 7f1c637fe700 1 -- 172.21.15.29:0/3462736931 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1c5c00b8c0 con 0x7f1c64104a00 2024-04-27T01:58:05.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.723+0000 7f1c637fe700 1 -- 172.21.15.29:0/3462736931 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c5c0044d0 con 0x7f1c64104a00 2024-04-27T01:58:05.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.724+0000 7f1c6ab33700 1 -- 172.21.15.29:0/3462736931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c64104a00 msgr2=0x7f1c64104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:05.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.724+0000 7f1c6ab33700 1 --2- 172.21.15.29:0/3462736931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c64104a00 0x7f1c64104e10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c009e80 tx=0x7f1c5c00b330 comp rx=0 tx=0).stop 2024-04-27T01:58:05.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.724+0000 7f1c6ab33700 1 -- 172.21.15.29:0/3462736931 shutdown_connections 2024-04-27T01:58:05.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.724+0000 7f1c6ab33700 1 --2- 172.21.15.29:0/3462736931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c64104a00 0x7f1c64104e10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:05.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.724+0000 7f1c6ab33700 1 -- 172.21.15.29:0/3462736931 >> 172.21.15.29:0/3462736931 conn(0x7f1c64100070 msgr2=0x7f1c64102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:05.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.724+0000 7f1c6ab33700 1 -- 172.21.15.29:0/3462736931 shutdown_connections 2024-04-27T01:58:05.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.724+0000 7f1c6ab33700 1 -- 172.21.15.29:0/3462736931 wait complete. 2024-04-27T01:58:05.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.725+0000 7f1c6ab33700 1 Processor -- start 2024-04-27T01:58:05.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.725+0000 7f1c6ab33700 1 -- start start 2024-04-27T01:58:05.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.725+0000 7f1c6ab33700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c64104a00 0x7f1c64198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:05.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.725+0000 7f1c6ab33700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c64198c60 con 0x7f1c64104a00 2024-04-27T01:58:05.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.725+0000 7f1c688cf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c64104a00 0x7f1c64198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:05.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.725+0000 7f1c688cf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c64104a00 0x7f1c64198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47780/0 (socket says 172.21.15.29:47780) 2024-04-27T01:58:05.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.725+0000 7f1c688cf700 1 -- 172.21.15.29:0/1865824673 learned_addr learned my addr 172.21.15.29:0/1865824673 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:58:05.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.725+0000 7f1c688cf700 1 -- 172.21.15.29:0/1865824673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5c009b30 con 0x7f1c64104a00 2024-04-27T01:58:05.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.726+0000 7f1c688cf700 1 --2- 172.21.15.29:0/1865824673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c64104a00 0x7f1c64198720 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c0072a0 tx=0x7f1c5c0049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:05.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.726+0000 7f1c61ffb700 1 -- 172.21.15.29:0/1865824673 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c5c00e040 con 0x7f1c64104a00 2024-04-27T01:58:05.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.726+0000 7f1c6ab33700 1 -- 172.21.15.29:0/1865824673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c64198e60 con 0x7f1c64104a00 2024-04-27T01:58:05.512 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.726+0000 7f1c61ffb700 1 -- 172.21.15.29:0/1865824673 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1c5c00bd80 con 0x7f1c64104a00 2024-04-27T01:58:05.512 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.726+0000 7f1c61ffb700 1 -- 172.21.15.29:0/1865824673 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c5c0126c0 con 0x7f1c64104a00 2024-04-27T01:58:05.512 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.726+0000 7f1c6ab33700 1 -- 172.21.15.29:0/1865824673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c641992e0 con 0x7f1c64104a00 2024-04-27T01:58:05.512 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.727+0000 7f1c61ffb700 1 -- 172.21.15.29:0/1865824673 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f1c5c019040 con 0x7f1c64104a00 2024-04-27T01:58:05.512 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.727+0000 7f1c61ffb700 1 --2- 172.21.15.29:0/1865824673 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f1c4c0384a0 0x7f1c4c03a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:05.512 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.727+0000 7f1c61ffb700 1 -- 172.21.15.29:0/1865824673 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1c5c04c030 con 0x7f1c64104a00 2024-04-27T01:58:05.512 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.727+0000 7f1c63fff700 1 --2- 172.21.15.29:0/1865824673 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f1c4c0384a0 0x7f1c4c03a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:05.512 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.727+0000 7f1c6ab33700 1 -- 172.21.15.29:0/1865824673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c64062380 con 0x7f1c64104a00 2024-04-27T01:58:05.512 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.728+0000 7f1c63fff700 1 --2- 172.21.15.29:0/1865824673 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f1c4c0384a0 0x7f1c4c03a950 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1c58006fd0 tx=0x7f1c58006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:05.512 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.732+0000 7f1c61ffb700 1 -- 172.21.15.29:0/1865824673 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1c5c01f080 con 0x7f1c64104a00 2024-04-27T01:58:05.512 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.931+0000 7f1c6ab33700 1 -- 172.21.15.29:0/1865824673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f1c6419c320 con 0x7f1c64104a00 2024-04-27T01:58:05.513 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.932+0000 7f1c61ffb700 1 -- 172.21.15.29:0/1865824673 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f1c5c04a0c0 con 0x7f1c64104a00 2024-04-27T01:58:05.513 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.934+0000 7f1c6ab33700 1 -- 172.21.15.29:0/1865824673 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f1c4c0384a0 msgr2=0x7f1c4c03a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:05.513 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.934+0000 7f1c6ab33700 1 --2- 172.21.15.29:0/1865824673 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f1c4c0384a0 0x7f1c4c03a950 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1c58006fd0 tx=0x7f1c58006e40 comp rx=0 tx=0).stop 2024-04-27T01:58:05.513 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.934+0000 7f1c6ab33700 1 -- 172.21.15.29:0/1865824673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c64104a00 msgr2=0x7f1c64198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:05.513 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.934+0000 7f1c6ab33700 1 --2- 172.21.15.29:0/1865824673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c64104a00 0x7f1c64198720 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c0072a0 tx=0x7f1c5c0049f0 comp rx=0 tx=0).stop 2024-04-27T01:58:05.513 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.934+0000 7f1c6ab33700 1 -- 172.21.15.29:0/1865824673 shutdown_connections 2024-04-27T01:58:05.513 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.934+0000 7f1c6ab33700 1 --2- 172.21.15.29:0/1865824673 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f1c4c0384a0 0x7f1c4c03a950 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:05.513 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.934+0000 7f1c6ab33700 1 --2- 172.21.15.29:0/1865824673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c64104a00 0x7f1c64198720 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:05.513 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.934+0000 7f1c6ab33700 1 -- 172.21.15.29:0/1865824673 >> 172.21.15.29:0/1865824673 conn(0x7f1c64100070 msgr2=0x7f1c64100dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:05.513 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.934+0000 7f1c6ab33700 1 -- 172.21.15.29:0/1865824673 shutdown_connections 2024-04-27T01:58:05.513 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:04.934+0000 7f1c6ab33700 1 -- 172.21.15.29:0/1865824673 wait complete. 2024-04-27T01:58:06.866 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.953+0000 7fbc0e5eb700 1 Processor -- start 2024-04-27T01:58:06.866 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.953+0000 7fbc0e5eb700 1 -- start start 2024-04-27T01:58:06.866 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.953+0000 7fbc0e5eb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc0807bdb0 0x7fbc0807a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:06.867 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.953+0000 7fbc0e5eb700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0807a7f0 con 0x7fbc0807bdb0 2024-04-27T01:58:06.867 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.953+0000 7fbc07fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc0807bdb0 0x7fbc0807a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:06.867 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.954+0000 7fbc07fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc0807bdb0 0x7fbc0807a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47794/0 (socket says 172.21.15.29:47794) 2024-04-27T01:58:06.867 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.954+0000 7fbc07fff700 1 -- 172.21.15.29:0/1884900524 learned_addr learned my addr 172.21.15.29:0/1884900524 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:58:06.867 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.954+0000 7fbc07fff700 1 -- 172.21.15.29:0/1884900524 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0807a930 con 0x7fbc0807bdb0 2024-04-27T01:58:06.867 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.954+0000 7fbc07fff700 1 --2- 172.21.15.29:0/1884900524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc0807bdb0 0x7fbc0807a2b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbbf80098a0 tx=0x7fbbf8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b13290401883ac3 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:06.867 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.954+0000 7fbc06ffd700 1 -- 172.21.15.29:0/1884900524 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbf8004180 con 0x7fbc0807bdb0 2024-04-27T01:58:06.867 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.954+0000 7fbc06ffd700 1 -- 172.21.15.29:0/1884900524 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbbf80042e0 con 0x7fbc0807bdb0 2024-04-27T01:58:06.867 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.954+0000 7fbc06ffd700 1 -- 172.21.15.29:0/1884900524 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbf8011af0 con 0x7fbc0807bdb0 2024-04-27T01:58:06.867 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.955+0000 7fbc0e5eb700 1 -- 172.21.15.29:0/1884900524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc0807bdb0 msgr2=0x7fbc0807a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:06.867 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.955+0000 7fbc0e5eb700 1 --2- 172.21.15.29:0/1884900524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc0807bdb0 0x7fbc0807a2b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbbf80098a0 tx=0x7fbbf8009bb0 comp rx=0 tx=0).stop 2024-04-27T01:58:06.867 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.955+0000 7fbc0e5eb700 1 -- 172.21.15.29:0/1884900524 shutdown_connections 2024-04-27T01:58:06.867 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.955+0000 7fbc0e5eb700 1 --2- 172.21.15.29:0/1884900524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc0807bdb0 0x7fbc0807a2b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:06.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.955+0000 7fbc0e5eb700 1 -- 172.21.15.29:0/1884900524 >> 172.21.15.29:0/1884900524 conn(0x7fbc08101180 msgr2=0x7fbc08103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:06.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.955+0000 7fbc0e5eb700 1 -- 172.21.15.29:0/1884900524 shutdown_connections 2024-04-27T01:58:06.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.955+0000 7fbc0e5eb700 1 -- 172.21.15.29:0/1884900524 wait complete. 2024-04-27T01:58:06.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.955+0000 7fbc0e5eb700 1 Processor -- start 2024-04-27T01:58:06.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.956+0000 7fbc0e5eb700 1 -- start start 2024-04-27T01:58:06.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.956+0000 7fbc0e5eb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc0807bdb0 0x7fbc0819baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:06.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.956+0000 7fbc0e5eb700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0819bfe0 con 0x7fbc0807bdb0 2024-04-27T01:58:06.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.956+0000 7fbc07fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc0807bdb0 0x7fbc0819baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:06.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.956+0000 7fbc07fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc0807bdb0 0x7fbc0819baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47808/0 (socket says 172.21.15.29:47808) 2024-04-27T01:58:06.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.956+0000 7fbc07fff700 1 -- 172.21.15.29:0/1021753380 learned_addr learned my addr 172.21.15.29:0/1021753380 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:58:06.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.956+0000 7fbc07fff700 1 -- 172.21.15.29:0/1021753380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf8009550 con 0x7fbc0807bdb0 2024-04-27T01:58:06.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.956+0000 7fbc07fff700 1 --2- 172.21.15.29:0/1021753380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc0807bdb0 0x7fbc0819baa0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8000c00 tx=0x7fbbf8011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:06.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.957+0000 7fbc057fa700 1 -- 172.21.15.29:0/1021753380 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbf8003a70 con 0x7fbc0807bdb0 2024-04-27T01:58:06.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.957+0000 7fbc057fa700 1 -- 172.21.15.29:0/1021753380 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbbf8003bd0 con 0x7fbc0807bdb0 2024-04-27T01:58:06.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.957+0000 7fbc057fa700 1 -- 172.21.15.29:0/1021753380 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbf80238c0 con 0x7fbc0807bdb0 2024-04-27T01:58:06.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.957+0000 7fbc0e5eb700 1 -- 172.21.15.29:0/1021753380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc0819c1e0 con 0x7fbc0807bdb0 2024-04-27T01:58:06.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.957+0000 7fbc0e5eb700 1 -- 172.21.15.29:0/1021753380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc0819c600 con 0x7fbc0807bdb0 2024-04-27T01:58:06.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.957+0000 7fbc057fa700 1 -- 172.21.15.29:0/1021753380 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7fbbf8023b80 con 0x7fbc0807bdb0 2024-04-27T01:58:06.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.958+0000 7fbc057fa700 1 --2- 172.21.15.29:0/1021753380 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fbbf00384e0 0x7fbbf003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:06.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.958+0000 7fbc057fa700 1 -- 172.21.15.29:0/1021753380 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbbf804c400 con 0x7fbc0807bdb0 2024-04-27T01:58:06.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.958+0000 7fbc0e5eb700 1 -- 172.21.15.29:0/1021753380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc0810be20 con 0x7fbc0807bdb0 2024-04-27T01:58:06.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.958+0000 7fbc077fe700 1 --2- 172.21.15.29:0/1021753380 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fbbf00384e0 0x7fbbf003a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:06.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.958+0000 7fbc077fe700 1 --2- 172.21.15.29:0/1021753380 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fbbf00384e0 0x7fbbf003a990 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc006fd0 tx=0x7fbbfc006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:06.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:05.963+0000 7fbc057fa700 1 -- 172.21.15.29:0/1021753380 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fbbf8010d10 con 0x7fbc0807bdb0 2024-04-27T01:58:06.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:06.274+0000 7fbc0e5eb700 1 -- 172.21.15.29:0/1021753380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fbc0819f460 con 0x7fbc0807bdb0 2024-04-27T01:58:06.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:06.277+0000 7fbc057fa700 1 -- 172.21.15.29:0/1021753380 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7fbbf8010b30 con 0x7fbc0807bdb0 2024-04-27T01:58:06.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:06.279+0000 7fbc0e5eb700 1 -- 172.21.15.29:0/1021753380 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fbbf00384e0 msgr2=0x7fbbf003a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:06.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:06.279+0000 7fbc0e5eb700 1 --2- 172.21.15.29:0/1021753380 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fbbf00384e0 0x7fbbf003a990 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc006fd0 tx=0x7fbbfc006e40 comp rx=0 tx=0).stop 2024-04-27T01:58:06.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:06.279+0000 7fbc0e5eb700 1 -- 172.21.15.29:0/1021753380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc0807bdb0 msgr2=0x7fbc0819baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:06.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:06.279+0000 7fbc0e5eb700 1 --2- 172.21.15.29:0/1021753380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc0807bdb0 0x7fbc0819baa0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8000c00 tx=0x7fbbf8011b50 comp rx=0 tx=0).stop 2024-04-27T01:58:06.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:06.279+0000 7fbc0e5eb700 1 -- 172.21.15.29:0/1021753380 shutdown_connections 2024-04-27T01:58:06.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:06.279+0000 7fbc0e5eb700 1 --2- 172.21.15.29:0/1021753380 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fbbf00384e0 0x7fbbf003a990 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:06.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:06.279+0000 7fbc0e5eb700 1 --2- 172.21.15.29:0/1021753380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc0807bdb0 0x7fbc0819baa0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:06.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:06.279+0000 7fbc0e5eb700 1 -- 172.21.15.29:0/1021753380 >> 172.21.15.29:0/1021753380 conn(0x7fbc08101180 msgr2=0x7fbc08102830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:06.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:06.279+0000 7fbc0e5eb700 1 -- 172.21.15.29:0/1021753380 shutdown_connections 2024-04-27T01:58:06.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-04-27T01:58:06.279+0000 7fbc0e5eb700 1 -- 172.21.15.29:0/1021753380 wait complete. 2024-04-27T01:58:06.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-27T01:58:06.871 INFO:teuthology.orchestra.run.smithi029.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-27T01:58:06.872 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:58:06.872 INFO:teuthology.orchestra.run.smithi029.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-27T01:58:06.872 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:58:06.872 INFO:teuthology.orchestra.run.smithi029.stdout:Or, if you are only running a single cluster on this host: 2024-04-27T01:58:06.872 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:58:06.872 INFO:teuthology.orchestra.run.smithi029.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-27T01:58:06.872 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:58:06.872 INFO:teuthology.orchestra.run.smithi029.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-27T01:58:06.872 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:58:06.872 INFO:teuthology.orchestra.run.smithi029.stdout: ceph telemetry on 2024-04-27T01:58:06.872 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:58:06.872 INFO:teuthology.orchestra.run.smithi029.stdout:For more information see: 2024-04-27T01:58:06.873 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:58:06.873 INFO:teuthology.orchestra.run.smithi029.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-27T01:58:06.873 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:58:06.873 INFO:teuthology.orchestra.run.smithi029.stdout:Bootstrap complete. 2024-04-27T01:58:06.897 INFO:tasks.cephadm:Fetching config... 2024-04-27T01:58:06.898 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-27T01:58:06.898 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-27T01:58:06.916 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-27T01:58:06.916 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-27T01:58:06.916 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-27T01:58:06.972 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-27T01:58:06.972 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-27T01:58:06.972 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/keyring of=/dev/stdout 2024-04-27T01:58:07.037 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-27T01:58:07.037 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-27T01:58:07.037 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-27T01:58:07.093 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-27T01:58:07.093 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrvDQxSFEILBV4WWqSNuz9odisfN66Pm2UacN9QGO+79rzFM2SXj61/pttSR176IQtxnTwGJO1FFwelosoe7Us7M6iAW+3n5zwOPAFsoCCDbd/JO+0THx7WD6PUA/5MUG4rlUzaI3WapyNx/6wRotuDanxe6HnNQ/tQXCk2pLo8JJxMc5HX+njVi1zPVwZx55K+AUAKcr9lOQqemUCdIr0zMgHJgL38FhbnvyZBUFr4oT3XnC1641Q77WrDrf/1Crko1JofmnUeqjVnRIh8uc4dtZnzTaUva8qQ7V58AzhBwMURxQbkkc7t7eFCBSgR55Qe0DeZhTq5c+CrVYnT0+A9RqX5ykG5mMroRI4bhKPmx+2X0r8WcoKaz94TB0J1TNMpE83QQJ+9eKnEyxQ487cOiNyaw02rxFMOW6FoKF4Ok/vgd9jN0DLYHerDgb6G08cHQxby5BE03drMYygt08rmxcDQTGev4bkQ3J4F7QBBkNpoLLh3/gZayKyjhOpJnE= ceph-4c587076-0439-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-27T01:58:07.167 INFO:teuthology.orchestra.run.smithi029.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrvDQxSFEILBV4WWqSNuz9odisfN66Pm2UacN9QGO+79rzFM2SXj61/pttSR176IQtxnTwGJO1FFwelosoe7Us7M6iAW+3n5zwOPAFsoCCDbd/JO+0THx7WD6PUA/5MUG4rlUzaI3WapyNx/6wRotuDanxe6HnNQ/tQXCk2pLo8JJxMc5HX+njVi1zPVwZx55K+AUAKcr9lOQqemUCdIr0zMgHJgL38FhbnvyZBUFr4oT3XnC1641Q77WrDrf/1Crko1JofmnUeqjVnRIh8uc4dtZnzTaUva8qQ7V58AzhBwMURxQbkkc7t7eFCBSgR55Qe0DeZhTq5c+CrVYnT0+A9RqX5ykG5mMroRI4bhKPmx+2X0r8WcoKaz94TB0J1TNMpE83QQJ+9eKnEyxQ487cOiNyaw02rxFMOW6FoKF4Ok/vgd9jN0DLYHerDgb6G08cHQxby5BE03drMYygt08rmxcDQTGev4bkQ3J4F7QBBkNpoLLh3/gZayKyjhOpJnE= ceph-4c587076-0439-11ef-bc93-c7b262605968 2024-04-27T01:58:07.181 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrvDQxSFEILBV4WWqSNuz9odisfN66Pm2UacN9QGO+79rzFM2SXj61/pttSR176IQtxnTwGJO1FFwelosoe7Us7M6iAW+3n5zwOPAFsoCCDbd/JO+0THx7WD6PUA/5MUG4rlUzaI3WapyNx/6wRotuDanxe6HnNQ/tQXCk2pLo8JJxMc5HX+njVi1zPVwZx55K+AUAKcr9lOQqemUCdIr0zMgHJgL38FhbnvyZBUFr4oT3XnC1641Q77WrDrf/1Crko1JofmnUeqjVnRIh8uc4dtZnzTaUva8qQ7V58AzhBwMURxQbkkc7t7eFCBSgR55Qe0DeZhTq5c+CrVYnT0+A9RqX5ykG5mMroRI4bhKPmx+2X0r8WcoKaz94TB0J1TNMpE83QQJ+9eKnEyxQ487cOiNyaw02rxFMOW6FoKF4Ok/vgd9jN0DLYHerDgb6G08cHQxby5BE03drMYygt08rmxcDQTGev4bkQ3J4F7QBBkNpoLLh3/gZayKyjhOpJnE= ceph-4c587076-0439-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-27T01:58:07.221 INFO:teuthology.orchestra.run.smithi052.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrvDQxSFEILBV4WWqSNuz9odisfN66Pm2UacN9QGO+79rzFM2SXj61/pttSR176IQtxnTwGJO1FFwelosoe7Us7M6iAW+3n5zwOPAFsoCCDbd/JO+0THx7WD6PUA/5MUG4rlUzaI3WapyNx/6wRotuDanxe6HnNQ/tQXCk2pLo8JJxMc5HX+njVi1zPVwZx55K+AUAKcr9lOQqemUCdIr0zMgHJgL38FhbnvyZBUFr4oT3XnC1641Q77WrDrf/1Crko1JofmnUeqjVnRIh8uc4dtZnzTaUva8qQ7V58AzhBwMURxQbkkc7t7eFCBSgR55Qe0DeZhTq5c+CrVYnT0+A9RqX5ykG5mMroRI4bhKPmx+2X0r8WcoKaz94TB0J1TNMpE83QQJ+9eKnEyxQ487cOiNyaw02rxFMOW6FoKF4Ok/vgd9jN0DLYHerDgb6G08cHQxby5BE03drMYygt08rmxcDQTGev4bkQ3J4F7QBBkNpoLLh3/gZayKyjhOpJnE= ceph-4c587076-0439-11ef-bc93-c7b262605968 2024-04-27T01:58:07.233 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-27T01:58:07.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:07 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/1021753380' entity='client.admin' 2024-04-27T01:58:07.475 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T01:58:10.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.059+0000 7fe8810bf700 1 -- 172.21.15.29:0/735223193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe87c0fed20 msgr2=0x7fe87c0ff130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:10.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.059+0000 7fe8810bf700 1 --2- 172.21.15.29:0/735223193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe87c0fed20 0x7fe87c0ff130 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00b0a0 tx=0x7fe86c00b3b0 comp rx=0 tx=0).stop 2024-04-27T01:58:10.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.059+0000 7fe8810bf700 1 -- 172.21.15.29:0/735223193 shutdown_connections 2024-04-27T01:58:10.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.059+0000 7fe8810bf700 1 --2- 172.21.15.29:0/735223193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe87c0fed20 0x7fe87c0ff130 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:10.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.059+0000 7fe8810bf700 1 -- 172.21.15.29:0/735223193 >> 172.21.15.29:0/735223193 conn(0x7fe87c0fa2b0 msgr2=0x7fe87c0fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:10.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.059+0000 7fe8810bf700 1 -- 172.21.15.29:0/735223193 shutdown_connections 2024-04-27T01:58:10.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.059+0000 7fe8810bf700 1 -- 172.21.15.29:0/735223193 wait complete. 2024-04-27T01:58:10.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.059+0000 7fe8810bf700 1 Processor -- start 2024-04-27T01:58:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.060+0000 7fe8810bf700 1 -- start start 2024-04-27T01:58:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.060+0000 7fe8810bf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe87c0fed20 0x7fe87c194fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.060+0000 7fe8810bf700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c1954f0 con 0x7fe87c0fed20 2024-04-27T01:58:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.060+0000 7fe87ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe87c0fed20 0x7fe87c194fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.060+0000 7fe87ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe87c0fed20 0x7fe87c194fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40038/0 (socket says 172.21.15.29:40038) 2024-04-27T01:58:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.060+0000 7fe87ad9d700 1 -- 172.21.15.29:0/3933284232 learned_addr learned my addr 172.21.15.29:0/3933284232 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:58:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.060+0000 7fe87ad9d700 1 -- 172.21.15.29:0/3933284232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe86c009d00 con 0x7fe87c0fed20 2024-04-27T01:58:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.060+0000 7fe87ad9d700 1 --2- 172.21.15.29:0/3933284232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe87c0fed20 0x7fe87c194fb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe86c009fd0 tx=0x7fe86c00bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.060+0000 7fe873fff700 1 -- 172.21.15.29:0/3933284232 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe86c00e040 con 0x7fe87c0fed20 2024-04-27T01:58:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.061+0000 7fe8810bf700 1 -- 172.21.15.29:0/3933284232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe87c1956f0 con 0x7fe87c0fed20 2024-04-27T01:58:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.061+0000 7fe873fff700 1 -- 172.21.15.29:0/3933284232 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe86c00bdc0 con 0x7fe87c0fed20 2024-04-27T01:58:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.061+0000 7fe873fff700 1 -- 172.21.15.29:0/3933284232 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe86c01da40 con 0x7fe87c0fed20 2024-04-27T01:58:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.061+0000 7fe8810bf700 1 -- 172.21.15.29:0/3933284232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe87c195b10 con 0x7fe87c0fed20 2024-04-27T01:58:10.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.061+0000 7fe873fff700 1 -- 172.21.15.29:0/3933284232 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7fe86c01dc30 con 0x7fe87c0fed20 2024-04-27T01:58:10.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.062+0000 7fe873fff700 1 --2- 172.21.15.29:0/3933284232 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fe868038530 0x7fe86803a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:10.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.062+0000 7fe873fff700 1 -- 172.21.15.29:0/3933284232 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe86c04d880 con 0x7fe87c0fed20 2024-04-27T01:58:10.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.062+0000 7fe8810bf700 1 -- 172.21.15.29:0/3933284232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe87c18ea10 con 0x7fe87c0fed20 2024-04-27T01:58:10.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.062+0000 7fe87a59c700 1 --2- 172.21.15.29:0/3933284232 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fe868038530 0x7fe86803a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:10.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.062+0000 7fe87a59c700 1 --2- 172.21.15.29:0/3933284232 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fe868038530 0x7fe86803a9e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe864006fd0 tx=0x7fe864006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:10.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.067+0000 7fe873fff700 1 -- 172.21.15.29:0/3933284232 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe86c01c3e0 con 0x7fe87c0fed20 2024-04-27T01:58:10.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.266+0000 7fe8810bf700 1 -- 172.21.15.29:0/3933284232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fe87c02cc30 con 0x7fe87c0fed20 2024-04-27T01:58:10.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.269+0000 7fe873fff700 1 -- 172.21.15.29:0/3933284232 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fe86c025360 con 0x7fe87c0fed20 2024-04-27T01:58:10.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.272+0000 7fe8810bf700 1 -- 172.21.15.29:0/3933284232 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fe868038530 msgr2=0x7fe86803a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:10.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.272+0000 7fe8810bf700 1 --2- 172.21.15.29:0/3933284232 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fe868038530 0x7fe86803a9e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe864006fd0 tx=0x7fe864006e40 comp rx=0 tx=0).stop 2024-04-27T01:58:10.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.272+0000 7fe8810bf700 1 -- 172.21.15.29:0/3933284232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe87c0fed20 msgr2=0x7fe87c194fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:10.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.272+0000 7fe8810bf700 1 --2- 172.21.15.29:0/3933284232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe87c0fed20 0x7fe87c194fb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe86c009fd0 tx=0x7fe86c00bb10 comp rx=0 tx=0).stop 2024-04-27T01:58:10.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.272+0000 7fe8810bf700 1 -- 172.21.15.29:0/3933284232 shutdown_connections 2024-04-27T01:58:10.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.272+0000 7fe8810bf700 1 --2- 172.21.15.29:0/3933284232 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fe868038530 0x7fe86803a9e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:10.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.272+0000 7fe8810bf700 1 --2- 172.21.15.29:0/3933284232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe87c0fed20 0x7fe87c194fb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:10.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.272+0000 7fe8810bf700 1 -- 172.21.15.29:0/3933284232 >> 172.21.15.29:0/3933284232 conn(0x7fe87c0fa2b0 msgr2=0x7fe87c0690b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:10.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.273+0000 7fe8810bf700 1 -- 172.21.15.29:0/3933284232 shutdown_connections 2024-04-27T01:58:10.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:10.273+0000 7fe8810bf700 1 -- 172.21.15.29:0/3933284232 wait complete. 2024-04-27T01:58:11.001 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-27T01:58:11.002 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-27T01:58:11.313 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T01:58:11.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:11 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/3933284232' entity='client.admin' 2024-04-27T01:58:13.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:12 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:13.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:12 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:13.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:12 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd/host:smithi029", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:58:13.032 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:12 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:13.032 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:12 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi029", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-27T01:58:13.032 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:12 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi029", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-27T01:58:13.032 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:12 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:58:13.032 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:12 smithi029 ceph-mon[26769]: Deploying daemon ceph-exporter.smithi029 on smithi029 2024-04-27T01:58:13.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.374+0000 7f68bd5c5700 1 -- 172.21.15.29:0/2919910666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b80fe0e0 msgr2=0x7f68b80fe4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:13.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.374+0000 7f68bd5c5700 1 --2- 172.21.15.29:0/2919910666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b80fe0e0 0x7f68b80fe4f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f68a800b0a0 tx=0x7f68a800b3b0 comp rx=0 tx=0).stop 2024-04-27T01:58:13.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.375+0000 7f68bd5c5700 1 -- 172.21.15.29:0/2919910666 shutdown_connections 2024-04-27T01:58:13.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.375+0000 7f68bd5c5700 1 --2- 172.21.15.29:0/2919910666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b80fe0e0 0x7f68b80fe4f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:13.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.375+0000 7f68bd5c5700 1 -- 172.21.15.29:0/2919910666 >> 172.21.15.29:0/2919910666 conn(0x7f68b80f9a50 msgr2=0x7f68b80fbe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:13.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.375+0000 7f68bd5c5700 1 -- 172.21.15.29:0/2919910666 shutdown_connections 2024-04-27T01:58:13.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.375+0000 7f68bd5c5700 1 -- 172.21.15.29:0/2919910666 wait complete. 2024-04-27T01:58:13.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.376+0000 7f68bd5c5700 1 Processor -- start 2024-04-27T01:58:13.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.376+0000 7f68bd5c5700 1 -- start start 2024-04-27T01:58:13.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.376+0000 7f68bd5c5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b80fe0e0 0x7f68b8194da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:13.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.376+0000 7f68bd5c5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b81952e0 con 0x7f68b80fe0e0 2024-04-27T01:58:13.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.376+0000 7f68b6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b80fe0e0 0x7f68b8194da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:13.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.376+0000 7f68b6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b80fe0e0 0x7f68b8194da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40068/0 (socket says 172.21.15.29:40068) 2024-04-27T01:58:13.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.376+0000 7f68b6ffd700 1 -- 172.21.15.29:0/1114117693 learned_addr learned my addr 172.21.15.29:0/1114117693 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:58:13.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.377+0000 7f68b6ffd700 1 -- 172.21.15.29:0/1114117693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a8009d00 con 0x7f68b80fe0e0 2024-04-27T01:58:13.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.377+0000 7f68b6ffd700 1 --2- 172.21.15.29:0/1114117693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b80fe0e0 0x7f68b8194da0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f68a800b070 tx=0x7f68a8003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:13.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.377+0000 7f689ffff700 1 -- 172.21.15.29:0/1114117693 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68a800e040 con 0x7f68b80fe0e0 2024-04-27T01:58:13.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.377+0000 7f689ffff700 1 -- 172.21.15.29:0/1114117693 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f68a8004700 con 0x7f68b80fe0e0 2024-04-27T01:58:13.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.377+0000 7f68bd5c5700 1 -- 172.21.15.29:0/1114117693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b81954e0 con 0x7f68b80fe0e0 2024-04-27T01:58:13.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.377+0000 7f689ffff700 1 -- 172.21.15.29:0/1114117693 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68a801db90 con 0x7f68b80fe0e0 2024-04-27T01:58:13.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.377+0000 7f68bd5c5700 1 -- 172.21.15.29:0/1114117693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b8195960 con 0x7f68b80fe0e0 2024-04-27T01:58:13.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.378+0000 7f689ffff700 1 -- 172.21.15.29:0/1114117693 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f68a8012430 con 0x7f68b80fe0e0 2024-04-27T01:58:13.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.378+0000 7f689ffff700 1 --2- 172.21.15.29:0/1114117693 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f68a00384e0 0x7f68a003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:13.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.378+0000 7f689ffff700 1 -- 172.21.15.29:0/1114117693 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f68a804d570 con 0x7f68b80fe0e0 2024-04-27T01:58:13.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.378+0000 7f68b67fc700 1 --2- 172.21.15.29:0/1114117693 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f68a00384e0 0x7f68a003a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:13.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.378+0000 7f68bd5c5700 1 -- 172.21.15.29:0/1114117693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b80623f0 con 0x7f68b80fe0e0 2024-04-27T01:58:13.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.379+0000 7f68b67fc700 1 --2- 172.21.15.29:0/1114117693 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f68a00384e0 0x7f68a003a990 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f68ac006fd0 tx=0x7f68ac006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:13.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.383+0000 7f689ffff700 1 -- 172.21.15.29:0/1114117693 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f68b80623f0 con 0x7f68b80fe0e0 2024-04-27T01:58:13.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.591+0000 7f68bd5c5700 1 -- 172.21.15.29:0/1114117693 --> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f68b80623f0 con 0x7f68a00384e0 2024-04-27T01:58:13.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.594+0000 7f689ffff700 1 -- 172.21.15.29:0/1114117693 <== mgr.14164 v2:172.21.15.29:6800/2469926221 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f68b80623f0 con 0x7f68a00384e0 2024-04-27T01:58:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.596+0000 7f68bd5c5700 1 -- 172.21.15.29:0/1114117693 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f68a00384e0 msgr2=0x7f68a003a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.596+0000 7f68bd5c5700 1 --2- 172.21.15.29:0/1114117693 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f68a00384e0 0x7f68a003a990 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f68ac006fd0 tx=0x7f68ac006e40 comp rx=0 tx=0).stop 2024-04-27T01:58:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.596+0000 7f68bd5c5700 1 -- 172.21.15.29:0/1114117693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b80fe0e0 msgr2=0x7f68b8194da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.596+0000 7f68bd5c5700 1 --2- 172.21.15.29:0/1114117693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b80fe0e0 0x7f68b8194da0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f68a800b070 tx=0x7f68a8003ce0 comp rx=0 tx=0).stop 2024-04-27T01:58:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.596+0000 7f68bd5c5700 1 -- 172.21.15.29:0/1114117693 shutdown_connections 2024-04-27T01:58:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.596+0000 7f68bd5c5700 1 --2- 172.21.15.29:0/1114117693 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f68a00384e0 0x7f68a003a990 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.596+0000 7f68bd5c5700 1 --2- 172.21.15.29:0/1114117693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b80fe0e0 0x7f68b8194da0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.597+0000 7f68bd5c5700 1 -- 172.21.15.29:0/1114117693 >> 172.21.15.29:0/1114117693 conn(0x7f68b80f9a50 msgr2=0x7f68b80fa700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.597+0000 7f68bd5c5700 1 -- 172.21.15.29:0/1114117693 shutdown_connections 2024-04-27T01:58:13.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:13.597+0000 7f68bd5c5700 1 -- 172.21.15.29:0/1114117693 wait complete. 2024-04-27T01:58:14.664 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:14 smithi029 ceph-mon[26769]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:58:14.664 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:14 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:15.102 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi052 2024-04-27T01:58:15.102 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-27T01:58:15.102 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/etc/ceph/ceph.conf 2024-04-27T01:58:15.122 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-27T01:58:15.122 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:58:15.180 INFO:tasks.cephadm:Adding host smithi052 to orchestrator... 2024-04-27T01:58:15.180 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph orch host add smithi052 2024-04-27T01:58:15.662 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T01:58:16.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:16 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:17.188 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:16 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:17.188 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:16 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:17.188 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:16 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:17.188 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:16 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi029", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T01:58:17.188 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:16 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi029", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-27T01:58:17.188 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:16 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:58:17.188 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:16 smithi029 ceph-mon[26769]: Deploying daemon crash.smithi029 on smithi029 2024-04-27T01:58:17.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.258+0000 7f3847d7d700 1 -- 172.21.15.29:0/3779456563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3840101400 msgr2=0x7f3840101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:17.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.258+0000 7f3847d7d700 1 --2- 172.21.15.29:0/3779456563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3840101400 0x7f3840101810 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3830009910 tx=0x7f3830009c20 comp rx=0 tx=0).stop 2024-04-27T01:58:17.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.258+0000 7f3847d7d700 1 -- 172.21.15.29:0/3779456563 shutdown_connections 2024-04-27T01:58:17.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.258+0000 7f3847d7d700 1 --2- 172.21.15.29:0/3779456563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3840101400 0x7f3840101810 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:17.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.258+0000 7f3847d7d700 1 -- 172.21.15.29:0/3779456563 >> 172.21.15.29:0/3779456563 conn(0x7f38400f9a50 msgr2=0x7f38400fbe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:17.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.258+0000 7f3847d7d700 1 -- 172.21.15.29:0/3779456563 shutdown_connections 2024-04-27T01:58:17.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.258+0000 7f3847d7d700 1 -- 172.21.15.29:0/3779456563 wait complete. 2024-04-27T01:58:17.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.259+0000 7f3847d7d700 1 Processor -- start 2024-04-27T01:58:17.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.259+0000 7f3847d7d700 1 -- start start 2024-04-27T01:58:17.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.259+0000 7f3847d7d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3840101400 0x7f3840194fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:17.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.259+0000 7f3847d7d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3840195500 con 0x7f3840101400 2024-04-27T01:58:17.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.259+0000 7f3845b19700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3840101400 0x7f3840194fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:17.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.259+0000 7f3845b19700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3840101400 0x7f3840194fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56582/0 (socket says 172.21.15.29:56582) 2024-04-27T01:58:17.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.259+0000 7f3845b19700 1 -- 172.21.15.29:0/2115972791 learned_addr learned my addr 172.21.15.29:0/2115972791 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:58:17.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.259+0000 7f3845b19700 1 -- 172.21.15.29:0/2115972791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38300095f0 con 0x7f3840101400 2024-04-27T01:58:17.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.259+0000 7f3845b19700 1 --2- 172.21.15.29:0/2115972791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3840101400 0x7f3840194fc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3830005950 tx=0x7f3830009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:17.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.259+0000 7f383effd700 1 -- 172.21.15.29:0/2115972791 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f383001d070 con 0x7f3840101400 2024-04-27T01:58:17.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.260+0000 7f3847d7d700 1 -- 172.21.15.29:0/2115972791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3840195700 con 0x7f3840101400 2024-04-27T01:58:17.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.260+0000 7f383effd700 1 -- 172.21.15.29:0/2115972791 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f383000be10 con 0x7f3840101400 2024-04-27T01:58:17.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.260+0000 7f383effd700 1 -- 172.21.15.29:0/2115972791 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f383000f460 con 0x7f3840101400 2024-04-27T01:58:17.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.260+0000 7f3847d7d700 1 -- 172.21.15.29:0/2115972791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3840195ba0 con 0x7f3840101400 2024-04-27T01:58:17.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.260+0000 7f383effd700 1 -- 172.21.15.29:0/2115972791 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f38300227c0 con 0x7f3840101400 2024-04-27T01:58:17.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.260+0000 7f3847d7d700 1 -- 172.21.15.29:0/2115972791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f384018eb00 con 0x7f3840101400 2024-04-27T01:58:17.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.261+0000 7f383effd700 1 --2- 172.21.15.29:0/2115972791 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f3828038490 0x7f382803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:17.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.261+0000 7f383effd700 1 -- 172.21.15.29:0/2115972791 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f383004c430 con 0x7f3840101400 2024-04-27T01:58:17.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.261+0000 7f3845318700 1 --2- 172.21.15.29:0/2115972791 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f3828038490 0x7f382803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:17.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.261+0000 7f3845318700 1 --2- 172.21.15.29:0/2115972791 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f3828038490 0x7f382803a940 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3834006fd0 tx=0x7f3834006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:17.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.265+0000 7f383effd700 1 -- 172.21.15.29:0/2115972791 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f383002bba0 con 0x7f3840101400 2024-04-27T01:58:17.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:17.476+0000 7f3847d7d700 1 -- 172.21.15.29:0/2115972791 --> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi052", "target": ["mon-mgr", ""]}) v1 -- 0x7f3840061190 con 0x7f3828038490 2024-04-27T01:58:18.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:18 smithi029 ceph-mon[26769]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi052", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:58:18.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:18 smithi029 ceph-mon[26769]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:58:18.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:18 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:19.933 INFO:teuthology.orchestra.run.smithi029.stdout:Added host 'smithi052' with addr '172.21.15.52' 2024-04-27T01:58:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:19.950+0000 7f383effd700 1 -- 172.21.15.29:0/2115972791 <== mgr.14164 v2:172.21.15.29:6800/2469926221 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f3840061190 con 0x7f3828038490 2024-04-27T01:58:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:19.953+0000 7f3847d7d700 1 -- 172.21.15.29:0/2115972791 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f3828038490 msgr2=0x7f382803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:19.953+0000 7f3847d7d700 1 --2- 172.21.15.29:0/2115972791 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f3828038490 0x7f382803a940 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3834006fd0 tx=0x7f3834006e40 comp rx=0 tx=0).stop 2024-04-27T01:58:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:19.953+0000 7f3847d7d700 1 -- 172.21.15.29:0/2115972791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3840101400 msgr2=0x7f3840194fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:19.953+0000 7f3847d7d700 1 --2- 172.21.15.29:0/2115972791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3840101400 0x7f3840194fc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3830005950 tx=0x7f3830009e60 comp rx=0 tx=0).stop 2024-04-27T01:58:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:19.953+0000 7f3847d7d700 1 -- 172.21.15.29:0/2115972791 shutdown_connections 2024-04-27T01:58:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:19.953+0000 7f3847d7d700 1 --2- 172.21.15.29:0/2115972791 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f3828038490 0x7f382803a940 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:19.953+0000 7f3847d7d700 1 --2- 172.21.15.29:0/2115972791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3840101400 0x7f3840194fc0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:19.953+0000 7f3847d7d700 1 -- 172.21.15.29:0/2115972791 >> 172.21.15.29:0/2115972791 conn(0x7f38400f9a50 msgr2=0x7f38400fa700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:19.954+0000 7f3847d7d700 1 -- 172.21.15.29:0/2115972791 shutdown_connections 2024-04-27T01:58:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:19.954+0000 7f3847d7d700 1 -- 172.21.15.29:0/2115972791 wait complete. 2024-04-27T01:58:20.081 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:19 smithi029 ceph-mon[26769]: Deploying cephadm binary to smithi052 2024-04-27T01:58:20.082 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:19 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:20.082 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:19 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:20.082 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:19 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:20.082 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:19 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:20.705 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-27T01:58:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T01:58:20.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:20 smithi029 ceph-mon[26769]: Deploying daemon node-exporter.smithi029 on smithi029 2024-04-27T01:58:20.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:20 smithi029 ceph-mon[26769]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:58:20.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:20 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:22.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:21 smithi029 ceph-mon[26769]: Added host smithi052 2024-04-27T01:58:23.118 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:22 smithi029 ceph-mon[26769]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:58:23.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.696+0000 7fca3bebc700 1 -- 172.21.15.29:0/3377575113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca340fe6a0 msgr2=0x7fca34100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:23.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.696+0000 7fca3bebc700 1 --2- 172.21.15.29:0/3377575113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca340fe6a0 0x7fca34100ac0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fca30009910 tx=0x7fca30009c20 comp rx=0 tx=0).stop 2024-04-27T01:58:23.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.696+0000 7fca3bebc700 1 -- 172.21.15.29:0/3377575113 shutdown_connections 2024-04-27T01:58:23.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.696+0000 7fca3bebc700 1 --2- 172.21.15.29:0/3377575113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca340fe6a0 0x7fca34100ac0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:23.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.696+0000 7fca3bebc700 1 -- 172.21.15.29:0/3377575113 >> 172.21.15.29:0/3377575113 conn(0x7fca340fa2b0 msgr2=0x7fca340fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:23.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.696+0000 7fca3bebc700 1 -- 172.21.15.29:0/3377575113 shutdown_connections 2024-04-27T01:58:23.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.696+0000 7fca3bebc700 1 -- 172.21.15.29:0/3377575113 wait complete. 2024-04-27T01:58:23.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.696+0000 7fca3bebc700 1 Processor -- start 2024-04-27T01:58:23.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.696+0000 7fca3bebc700 1 -- start start 2024-04-27T01:58:23.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.696+0000 7fca3bebc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca340fe6a0 0x7fca3419b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:23.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.696+0000 7fca3bebc700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3419baa0 con 0x7fca340fe6a0 2024-04-27T01:58:23.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.697+0000 7fca39c58700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca340fe6a0 0x7fca3419b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:23.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.697+0000 7fca39c58700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca340fe6a0 0x7fca3419b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56602/0 (socket says 172.21.15.29:56602) 2024-04-27T01:58:23.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.697+0000 7fca39c58700 1 -- 172.21.15.29:0/2495828967 learned_addr learned my addr 172.21.15.29:0/2495828967 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:58:23.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.697+0000 7fca39c58700 1 -- 172.21.15.29:0/2495828967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca300095f0 con 0x7fca340fe6a0 2024-04-27T01:58:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.697+0000 7fca39c58700 1 --2- 172.21.15.29:0/2495828967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca340fe6a0 0x7fca3419b560 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fca30005b40 tx=0x7fca30004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.697+0000 7fca2affd700 1 -- 172.21.15.29:0/2495828967 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca3001d070 con 0x7fca340fe6a0 2024-04-27T01:58:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.697+0000 7fca2affd700 1 -- 172.21.15.29:0/2495828967 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fca3000be10 con 0x7fca340fe6a0 2024-04-27T01:58:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.697+0000 7fca3bebc700 1 -- 172.21.15.29:0/2495828967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca3419bca0 con 0x7fca340fe6a0 2024-04-27T01:58:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.697+0000 7fca2affd700 1 -- 172.21.15.29:0/2495828967 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca3000f460 con 0x7fca340fe6a0 2024-04-27T01:58:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.697+0000 7fca3bebc700 1 -- 172.21.15.29:0/2495828967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca3419c140 con 0x7fca340fe6a0 2024-04-27T01:58:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.698+0000 7fca2affd700 1 -- 172.21.15.29:0/2495828967 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7fca3000f6d0 con 0x7fca340fe6a0 2024-04-27T01:58:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.698+0000 7fca2affd700 1 --2- 172.21.15.29:0/2495828967 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fca20038490 0x7fca2003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.698+0000 7fca2affd700 1 -- 172.21.15.29:0/2495828967 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fca3004c730 con 0x7fca340fe6a0 2024-04-27T01:58:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.698+0000 7fca39457700 1 --2- 172.21.15.29:0/2495828967 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fca20038490 0x7fca2003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.698+0000 7fca3bebc700 1 -- 172.21.15.29:0/2495828967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca340623f0 con 0x7fca340fe6a0 2024-04-27T01:58:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.699+0000 7fca39457700 1 --2- 172.21.15.29:0/2495828967 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fca20038490 0x7fca2003a940 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fca24006fd0 tx=0x7fca24006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.703+0000 7fca2affd700 1 -- 172.21.15.29:0/2495828967 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fca340623f0 con 0x7fca340fe6a0 2024-04-27T01:58:23.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.911+0000 7fca3bebc700 1 -- 172.21.15.29:0/2495828967 --> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fca34062380 con 0x7fca20038490 2024-04-27T01:58:23.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.912+0000 7fca2affd700 1 -- 172.21.15.29:0/2495828967 <== mgr.14164 v2:172.21.15.29:6800/2469926221 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7fca34062380 con 0x7fca20038490 2024-04-27T01:58:23.894 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T01:58:23.894 INFO:teuthology.orchestra.run.smithi029.stdout:[{"addr": "172.21.15.29", "hostname": "smithi029", "labels": [], "status": ""}, {"addr": "172.21.15.52", "hostname": "smithi052", "labels": [], "status": ""}] 2024-04-27T01:58:23.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.915+0000 7fca3bebc700 1 -- 172.21.15.29:0/2495828967 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fca20038490 msgr2=0x7fca2003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:23.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.915+0000 7fca3bebc700 1 --2- 172.21.15.29:0/2495828967 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fca20038490 0x7fca2003a940 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fca24006fd0 tx=0x7fca24006e40 comp rx=0 tx=0).stop 2024-04-27T01:58:23.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.915+0000 7fca3bebc700 1 -- 172.21.15.29:0/2495828967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca340fe6a0 msgr2=0x7fca3419b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:23.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.915+0000 7fca3bebc700 1 --2- 172.21.15.29:0/2495828967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca340fe6a0 0x7fca3419b560 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fca30005b40 tx=0x7fca30004740 comp rx=0 tx=0).stop 2024-04-27T01:58:23.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.915+0000 7fca3bebc700 1 -- 172.21.15.29:0/2495828967 shutdown_connections 2024-04-27T01:58:23.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.915+0000 7fca3bebc700 1 --2- 172.21.15.29:0/2495828967 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fca20038490 0x7fca2003a940 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:23.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.915+0000 7fca3bebc700 1 --2- 172.21.15.29:0/2495828967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca340fe6a0 0x7fca3419b560 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:23.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.915+0000 7fca3bebc700 1 -- 172.21.15.29:0/2495828967 >> 172.21.15.29:0/2495828967 conn(0x7fca340fa2b0 msgr2=0x7fca340faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:23.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.915+0000 7fca3bebc700 1 -- 172.21.15.29:0/2495828967 shutdown_connections 2024-04-27T01:58:23.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:23.915+0000 7fca3bebc700 1 -- 172.21.15.29:0/2495828967 wait complete. 2024-04-27T01:58:23.936 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:23 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:24.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:24 smithi029 ceph-mon[26769]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:58:24.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:24 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:24.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:24 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:24.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:24 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:24.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:24 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:25.034 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-27T01:58:25.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-27T01:58:25.251 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T01:58:25.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:25 smithi029 ceph-mon[26769]: from='client.14193 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:58:25.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:25 smithi029 ceph-mon[26769]: Deploying daemon alertmanager.smithi029 on smithi029 2024-04-27T01:58:27.045 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:26 smithi029 ceph-mon[26769]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:58:27.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.762+0000 7ff8fd6c6700 1 -- 172.21.15.29:0/3918640519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80ffe20 msgr2=0x7ff8f8100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:27.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.762+0000 7ff8fd6c6700 1 --2- 172.21.15.29:0/3918640519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80ffe20 0x7ff8f8100230 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec009910 tx=0x7ff8ec009c20 comp rx=0 tx=0).stop 2024-04-27T01:58:27.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.762+0000 7ff8fd6c6700 1 -- 172.21.15.29:0/3918640519 shutdown_connections 2024-04-27T01:58:27.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.762+0000 7ff8fd6c6700 1 --2- 172.21.15.29:0/3918640519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80ffe20 0x7ff8f8100230 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:27.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.762+0000 7ff8fd6c6700 1 -- 172.21.15.29:0/3918640519 >> 172.21.15.29:0/3918640519 conn(0x7ff8f80fb3d0 msgr2=0x7ff8f80fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:27.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.762+0000 7ff8fd6c6700 1 -- 172.21.15.29:0/3918640519 shutdown_connections 2024-04-27T01:58:27.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.762+0000 7ff8fd6c6700 1 -- 172.21.15.29:0/3918640519 wait complete. 2024-04-27T01:58:27.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.763+0000 7ff8fd6c6700 1 Processor -- start 2024-04-27T01:58:27.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.763+0000 7ff8fd6c6700 1 -- start start 2024-04-27T01:58:27.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.763+0000 7ff8fd6c6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80ffe20 0x7ff8f8074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:27.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.763+0000 7ff8fd6c6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f8075030 con 0x7ff8f80ffe20 2024-04-27T01:58:27.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.763+0000 7ff8f6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80ffe20 0x7ff8f8074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:27.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.763+0000 7ff8f6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80ffe20 0x7ff8f8074af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38514/0 (socket says 172.21.15.29:38514) 2024-04-27T01:58:27.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.763+0000 7ff8f6ffd700 1 -- 172.21.15.29:0/841545795 learned_addr learned my addr 172.21.15.29:0/841545795 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:58:27.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.764+0000 7ff8f6ffd700 1 -- 172.21.15.29:0/841545795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ec0095f0 con 0x7ff8f80ffe20 2024-04-27T01:58:27.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.764+0000 7ff8f6ffd700 1 --2- 172.21.15.29:0/841545795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80ffe20 0x7ff8f8074af0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec0092e0 tx=0x7ff8ec009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:27.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.764+0000 7ff8dffff700 1 -- 172.21.15.29:0/841545795 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8ec01d070 con 0x7ff8f80ffe20 2024-04-27T01:58:27.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.764+0000 7ff8fd6c6700 1 -- 172.21.15.29:0/841545795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f8073140 con 0x7ff8f80ffe20 2024-04-27T01:58:27.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.764+0000 7ff8dffff700 1 -- 172.21.15.29:0/841545795 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff8ec00be10 con 0x7ff8f80ffe20 2024-04-27T01:58:27.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.764+0000 7ff8dffff700 1 -- 172.21.15.29:0/841545795 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8ec00f460 con 0x7ff8f80ffe20 2024-04-27T01:58:27.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.764+0000 7ff8fd6c6700 1 -- 172.21.15.29:0/841545795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f80735e0 con 0x7ff8f80ffe20 2024-04-27T01:58:27.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.765+0000 7ff8dffff700 1 -- 172.21.15.29:0/841545795 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7ff8ec00f6d0 con 0x7ff8f80ffe20 2024-04-27T01:58:27.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.765+0000 7ff8fd6c6700 1 -- 172.21.15.29:0/841545795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f8062380 con 0x7ff8f80ffe20 2024-04-27T01:58:27.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.765+0000 7ff8dffff700 1 --2- 172.21.15.29:0/841545795 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7ff8e0038490 0x7ff8e003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:27.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.765+0000 7ff8dffff700 1 -- 172.21.15.29:0/841545795 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff8ec04c300 con 0x7ff8f80ffe20 2024-04-27T01:58:27.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.766+0000 7ff8f67fc700 1 --2- 172.21.15.29:0/841545795 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7ff8e0038490 0x7ff8e003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:27.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.766+0000 7ff8f67fc700 1 --2- 172.21.15.29:0/841545795 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7ff8e0038490 0x7ff8e003a940 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8006fd0 tx=0x7ff8e8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:27.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:27.770+0000 7ff8dffff700 1 -- 172.21.15.29:0/841545795 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff8ec020960 con 0x7ff8f80ffe20 2024-04-27T01:58:27.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:28.003+0000 7ff8fd6c6700 1 -- 172.21.15.29:0/841545795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7ff8f818e460 con 0x7ff8f80ffe20 2024-04-27T01:58:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:28.845+0000 7ff8dffff700 1 -- 172.21.15.29:0/841545795 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7ff8ec027030 con 0x7ff8f80ffe20 2024-04-27T01:58:28.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:28.847+0000 7ff8fd6c6700 1 -- 172.21.15.29:0/841545795 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7ff8e0038490 msgr2=0x7ff8e003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:28.847+0000 7ff8fd6c6700 1 --2- 172.21.15.29:0/841545795 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7ff8e0038490 0x7ff8e003a940 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8006fd0 tx=0x7ff8e8006e40 comp rx=0 tx=0).stop 2024-04-27T01:58:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:28.847+0000 7ff8fd6c6700 1 -- 172.21.15.29:0/841545795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80ffe20 msgr2=0x7ff8f8074af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:28.847+0000 7ff8fd6c6700 1 --2- 172.21.15.29:0/841545795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80ffe20 0x7ff8f8074af0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec0092e0 tx=0x7ff8ec009e60 comp rx=0 tx=0).stop 2024-04-27T01:58:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:28.847+0000 7ff8fd6c6700 1 -- 172.21.15.29:0/841545795 shutdown_connections 2024-04-27T01:58:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:28.847+0000 7ff8fd6c6700 1 --2- 172.21.15.29:0/841545795 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7ff8e0038490 0x7ff8e003a940 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:28.847+0000 7ff8fd6c6700 1 --2- 172.21.15.29:0/841545795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f80ffe20 0x7ff8f8074af0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:28.847+0000 7ff8fd6c6700 1 -- 172.21.15.29:0/841545795 >> 172.21.15.29:0/841545795 conn(0x7ff8f80fb3d0 msgr2=0x7ff8f80fc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:28.848+0000 7ff8fd6c6700 1 -- 172.21.15.29:0/841545795 shutdown_connections 2024-04-27T01:58:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:58:28.848+0000 7ff8fd6c6700 1 -- 172.21.15.29:0/841545795 wait complete. 2024-04-27T01:58:28.830 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2024-04-27T01:58:29.194 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:28 smithi029 ceph-mon[26769]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:58:29.194 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:28 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:29.194 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:28 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/841545795' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-27T01:58:29.726 INFO:tasks.cephadm:Adding mon.smithi029 on smithi029 2024-04-27T01:58:29.726 INFO:tasks.cephadm:Adding mon.smithi052 on smithi052 2024-04-27T01:58:29.727 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph orch apply mon '2;smithi029:172.21.15.29=smithi029;smithi052:172.21.15.52=smithi052' 2024-04-27T01:58:29.934 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:58:29.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:29 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/841545795' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-27T01:58:29.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:29 smithi029 ceph-mon[26769]: osdmap e4: 0 total, 0 up, 0 in 2024-04-27T01:58:29.981 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:58:31.084 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:30 smithi029 ceph-mon[26769]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:58:31.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.497+0000 7fa2fe059700 1 -- 172.21.15.52:0/655225617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2f80ffe20 msgr2=0x7fa2f8100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:31.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.497+0000 7fa2fe059700 1 --2- 172.21.15.52:0/655225617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2f80ffe20 0x7fa2f8100230 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8009910 tx=0x7fa2e8009c20 comp rx=0 tx=0).stop 2024-04-27T01:58:31.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.497+0000 7fa2fe059700 1 -- 172.21.15.52:0/655225617 shutdown_connections 2024-04-27T01:58:31.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.497+0000 7fa2fe059700 1 --2- 172.21.15.52:0/655225617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2f80ffe20 0x7fa2f8100230 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:31.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.497+0000 7fa2fe059700 1 -- 172.21.15.52:0/655225617 >> 172.21.15.52:0/655225617 conn(0x7fa2f80fb3d0 msgr2=0x7fa2f80fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:31.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.498+0000 7fa2fe059700 1 -- 172.21.15.52:0/655225617 shutdown_connections 2024-04-27T01:58:31.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.498+0000 7fa2fe059700 1 -- 172.21.15.52:0/655225617 wait complete. 2024-04-27T01:58:31.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.498+0000 7fa2fe059700 1 Processor -- start 2024-04-27T01:58:31.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.498+0000 7fa2fe059700 1 -- start start 2024-04-27T01:58:31.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.499+0000 7fa2fe059700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2f80ffe20 0x7fa2f8194f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:31.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.499+0000 7fa2fe059700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f81954a0 con 0x7fa2f80ffe20 2024-04-27T01:58:31.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.499+0000 7fa2f77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2f80ffe20 0x7fa2f8194f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:31.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.499+0000 7fa2f77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2f80ffe20 0x7fa2f8194f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.52:57916/0 (socket says 172.21.15.52:57916) 2024-04-27T01:58:31.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.499+0000 7fa2f77fe700 1 -- 172.21.15.52:0/1858110078 learned_addr learned my addr 172.21.15.52:0/1858110078 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T01:58:31.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.499+0000 7fa2f77fe700 1 -- 172.21.15.52:0/1858110078 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e80095f0 con 0x7fa2f80ffe20 2024-04-27T01:58:31.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.499+0000 7fa2f77fe700 1 --2- 172.21.15.52:0/1858110078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2f80ffe20 0x7fa2f8194f60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8005b40 tx=0x7fa2e8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:31.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.500+0000 7fa2f4ff9700 1 -- 172.21.15.52:0/1858110078 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2e801d070 con 0x7fa2f80ffe20 2024-04-27T01:58:31.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.500+0000 7fa2fe059700 1 -- 172.21.15.52:0/1858110078 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f81956a0 con 0x7fa2f80ffe20 2024-04-27T01:58:31.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.500+0000 7fa2f4ff9700 1 -- 172.21.15.52:0/1858110078 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa2e800be10 con 0x7fa2f80ffe20 2024-04-27T01:58:31.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.500+0000 7fa2f4ff9700 1 -- 172.21.15.52:0/1858110078 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2e800f460 con 0x7fa2f80ffe20 2024-04-27T01:58:31.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.500+0000 7fa2fe059700 1 -- 172.21.15.52:0/1858110078 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f8195b40 con 0x7fa2f80ffe20 2024-04-27T01:58:31.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.501+0000 7fa2f4ff9700 1 -- 172.21.15.52:0/1858110078 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7fa2e80227e0 con 0x7fa2f80ffe20 2024-04-27T01:58:31.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.501+0000 7fa2fe059700 1 -- 172.21.15.52:0/1858110078 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f818ea80 con 0x7fa2f80ffe20 2024-04-27T01:58:31.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.501+0000 7fa2f4ff9700 1 --2- 172.21.15.52:0/1858110078 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fa2e0038490 0x7fa2e003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:31.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.501+0000 7fa2f4ff9700 1 -- 172.21.15.52:0/1858110078 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa2e804c2a0 con 0x7fa2f80ffe20 2024-04-27T01:58:31.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.501+0000 7fa2f6ffd700 1 --2- 172.21.15.52:0/1858110078 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fa2e0038490 0x7fa2e003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:31.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.502+0000 7fa2f6ffd700 1 --2- 172.21.15.52:0/1858110078 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fa2e0038490 0x7fa2e003a940 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec006fd0 tx=0x7fa2ec006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:31.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.506+0000 7fa2f4ff9700 1 -- 172.21.15.52:0/1858110078 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa2e802b590 con 0x7fa2f80ffe20 2024-04-27T01:58:31.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.717+0000 7fa2fe059700 1 -- 172.21.15.52:0/1858110078 --> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi029:172.21.15.29=smithi029;smithi052:172.21.15.52=smithi052", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2f8061190 con 0x7fa2e0038490 2024-04-27T01:58:31.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.724+0000 7fa2f4ff9700 1 -- 172.21.15.52:0/1858110078 <== mgr.14164 v2:172.21.15.29:6800/2469926221 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa2f8061190 con 0x7fa2e0038490 2024-04-27T01:58:31.705 INFO:teuthology.orchestra.run.smithi052.stdout:Scheduled mon update... 2024-04-27T01:58:31.710 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.726+0000 7fa2fe059700 1 -- 172.21.15.52:0/1858110078 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fa2e0038490 msgr2=0x7fa2e003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:31.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.726+0000 7fa2fe059700 1 --2- 172.21.15.52:0/1858110078 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fa2e0038490 0x7fa2e003a940 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec006fd0 tx=0x7fa2ec006e40 comp rx=0 tx=0).stop 2024-04-27T01:58:31.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.726+0000 7fa2fe059700 1 -- 172.21.15.52:0/1858110078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2f80ffe20 msgr2=0x7fa2f8194f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:31.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.726+0000 7fa2fe059700 1 --2- 172.21.15.52:0/1858110078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2f80ffe20 0x7fa2f8194f60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8005b40 tx=0x7fa2e8004740 comp rx=0 tx=0).stop 2024-04-27T01:58:31.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.726+0000 7fa2fe059700 1 -- 172.21.15.52:0/1858110078 shutdown_connections 2024-04-27T01:58:31.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.726+0000 7fa2fe059700 1 --2- 172.21.15.52:0/1858110078 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fa2e0038490 0x7fa2e003a940 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:31.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.726+0000 7fa2fe059700 1 --2- 172.21.15.52:0/1858110078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2f80ffe20 0x7fa2f8194f60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:31.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.726+0000 7fa2fe059700 1 -- 172.21.15.52:0/1858110078 >> 172.21.15.52:0/1858110078 conn(0x7fa2f80fb3d0 msgr2=0x7fa2f80fc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:31.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.727+0000 7fa2fe059700 1 -- 172.21.15.52:0/1858110078 shutdown_connections 2024-04-27T01:58:31.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:31.727+0000 7fa2fe059700 1 -- 172.21.15.52:0/1858110078 wait complete. 2024-04-27T01:58:32.278 DEBUG:teuthology.orchestra.run.smithi052:mon.smithi052> sudo journalctl -f -n 0 -u ceph-4c587076-0439-11ef-bc93-c7b262605968@mon.smithi052.service 2024-04-27T01:58:32.280 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:58:32.280 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:58:32.512 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:58:32.560 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:58:32.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:32 smithi029 ceph-mon[26769]: from='client.14197 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi029:172.21.15.29=smithi029;smithi052:172.21.15.52=smithi052", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:58:32.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:32 smithi029 ceph-mon[26769]: Saving service mon spec with placement smithi029:172.21.15.29=smithi029;smithi052:172.21.15.52=smithi052;count:2 2024-04-27T01:58:32.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:32 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:32.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:32 smithi029 ceph-mon[26769]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:58:33.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.008+0000 7fbb1279d700 1 -- 172.21.15.52:0/4003648006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb0c0fe570 msgr2=0x7fbb0c0fe980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:33.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.008+0000 7fbb1279d700 1 --2- 172.21.15.52:0/4003648006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb0c0fe570 0x7fbb0c0fe980 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbb08009910 tx=0x7fbb08009c20 comp rx=0 tx=0).stop 2024-04-27T01:58:33.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.008+0000 7fbb1279d700 1 -- 172.21.15.52:0/4003648006 shutdown_connections 2024-04-27T01:58:33.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.008+0000 7fbb1279d700 1 --2- 172.21.15.52:0/4003648006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb0c0fe570 0x7fbb0c0fe980 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:33.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.008+0000 7fbb1279d700 1 -- 172.21.15.52:0/4003648006 >> 172.21.15.52:0/4003648006 conn(0x7fbb0c0fa1a0 msgr2=0x7fbb0c0fc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:33.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.008+0000 7fbb1279d700 1 -- 172.21.15.52:0/4003648006 shutdown_connections 2024-04-27T01:58:33.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.008+0000 7fbb1279d700 1 -- 172.21.15.52:0/4003648006 wait complete. 2024-04-27T01:58:33.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.009+0000 7fbb1279d700 1 Processor -- start 2024-04-27T01:58:33.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.009+0000 7fbb1279d700 1 -- start start 2024-04-27T01:58:33.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.009+0000 7fbb1279d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb0c0fe570 0x7fbb0c196fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:33.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.009+0000 7fbb1279d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0c1974f0 con 0x7fbb0c0fe570 2024-04-27T01:58:33.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.009+0000 7fbb1179b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb0c0fe570 0x7fbb0c196fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:33.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.009+0000 7fbb1179b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb0c0fe570 0x7fbb0c196fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.52:57938/0 (socket says 172.21.15.52:57938) 2024-04-27T01:58:33.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.009+0000 7fbb1179b700 1 -- 172.21.15.52:0/467021385 learned_addr learned my addr 172.21.15.52:0/467021385 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T01:58:33.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.010+0000 7fbb1179b700 1 -- 172.21.15.52:0/467021385 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb080095f0 con 0x7fbb0c0fe570 2024-04-27T01:58:33.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.010+0000 7fbb1179b700 1 --2- 172.21.15.52:0/467021385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb0c0fe570 0x7fbb0c196fb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbb08005b40 tx=0x7fbb08004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:33.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.010+0000 7fbb027fc700 1 -- 172.21.15.52:0/467021385 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb0801d070 con 0x7fbb0c0fe570 2024-04-27T01:58:33.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.010+0000 7fbb027fc700 1 -- 172.21.15.52:0/467021385 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbb0800be10 con 0x7fbb0c0fe570 2024-04-27T01:58:33.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.010+0000 7fbb1279d700 1 -- 172.21.15.52:0/467021385 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb0c1976f0 con 0x7fbb0c0fe570 2024-04-27T01:58:33.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.010+0000 7fbb027fc700 1 -- 172.21.15.52:0/467021385 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb0800f460 con 0x7fbb0c0fe570 2024-04-27T01:58:33.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.010+0000 7fbb1279d700 1 -- 172.21.15.52:0/467021385 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb0c197b90 con 0x7fbb0c0fe570 2024-04-27T01:58:33.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.011+0000 7fbb027fc700 1 -- 172.21.15.52:0/467021385 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7fbb0800f6d0 con 0x7fbb0c0fe570 2024-04-27T01:58:33.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.011+0000 7fbb1279d700 1 -- 172.21.15.52:0/467021385 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb0c190aa0 con 0x7fbb0c0fe570 2024-04-27T01:58:33.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.011+0000 7fbb027fc700 1 --2- 172.21.15.52:0/467021385 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fbaf8038490 0x7fbaf803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:33.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.011+0000 7fbb027fc700 1 -- 172.21.15.52:0/467021385 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbb0804c640 con 0x7fbb0c0fe570 2024-04-27T01:58:33.994 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.011+0000 7fbb10f9a700 1 --2- 172.21.15.52:0/467021385 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fbaf8038490 0x7fbaf803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:33.994 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.012+0000 7fbb10f9a700 1 --2- 172.21.15.52:0/467021385 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fbaf8038490 0x7fbaf803a940 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbafc006fd0 tx=0x7fbafc006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:33.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.016+0000 7fbb027fc700 1 -- 172.21.15.52:0/467021385 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fbb0802a360 con 0x7fbb0c0fe570 2024-04-27T01:58:34.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.303+0000 7fbb1279d700 1 -- 172.21.15.52:0/467021385 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb0c02cc30 con 0x7fbb0c0fe570 2024-04-27T01:58:34.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.303+0000 7fbb027fc700 1 -- 172.21.15.52:0/467021385 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fbb08027030 con 0x7fbb0c0fe570 2024-04-27T01:58:34.285 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:58:34.285 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":1,"fsid":"4c587076-0439-11ef-bc93-c7b262605968","modified":"2024-04-27T01:56:49.232926Z","created":"2024-04-27T01:56:49.232926Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:58:34.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.305+0000 7fbb1279d700 1 -- 172.21.15.52:0/467021385 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fbaf8038490 msgr2=0x7fbaf803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:34.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.305+0000 7fbb1279d700 1 --2- 172.21.15.52:0/467021385 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fbaf8038490 0x7fbaf803a940 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbafc006fd0 tx=0x7fbafc006e40 comp rx=0 tx=0).stop 2024-04-27T01:58:34.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.305+0000 7fbb1279d700 1 -- 172.21.15.52:0/467021385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb0c0fe570 msgr2=0x7fbb0c196fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:34.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.305+0000 7fbb1279d700 1 --2- 172.21.15.52:0/467021385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb0c0fe570 0x7fbb0c196fb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbb08005b40 tx=0x7fbb08004740 comp rx=0 tx=0).stop 2024-04-27T01:58:34.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.305+0000 7fbb1279d700 1 -- 172.21.15.52:0/467021385 shutdown_connections 2024-04-27T01:58:34.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.305+0000 7fbb1279d700 1 --2- 172.21.15.52:0/467021385 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fbaf8038490 0x7fbaf803a940 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:34.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.305+0000 7fbb1279d700 1 --2- 172.21.15.52:0/467021385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb0c0fe570 0x7fbb0c196fb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:34.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.305+0000 7fbb1279d700 1 -- 172.21.15.52:0/467021385 >> 172.21.15.52:0/467021385 conn(0x7fbb0c0fa1a0 msgr2=0x7fbb0c0fae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:34.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.306+0000 7fbb1279d700 1 -- 172.21.15.52:0/467021385 shutdown_connections 2024-04-27T01:58:34.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:34.306+0000 7fbb1279d700 1 -- 172.21.15.52:0/467021385 wait complete. 2024-04-27T01:58:34.288 INFO:teuthology.orchestra.run.smithi052.stderr:dumped monmap epoch 1 2024-04-27T01:58:35.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:34 smithi029 ceph-mon[26769]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:58:35.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:34 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/467021385' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:58:35.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:34 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:35.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:34 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:35.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:34 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:35.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:34 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:35.203 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:34 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:35.203 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:34 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:35.203 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:34 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T01:58:35.203 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:34 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:35.848 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:58:35.849 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:58:36.056 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:58:36.105 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:58:36.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:35 smithi029 ceph-mon[26769]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-27T01:58:36.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:35 smithi029 ceph-mon[26769]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T01:58:36.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:35 smithi029 ceph-mon[26769]: Deploying daemon grafana.smithi029 on smithi029 2024-04-27T01:58:37.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:36 smithi029 ceph-mon[26769]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:58:37.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.555+0000 7f316546f700 1 -- 172.21.15.52:0/4233542867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31600fed10 msgr2=0x7f31600ff120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:37.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.555+0000 7f316546f700 1 --2- 172.21.15.52:0/4233542867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31600fed10 0x7f31600ff120 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3150009910 tx=0x7f3150009c20 comp rx=0 tx=0).stop 2024-04-27T01:58:37.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.555+0000 7f316546f700 1 -- 172.21.15.52:0/4233542867 shutdown_connections 2024-04-27T01:58:37.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.555+0000 7f316546f700 1 --2- 172.21.15.52:0/4233542867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31600fed10 0x7f31600ff120 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:37.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.555+0000 7f316546f700 1 -- 172.21.15.52:0/4233542867 >> 172.21.15.52:0/4233542867 conn(0x7f31600fa380 msgr2=0x7f31600fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:37.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.556+0000 7f316546f700 1 -- 172.21.15.52:0/4233542867 shutdown_connections 2024-04-27T01:58:37.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.556+0000 7f316546f700 1 -- 172.21.15.52:0/4233542867 wait complete. 2024-04-27T01:58:37.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.556+0000 7f316546f700 1 Processor -- start 2024-04-27T01:58:37.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.556+0000 7f316546f700 1 -- start start 2024-04-27T01:58:37.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.556+0000 7f316546f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31600fed10 0x7f31600fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:37.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.556+0000 7f316546f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31600fc800 con 0x7f31600fed10 2024-04-27T01:58:37.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.557+0000 7f315effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31600fed10 0x7f31600fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:37.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.557+0000 7f315effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31600fed10 0x7f31600fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.52:46098/0 (socket says 172.21.15.52:46098) 2024-04-27T01:58:37.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.557+0000 7f315effd700 1 -- 172.21.15.52:0/4201196459 learned_addr learned my addr 172.21.15.52:0/4201196459 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T01:58:37.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.557+0000 7f315effd700 1 -- 172.21.15.52:0/4201196459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31500095f0 con 0x7f31600fed10 2024-04-27T01:58:37.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.557+0000 7f315effd700 1 --2- 172.21.15.52:0/4201196459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31600fed10 0x7f31600fc2c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3150005b40 tx=0x7f3150004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:37.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.557+0000 7f3147fff700 1 -- 172.21.15.52:0/4201196459 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f315001d070 con 0x7f31600fed10 2024-04-27T01:58:37.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.557+0000 7f3147fff700 1 -- 172.21.15.52:0/4201196459 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f315000be10 con 0x7f31600fed10 2024-04-27T01:58:37.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.557+0000 7f3147fff700 1 -- 172.21.15.52:0/4201196459 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f315000f460 con 0x7f31600fed10 2024-04-27T01:58:37.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.557+0000 7f316546f700 1 -- 172.21.15.52:0/4201196459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31600fa910 con 0x7f31600fed10 2024-04-27T01:58:37.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.557+0000 7f316546f700 1 -- 172.21.15.52:0/4201196459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31600fadb0 con 0x7f31600fed10 2024-04-27T01:58:37.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.558+0000 7f316546f700 1 -- 172.21.15.52:0/4201196459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3160062380 con 0x7f31600fed10 2024-04-27T01:58:37.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.558+0000 7f3147fff700 1 -- 172.21.15.52:0/4201196459 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f31500227e0 con 0x7f31600fed10 2024-04-27T01:58:37.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.559+0000 7f3147fff700 1 --2- 172.21.15.52:0/4201196459 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f31480384e0 0x7f314803a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:37.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.559+0000 7f3147fff700 1 -- 172.21.15.52:0/4201196459 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f315004c450 con 0x7f31600fed10 2024-04-27T01:58:37.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.559+0000 7f315e7fc700 1 --2- 172.21.15.52:0/4201196459 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f31480384e0 0x7f314803a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:37.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.559+0000 7f315e7fc700 1 --2- 172.21.15.52:0/4201196459 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f31480384e0 0x7f314803a990 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3154006fd0 tx=0x7f3154006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:37.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.563+0000 7f3147fff700 1 -- 172.21.15.52:0/4201196459 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f315002ba00 con 0x7f31600fed10 2024-04-27T01:58:37.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.851+0000 7f316546f700 1 -- 172.21.15.52:0/4201196459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f316018e360 con 0x7f31600fed10 2024-04-27T01:58:37.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.851+0000 7f3147fff700 1 -- 172.21.15.52:0/4201196459 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f3150027030 con 0x7f31600fed10 2024-04-27T01:58:37.835 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:58:37.835 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":1,"fsid":"4c587076-0439-11ef-bc93-c7b262605968","modified":"2024-04-27T01:56:49.232926Z","created":"2024-04-27T01:56:49.232926Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:58:37.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.853+0000 7f316546f700 1 -- 172.21.15.52:0/4201196459 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f31480384e0 msgr2=0x7f314803a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:37.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.853+0000 7f316546f700 1 --2- 172.21.15.52:0/4201196459 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f31480384e0 0x7f314803a990 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3154006fd0 tx=0x7f3154006e40 comp rx=0 tx=0).stop 2024-04-27T01:58:37.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.853+0000 7f316546f700 1 -- 172.21.15.52:0/4201196459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31600fed10 msgr2=0x7f31600fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:37.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.853+0000 7f316546f700 1 --2- 172.21.15.52:0/4201196459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31600fed10 0x7f31600fc2c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3150005b40 tx=0x7f3150004740 comp rx=0 tx=0).stop 2024-04-27T01:58:37.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.853+0000 7f316546f700 1 -- 172.21.15.52:0/4201196459 shutdown_connections 2024-04-27T01:58:37.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.853+0000 7f316546f700 1 --2- 172.21.15.52:0/4201196459 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f31480384e0 0x7f314803a990 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:37.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.853+0000 7f316546f700 1 --2- 172.21.15.52:0/4201196459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31600fed10 0x7f31600fc2c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:37.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.853+0000 7f316546f700 1 -- 172.21.15.52:0/4201196459 >> 172.21.15.52:0/4201196459 conn(0x7f31600fa380 msgr2=0x7f31600690d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:37.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.853+0000 7f316546f700 1 -- 172.21.15.52:0/4201196459 shutdown_connections 2024-04-27T01:58:37.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:37.853+0000 7f316546f700 1 -- 172.21.15.52:0/4201196459 wait complete. 2024-04-27T01:58:37.836 INFO:teuthology.orchestra.run.smithi052.stderr:dumped monmap epoch 1 2024-04-27T01:58:39.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:38 smithi029 ceph-mon[26769]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:58:39.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:38 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:39.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:38 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/4201196459' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:58:39.427 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:58:39.427 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:58:39.628 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:58:39.677 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:58:40.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:40 smithi029 ceph-mon[26769]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:58:41.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.143+0000 7fb4a7a72700 1 -- 172.21.15.52:0/481053145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4a00fed20 msgr2=0x7fb4a00ff130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:41.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.143+0000 7fb4a7a72700 1 --2- 172.21.15.52:0/481053145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4a00fed20 0x7fb4a00ff130 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00b0a0 tx=0x7fb49c00b3b0 comp rx=0 tx=0).stop 2024-04-27T01:58:41.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.143+0000 7fb4a7a72700 1 -- 172.21.15.52:0/481053145 shutdown_connections 2024-04-27T01:58:41.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.143+0000 7fb4a7a72700 1 --2- 172.21.15.52:0/481053145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4a00fed20 0x7fb4a00ff130 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:41.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.143+0000 7fb4a7a72700 1 -- 172.21.15.52:0/481053145 >> 172.21.15.52:0/481053145 conn(0x7fb4a00fa2b0 msgr2=0x7fb4a00fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:41.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.143+0000 7fb4a7a72700 1 -- 172.21.15.52:0/481053145 shutdown_connections 2024-04-27T01:58:41.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.143+0000 7fb4a7a72700 1 -- 172.21.15.52:0/481053145 wait complete. 2024-04-27T01:58:41.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.144+0000 7fb4a7a72700 1 Processor -- start 2024-04-27T01:58:41.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.144+0000 7fb4a7a72700 1 -- start start 2024-04-27T01:58:41.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.144+0000 7fb4a7a72700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4a00fed20 0x7fb4a01971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:41.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.144+0000 7fb4a7a72700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a01976e0 con 0x7fb4a00fed20 2024-04-27T01:58:41.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.144+0000 7fb4a580e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4a00fed20 0x7fb4a01971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:41.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.144+0000 7fb4a580e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4a00fed20 0x7fb4a01971a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.52:46122/0 (socket says 172.21.15.52:46122) 2024-04-27T01:58:41.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.144+0000 7fb4a580e700 1 -- 172.21.15.52:0/1857688718 learned_addr learned my addr 172.21.15.52:0/1857688718 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T01:58:41.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.144+0000 7fb4a580e700 1 -- 172.21.15.52:0/1857688718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49c009d00 con 0x7fb4a00fed20 2024-04-27T01:58:41.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.145+0000 7fb4a580e700 1 --2- 172.21.15.52:0/1857688718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4a00fed20 0x7fb4a01971a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00b830 tx=0x7fb49c00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:41.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.145+0000 7fb496ffd700 1 -- 172.21.15.52:0/1857688718 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb49c00e040 con 0x7fb4a00fed20 2024-04-27T01:58:41.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.145+0000 7fb4a7a72700 1 -- 172.21.15.52:0/1857688718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a01978e0 con 0x7fb4a00fed20 2024-04-27T01:58:41.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.145+0000 7fb496ffd700 1 -- 172.21.15.52:0/1857688718 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb49c012af0 con 0x7fb4a00fed20 2024-04-27T01:58:41.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.145+0000 7fb496ffd700 1 -- 172.21.15.52:0/1857688718 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb49c01d9f0 con 0x7fb4a00fed20 2024-04-27T01:58:41.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.145+0000 7fb4a7a72700 1 -- 172.21.15.52:0/1857688718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a0197d00 con 0x7fb4a00fed20 2024-04-27T01:58:41.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.146+0000 7fb496ffd700 1 -- 172.21.15.52:0/1857688718 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7fb49c01dcb0 con 0x7fb4a00fed20 2024-04-27T01:58:41.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.146+0000 7fb4a7a72700 1 -- 172.21.15.52:0/1857688718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a0190e10 con 0x7fb4a00fed20 2024-04-27T01:58:41.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.146+0000 7fb496ffd700 1 --2- 172.21.15.52:0/1857688718 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fb48c038490 0x7fb48c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:41.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.146+0000 7fb496ffd700 1 -- 172.21.15.52:0/1857688718 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb49c04cae0 con 0x7fb4a00fed20 2024-04-27T01:58:41.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.146+0000 7fb4a500d700 1 --2- 172.21.15.52:0/1857688718 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fb48c038490 0x7fb48c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:41.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.146+0000 7fb4a500d700 1 --2- 172.21.15.52:0/1857688718 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fb48c038490 0x7fb48c03a940 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb490006fd0 tx=0x7fb490006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:41.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.151+0000 7fb496ffd700 1 -- 172.21.15.52:0/1857688718 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb49c01aba0 con 0x7fb4a00fed20 2024-04-27T01:58:41.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.440+0000 7fb4a7a72700 1 -- 172.21.15.52:0/1857688718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4a0062380 con 0x7fb4a00fed20 2024-04-27T01:58:41.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.440+0000 7fb496ffd700 1 -- 172.21.15.52:0/1857688718 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fb49c01aba0 con 0x7fb4a00fed20 2024-04-27T01:58:41.422 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:58:41.422 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":1,"fsid":"4c587076-0439-11ef-bc93-c7b262605968","modified":"2024-04-27T01:56:49.232926Z","created":"2024-04-27T01:56:49.232926Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:58:41.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.443+0000 7fb4a7a72700 1 -- 172.21.15.52:0/1857688718 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fb48c038490 msgr2=0x7fb48c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:41.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.443+0000 7fb4a7a72700 1 --2- 172.21.15.52:0/1857688718 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fb48c038490 0x7fb48c03a940 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb490006fd0 tx=0x7fb490006e40 comp rx=0 tx=0).stop 2024-04-27T01:58:41.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.443+0000 7fb4a7a72700 1 -- 172.21.15.52:0/1857688718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4a00fed20 msgr2=0x7fb4a01971a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:41.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.443+0000 7fb4a7a72700 1 --2- 172.21.15.52:0/1857688718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4a00fed20 0x7fb4a01971a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00b830 tx=0x7fb49c00bd40 comp rx=0 tx=0).stop 2024-04-27T01:58:41.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.443+0000 7fb4a7a72700 1 -- 172.21.15.52:0/1857688718 shutdown_connections 2024-04-27T01:58:41.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.443+0000 7fb4a7a72700 1 --2- 172.21.15.52:0/1857688718 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fb48c038490 0x7fb48c03a940 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:41.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.443+0000 7fb4a7a72700 1 --2- 172.21.15.52:0/1857688718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4a00fed20 0x7fb4a01971a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:41.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.443+0000 7fb4a7a72700 1 -- 172.21.15.52:0/1857688718 >> 172.21.15.52:0/1857688718 conn(0x7fb4a00fa2b0 msgr2=0x7fb4a00690b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:41.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.443+0000 7fb4a7a72700 1 -- 172.21.15.52:0/1857688718 shutdown_connections 2024-04-27T01:58:41.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:41.443+0000 7fb4a7a72700 1 -- 172.21.15.52:0/1857688718 wait complete. 2024-04-27T01:58:41.425 INFO:teuthology.orchestra.run.smithi052.stderr:dumped monmap epoch 1 2024-04-27T01:58:42.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:41 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/1857688718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:58:43.000 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:58:43.000 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:58:43.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:42 smithi029 ceph-mon[26769]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:58:43.211 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:58:43.258 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:58:44.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.714+0000 7f405cf8f700 1 -- 172.21.15.52:0/3591078350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40581008b0 msgr2=0x7f4058102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:44.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.714+0000 7f405cf8f700 1 --2- 172.21.15.52:0/3591078350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40581008b0 0x7f4058102c90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f404c009910 tx=0x7f404c009c20 comp rx=0 tx=0).stop 2024-04-27T01:58:44.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.714+0000 7f405cf8f700 1 -- 172.21.15.52:0/3591078350 shutdown_connections 2024-04-27T01:58:44.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.714+0000 7f405cf8f700 1 --2- 172.21.15.52:0/3591078350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40581008b0 0x7f4058102c90 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:44.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.714+0000 7f405cf8f700 1 -- 172.21.15.52:0/3591078350 >> 172.21.15.52:0/3591078350 conn(0x7f40580fa2b0 msgr2=0x7f40580fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:44.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.714+0000 7f405cf8f700 1 -- 172.21.15.52:0/3591078350 shutdown_connections 2024-04-27T01:58:44.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.714+0000 7f405cf8f700 1 -- 172.21.15.52:0/3591078350 wait complete. 2024-04-27T01:58:44.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.715+0000 7f405cf8f700 1 Processor -- start 2024-04-27T01:58:44.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.715+0000 7f405cf8f700 1 -- start start 2024-04-27T01:58:44.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.715+0000 7f405cf8f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40581008b0 0x7f4058100210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:44.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.715+0000 7f405cf8f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4058100750 con 0x7f40581008b0 2024-04-27T01:58:44.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.715+0000 7f405659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40581008b0 0x7f4058100210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:44.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.715+0000 7f405659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40581008b0 0x7f4058100210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.52:46144/0 (socket says 172.21.15.52:46144) 2024-04-27T01:58:44.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.716+0000 7f405659c700 1 -- 172.21.15.52:0/3510095795 learned_addr learned my addr 172.21.15.52:0/3510095795 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T01:58:44.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.716+0000 7f405659c700 1 -- 172.21.15.52:0/3510095795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404c0095f0 con 0x7f40581008b0 2024-04-27T01:58:44.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.716+0000 7f405659c700 1 --2- 172.21.15.52:0/3510095795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40581008b0 0x7f4058100210 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f404c005b40 tx=0x7f404c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:44.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.716+0000 7f40477fe700 1 -- 172.21.15.52:0/3510095795 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f404c01d070 con 0x7f40581008b0 2024-04-27T01:58:44.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.716+0000 7f405cf8f700 1 -- 172.21.15.52:0/3510095795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40580fe860 con 0x7f40581008b0 2024-04-27T01:58:44.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.716+0000 7f40477fe700 1 -- 172.21.15.52:0/3510095795 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f404c00be10 con 0x7f40581008b0 2024-04-27T01:58:44.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.716+0000 7f40477fe700 1 -- 172.21.15.52:0/3510095795 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f404c00f460 con 0x7f40581008b0 2024-04-27T01:58:44.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.716+0000 7f405cf8f700 1 -- 172.21.15.52:0/3510095795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40580fed00 con 0x7f40581008b0 2024-04-27T01:58:44.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.717+0000 7f40477fe700 1 -- 172.21.15.52:0/3510095795 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f404c00f6d0 con 0x7f40581008b0 2024-04-27T01:58:44.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.717+0000 7f405cf8f700 1 -- 172.21.15.52:0/3510095795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4058062380 con 0x7f40581008b0 2024-04-27T01:58:44.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.717+0000 7f40477fe700 1 --2- 172.21.15.52:0/3510095795 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f40400384e0 0x7f404003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:44.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.717+0000 7f40477fe700 1 -- 172.21.15.52:0/3510095795 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f404c04c4e0 con 0x7f40581008b0 2024-04-27T01:58:44.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.718+0000 7f4055d9b700 1 --2- 172.21.15.52:0/3510095795 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f40400384e0 0x7f404003a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:44.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.718+0000 7f4055d9b700 1 --2- 172.21.15.52:0/3510095795 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f40400384e0 0x7f404003a990 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4048006fd0 tx=0x7f4048006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:44.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:44.723+0000 7f40477fe700 1 -- 172.21.15.52:0/3510095795 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f404c02a360 con 0x7f40581008b0 2024-04-27T01:58:44.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:45.001+0000 7f405cf8f700 1 -- 172.21.15.52:0/3510095795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f405818e4e0 con 0x7f40581008b0 2024-04-27T01:58:44.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:45.002+0000 7f40477fe700 1 -- 172.21.15.52:0/3510095795 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f404c045e60 con 0x7f40581008b0 2024-04-27T01:58:44.985 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:58:44.985 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":1,"fsid":"4c587076-0439-11ef-bc93-c7b262605968","modified":"2024-04-27T01:56:49.232926Z","created":"2024-04-27T01:56:49.232926Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:58:44.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:45.005+0000 7f405cf8f700 1 -- 172.21.15.52:0/3510095795 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f40400384e0 msgr2=0x7f404003a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:44.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:45.005+0000 7f405cf8f700 1 --2- 172.21.15.52:0/3510095795 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f40400384e0 0x7f404003a990 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4048006fd0 tx=0x7f4048006e40 comp rx=0 tx=0).stop 2024-04-27T01:58:44.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:45.005+0000 7f405cf8f700 1 -- 172.21.15.52:0/3510095795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40581008b0 msgr2=0x7f4058100210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:44.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:45.005+0000 7f405cf8f700 1 --2- 172.21.15.52:0/3510095795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40581008b0 0x7f4058100210 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f404c005b40 tx=0x7f404c004740 comp rx=0 tx=0).stop 2024-04-27T01:58:44.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:45.005+0000 7f405cf8f700 1 -- 172.21.15.52:0/3510095795 shutdown_connections 2024-04-27T01:58:44.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:45.005+0000 7f405cf8f700 1 --2- 172.21.15.52:0/3510095795 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f40400384e0 0x7f404003a990 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:44.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:45.005+0000 7f405cf8f700 1 --2- 172.21.15.52:0/3510095795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40581008b0 0x7f4058100210 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:44.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:45.005+0000 7f405cf8f700 1 -- 172.21.15.52:0/3510095795 >> 172.21.15.52:0/3510095795 conn(0x7f40580fa2b0 msgr2=0x7f40580faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:44.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:45.005+0000 7f405cf8f700 1 -- 172.21.15.52:0/3510095795 shutdown_connections 2024-04-27T01:58:44.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:45.005+0000 7f405cf8f700 1 -- 172.21.15.52:0/3510095795 wait complete. 2024-04-27T01:58:44.988 INFO:teuthology.orchestra.run.smithi052.stderr:dumped monmap epoch 1 2024-04-27T01:58:45.179 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:44 smithi029 ceph-mon[26769]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:58:45.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:45 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/3510095795' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:58:46.555 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:58:46.555 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:58:46.760 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:58:46.807 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:58:47.125 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:46 smithi029 ceph-mon[26769]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:58:48.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.351+0000 7fb9866ee700 1 -- 172.21.15.52:0/3536547835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9800fed20 msgr2=0x7fb9800ff130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:48.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.351+0000 7fb9866ee700 1 --2- 172.21.15.52:0/3536547835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9800fed20 0x7fb9800ff130 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb968009910 tx=0x7fb968009c20 comp rx=0 tx=0).stop 2024-04-27T01:58:48.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.351+0000 7fb9866ee700 1 -- 172.21.15.52:0/3536547835 shutdown_connections 2024-04-27T01:58:48.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.351+0000 7fb9866ee700 1 --2- 172.21.15.52:0/3536547835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9800fed20 0x7fb9800ff130 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:48.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.351+0000 7fb9866ee700 1 -- 172.21.15.52:0/3536547835 >> 172.21.15.52:0/3536547835 conn(0x7fb9800fa2b0 msgr2=0x7fb9800fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:48.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.352+0000 7fb9866ee700 1 -- 172.21.15.52:0/3536547835 shutdown_connections 2024-04-27T01:58:48.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.352+0000 7fb9866ee700 1 -- 172.21.15.52:0/3536547835 wait complete. 2024-04-27T01:58:48.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.352+0000 7fb9866ee700 1 Processor -- start 2024-04-27T01:58:48.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.352+0000 7fb9866ee700 1 -- start start 2024-04-27T01:58:48.335 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.353+0000 7fb9866ee700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9800fed20 0x7fb980068ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:48.335 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.353+0000 7fb9866ee700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb980195b20 con 0x7fb9800fed20 2024-04-27T01:58:48.335 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.353+0000 7fb97ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9800fed20 0x7fb980068ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:48.335 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.353+0000 7fb97ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9800fed20 0x7fb980068ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.52:52654/0 (socket says 172.21.15.52:52654) 2024-04-27T01:58:48.335 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.353+0000 7fb97ffff700 1 -- 172.21.15.52:0/549881114 learned_addr learned my addr 172.21.15.52:0/549881114 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T01:58:48.335 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.353+0000 7fb97ffff700 1 -- 172.21.15.52:0/549881114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9680095f0 con 0x7fb9800fed20 2024-04-27T01:58:48.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.353+0000 7fb97ffff700 1 --2- 172.21.15.52:0/549881114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9800fed20 0x7fb980068ab0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb968005b40 tx=0x7fb968004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:48.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.353+0000 7fb97d7fa700 1 -- 172.21.15.52:0/549881114 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb96801d070 con 0x7fb9800fed20 2024-04-27T01:58:48.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.353+0000 7fb97d7fa700 1 -- 172.21.15.52:0/549881114 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb96800be10 con 0x7fb9800fed20 2024-04-27T01:58:48.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.353+0000 7fb9866ee700 1 -- 172.21.15.52:0/549881114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb98019a130 con 0x7fb9800fed20 2024-04-27T01:58:48.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.354+0000 7fb97d7fa700 1 -- 172.21.15.52:0/549881114 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb96800f460 con 0x7fb9800fed20 2024-04-27T01:58:48.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.354+0000 7fb9866ee700 1 -- 172.21.15.52:0/549881114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb98019a490 con 0x7fb9800fed20 2024-04-27T01:58:48.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.354+0000 7fb97d7fa700 1 -- 172.21.15.52:0/549881114 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7fb96800f5c0 con 0x7fb9800fed20 2024-04-27T01:58:48.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.354+0000 7fb9866ee700 1 -- 172.21.15.52:0/549881114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb98018da00 con 0x7fb9800fed20 2024-04-27T01:58:48.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.354+0000 7fb97d7fa700 1 --2- 172.21.15.52:0/549881114 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fb96c0384e0 0x7fb96c03a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:48.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.355+0000 7fb97d7fa700 1 -- 172.21.15.52:0/549881114 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb96804c550 con 0x7fb9800fed20 2024-04-27T01:58:48.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.355+0000 7fb97f7fe700 1 --2- 172.21.15.52:0/549881114 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fb96c0384e0 0x7fb96c03a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:48.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.355+0000 7fb97f7fe700 1 --2- 172.21.15.52:0/549881114 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fb96c0384e0 0x7fb96c03a990 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb970006fd0 tx=0x7fb970006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:48.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.360+0000 7fb97d7fa700 1 -- 172.21.15.52:0/549881114 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb968020550 con 0x7fb9800fed20 2024-04-27T01:58:48.623 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.641+0000 7fb9866ee700 1 -- 172.21.15.52:0/549881114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb98002cc30 con 0x7fb9800fed20 2024-04-27T01:58:48.623 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.642+0000 7fb97d7fa700 1 -- 172.21.15.52:0/549881114 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fb968027020 con 0x7fb9800fed20 2024-04-27T01:58:48.624 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:58:48.624 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":1,"fsid":"4c587076-0439-11ef-bc93-c7b262605968","modified":"2024-04-27T01:56:49.232926Z","created":"2024-04-27T01:56:49.232926Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:58:48.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.644+0000 7fb9866ee700 1 -- 172.21.15.52:0/549881114 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fb96c0384e0 msgr2=0x7fb96c03a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:48.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.644+0000 7fb9866ee700 1 --2- 172.21.15.52:0/549881114 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fb96c0384e0 0x7fb96c03a990 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb970006fd0 tx=0x7fb970006e40 comp rx=0 tx=0).stop 2024-04-27T01:58:48.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.644+0000 7fb9866ee700 1 -- 172.21.15.52:0/549881114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9800fed20 msgr2=0x7fb980068ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:48.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.644+0000 7fb9866ee700 1 --2- 172.21.15.52:0/549881114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9800fed20 0x7fb980068ab0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb968005b40 tx=0x7fb968004740 comp rx=0 tx=0).stop 2024-04-27T01:58:48.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.644+0000 7fb9866ee700 1 -- 172.21.15.52:0/549881114 shutdown_connections 2024-04-27T01:58:48.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.644+0000 7fb9866ee700 1 --2- 172.21.15.52:0/549881114 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fb96c0384e0 0x7fb96c03a990 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:48.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.644+0000 7fb9866ee700 1 --2- 172.21.15.52:0/549881114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9800fed20 0x7fb980068ab0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:48.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.644+0000 7fb9866ee700 1 -- 172.21.15.52:0/549881114 >> 172.21.15.52:0/549881114 conn(0x7fb9800fa2b0 msgr2=0x7fb9800faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:48.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.644+0000 7fb9866ee700 1 -- 172.21.15.52:0/549881114 shutdown_connections 2024-04-27T01:58:48.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:48.644+0000 7fb9866ee700 1 -- 172.21.15.52:0/549881114 wait complete. 2024-04-27T01:58:48.627 INFO:teuthology.orchestra.run.smithi052.stderr:dumped monmap epoch 1 2024-04-27T01:58:49.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:48 smithi029 ceph-mon[26769]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:58:49.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:48 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:49.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:48 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:49.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:48 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:49.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:48 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:49.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:48 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:49.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:48 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:49.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:48 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/549881114' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:58:50.137 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:58:50.138 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:58:50.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:49 smithi029 ceph-mon[26769]: Deploying daemon prometheus.smithi029 on smithi029 2024-04-27T01:58:50.345 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:58:50.392 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:58:50.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:50 smithi029 ceph-mon[26769]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:58:51.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.899+0000 7f15027bd700 1 -- 172.21.15.52:0/1854639235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14fc0fed20 msgr2=0x7f14fc0ff130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:51.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.899+0000 7f15027bd700 1 --2- 172.21.15.52:0/1854639235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14fc0fed20 0x7f14fc0ff130 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f14f0009910 tx=0x7f14f0009c20 comp rx=0 tx=0).stop 2024-04-27T01:58:51.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.899+0000 7f15027bd700 1 -- 172.21.15.52:0/1854639235 shutdown_connections 2024-04-27T01:58:51.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.899+0000 7f15027bd700 1 --2- 172.21.15.52:0/1854639235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14fc0fed20 0x7f14fc0ff130 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:51.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.899+0000 7f15027bd700 1 -- 172.21.15.52:0/1854639235 >> 172.21.15.52:0/1854639235 conn(0x7f14fc0fa2b0 msgr2=0x7f14fc0fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:51.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.899+0000 7f15027bd700 1 -- 172.21.15.52:0/1854639235 shutdown_connections 2024-04-27T01:58:51.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.899+0000 7f15027bd700 1 -- 172.21.15.52:0/1854639235 wait complete. 2024-04-27T01:58:51.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.900+0000 7f15027bd700 1 Processor -- start 2024-04-27T01:58:51.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.900+0000 7f15027bd700 1 -- start start 2024-04-27T01:58:51.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.900+0000 7f15027bd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14fc0fed20 0x7f14fc068ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:51.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.900+0000 7f15027bd700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14fc197d30 con 0x7f14fc0fed20 2024-04-27T01:58:51.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.900+0000 7f14fbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14fc0fed20 0x7f14fc068ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:51.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.901+0000 7f14fbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14fc0fed20 0x7f14fc068ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.52:52670/0 (socket says 172.21.15.52:52670) 2024-04-27T01:58:51.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.901+0000 7f14fbfff700 1 -- 172.21.15.52:0/3115803316 learned_addr learned my addr 172.21.15.52:0/3115803316 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T01:58:51.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.901+0000 7f14fbfff700 1 -- 172.21.15.52:0/3115803316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f00095f0 con 0x7f14fc0fed20 2024-04-27T01:58:51.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.901+0000 7f14fbfff700 1 --2- 172.21.15.52:0/3115803316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14fc0fed20 0x7f14fc068ab0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f14f0005b40 tx=0x7f14f0004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:51.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.901+0000 7f14f97fa700 1 -- 172.21.15.52:0/3115803316 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14f001d070 con 0x7f14fc0fed20 2024-04-27T01:58:51.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.901+0000 7f14f97fa700 1 -- 172.21.15.52:0/3115803316 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f14f000be10 con 0x7f14fc0fed20 2024-04-27T01:58:51.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.901+0000 7f15027bd700 1 -- 172.21.15.52:0/3115803316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14fc197e70 con 0x7f14fc0fed20 2024-04-27T01:58:51.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.901+0000 7f14f97fa700 1 -- 172.21.15.52:0/3115803316 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14f000f460 con 0x7f14fc0fed20 2024-04-27T01:58:51.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.901+0000 7f15027bd700 1 -- 172.21.15.52:0/3115803316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14fc1982b0 con 0x7f14fc0fed20 2024-04-27T01:58:51.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.902+0000 7f14f97fa700 1 -- 172.21.15.52:0/3115803316 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f14f000f6d0 con 0x7f14fc0fed20 2024-04-27T01:58:51.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.902+0000 7f14f97fa700 1 --2- 172.21.15.52:0/3115803316 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f14e4038490 0x7f14e403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:51.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.902+0000 7f15027bd700 1 -- 172.21.15.52:0/3115803316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14fc18da00 con 0x7f14fc0fed20 2024-04-27T01:58:51.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.902+0000 7f14f97fa700 1 -- 172.21.15.52:0/3115803316 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f14f004c4a0 con 0x7f14fc0fed20 2024-04-27T01:58:51.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.902+0000 7f14fb7fe700 1 --2- 172.21.15.52:0/3115803316 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f14e4038490 0x7f14e403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:51.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.903+0000 7f14fb7fe700 1 --2- 172.21.15.52:0/3115803316 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f14e4038490 0x7f14e403a940 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f14ec006fd0 tx=0x7f14ec006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:51.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:51.908+0000 7f14f97fa700 1 -- 172.21.15.52:0/3115803316 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f14f0027070 con 0x7f14fc0fed20 2024-04-27T01:58:52.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:52.192+0000 7f15027bd700 1 -- 172.21.15.52:0/3115803316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14fc02cc30 con 0x7f14fc0fed20 2024-04-27T01:58:52.174 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:52.192+0000 7f14f97fa700 1 -- 172.21.15.52:0/3115803316 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f14f002c440 con 0x7f14fc0fed20 2024-04-27T01:58:52.174 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:58:52.174 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":1,"fsid":"4c587076-0439-11ef-bc93-c7b262605968","modified":"2024-04-27T01:56:49.232926Z","created":"2024-04-27T01:56:49.232926Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:58:52.176 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:52.194+0000 7f15027bd700 1 -- 172.21.15.52:0/3115803316 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f14e4038490 msgr2=0x7f14e403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:52.176 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:52.194+0000 7f15027bd700 1 --2- 172.21.15.52:0/3115803316 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f14e4038490 0x7f14e403a940 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f14ec006fd0 tx=0x7f14ec006e40 comp rx=0 tx=0).stop 2024-04-27T01:58:52.176 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:52.194+0000 7f15027bd700 1 -- 172.21.15.52:0/3115803316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14fc0fed20 msgr2=0x7f14fc068ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:52.176 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:52.194+0000 7f15027bd700 1 --2- 172.21.15.52:0/3115803316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14fc0fed20 0x7f14fc068ab0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f14f0005b40 tx=0x7f14f0004740 comp rx=0 tx=0).stop 2024-04-27T01:58:52.176 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:52.195+0000 7f15027bd700 1 -- 172.21.15.52:0/3115803316 shutdown_connections 2024-04-27T01:58:52.176 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:52.195+0000 7f15027bd700 1 --2- 172.21.15.52:0/3115803316 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f14e4038490 0x7f14e403a940 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:52.176 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:52.195+0000 7f15027bd700 1 --2- 172.21.15.52:0/3115803316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14fc0fed20 0x7f14fc068ab0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:52.176 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:52.195+0000 7f15027bd700 1 -- 172.21.15.52:0/3115803316 >> 172.21.15.52:0/3115803316 conn(0x7f14fc0fa2b0 msgr2=0x7f14fc0faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:52.177 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:52.195+0000 7f15027bd700 1 -- 172.21.15.52:0/3115803316 shutdown_connections 2024-04-27T01:58:52.177 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:52.195+0000 7f15027bd700 1 -- 172.21.15.52:0/3115803316 wait complete. 2024-04-27T01:58:52.177 INFO:teuthology.orchestra.run.smithi052.stderr:dumped monmap epoch 1 2024-04-27T01:58:53.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:52 smithi029 ceph-mon[26769]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:58:53.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:52 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/3115803316' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:58:53.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:52 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:53.980 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:58:53.980 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:58:54.187 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:58:54.235 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:58:55.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:54 smithi029 ceph-mon[26769]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:58:55.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.866+0000 7f5f8ab39700 1 -- 172.21.15.52:0/2775607253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f841008b0 msgr2=0x7f5f84102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:55.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.866+0000 7f5f8ab39700 1 --2- 172.21.15.52:0/2775607253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f841008b0 0x7f5f84102c90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009910 tx=0x7f5f78009c20 comp rx=0 tx=0).stop 2024-04-27T01:58:55.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.866+0000 7f5f8ab39700 1 -- 172.21.15.52:0/2775607253 shutdown_connections 2024-04-27T01:58:55.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.866+0000 7f5f8ab39700 1 --2- 172.21.15.52:0/2775607253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f841008b0 0x7f5f84102c90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:55.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.866+0000 7f5f8ab39700 1 -- 172.21.15.52:0/2775607253 >> 172.21.15.52:0/2775607253 conn(0x7f5f840fa2b0 msgr2=0x7f5f840fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:55.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.866+0000 7f5f8ab39700 1 -- 172.21.15.52:0/2775607253 shutdown_connections 2024-04-27T01:58:55.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.866+0000 7f5f8ab39700 1 -- 172.21.15.52:0/2775607253 wait complete. 2024-04-27T01:58:55.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.867+0000 7f5f8ab39700 1 Processor -- start 2024-04-27T01:58:55.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.867+0000 7f5f8ab39700 1 -- start start 2024-04-27T01:58:55.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.867+0000 7f5f8ab39700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f841008b0 0x7f5f84100210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:55.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.867+0000 7f5f8ab39700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f84100750 con 0x7f5f841008b0 2024-04-27T01:58:55.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.867+0000 7f5f888d5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f841008b0 0x7f5f84100210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:55.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.867+0000 7f5f888d5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f841008b0 0x7f5f84100210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.52:38580/0 (socket says 172.21.15.52:38580) 2024-04-27T01:58:55.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.867+0000 7f5f888d5700 1 -- 172.21.15.52:0/287511306 learned_addr learned my addr 172.21.15.52:0/287511306 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T01:58:55.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.868+0000 7f5f888d5700 1 -- 172.21.15.52:0/287511306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f780095f0 con 0x7f5f841008b0 2024-04-27T01:58:55.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.868+0000 7f5f888d5700 1 --2- 172.21.15.52:0/287511306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f841008b0 0x7f5f84100210 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5f78005b40 tx=0x7f5f78004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:55.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.868+0000 7f5f81ffb700 1 -- 172.21.15.52:0/287511306 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f7801d070 con 0x7f5f841008b0 2024-04-27T01:58:55.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.868+0000 7f5f81ffb700 1 -- 172.21.15.52:0/287511306 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5f7800be10 con 0x7f5f841008b0 2024-04-27T01:58:55.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.868+0000 7f5f8ab39700 1 -- 172.21.15.52:0/287511306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f840fe860 con 0x7f5f841008b0 2024-04-27T01:58:55.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.868+0000 7f5f81ffb700 1 -- 172.21.15.52:0/287511306 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f7800f460 con 0x7f5f841008b0 2024-04-27T01:58:55.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.869+0000 7f5f8ab39700 1 -- 172.21.15.52:0/287511306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f840fed00 con 0x7f5f841008b0 2024-04-27T01:58:55.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.869+0000 7f5f81ffb700 1 -- 172.21.15.52:0/287511306 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f5f7800f5c0 con 0x7f5f841008b0 2024-04-27T01:58:55.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.869+0000 7f5f8ab39700 1 -- 172.21.15.52:0/287511306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f84062380 con 0x7f5f841008b0 2024-04-27T01:58:55.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.870+0000 7f5f81ffb700 1 --2- 172.21.15.52:0/287511306 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f5f740384e0 0x7f5f7403a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:55.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.870+0000 7f5f81ffb700 1 -- 172.21.15.52:0/287511306 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5f7804c550 con 0x7f5f841008b0 2024-04-27T01:58:55.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.870+0000 7f5f83fff700 1 --2- 172.21.15.52:0/287511306 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f5f740384e0 0x7f5f7403a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:55.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.870+0000 7f5f83fff700 1 --2- 172.21.15.52:0/287511306 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f5f740384e0 0x7f5f7403a990 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5f70006fd0 tx=0x7f5f70006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:55.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:55.875+0000 7f5f81ffb700 1 -- 172.21.15.52:0/287511306 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f5f78020550 con 0x7f5f841008b0 2024-04-27T01:58:56.140 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:56.159+0000 7f5f8ab39700 1 -- 172.21.15.52:0/287511306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f8418e4e0 con 0x7f5f841008b0 2024-04-27T01:58:56.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:56.160+0000 7f5f81ffb700 1 -- 172.21.15.52:0/287511306 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f5f78027020 con 0x7f5f841008b0 2024-04-27T01:58:56.141 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:58:56.141 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":1,"fsid":"4c587076-0439-11ef-bc93-c7b262605968","modified":"2024-04-27T01:56:49.232926Z","created":"2024-04-27T01:56:49.232926Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:58:56.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:56.163+0000 7f5f8ab39700 1 -- 172.21.15.52:0/287511306 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f5f740384e0 msgr2=0x7f5f7403a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:56.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:56.163+0000 7f5f8ab39700 1 --2- 172.21.15.52:0/287511306 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f5f740384e0 0x7f5f7403a990 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5f70006fd0 tx=0x7f5f70006e40 comp rx=0 tx=0).stop 2024-04-27T01:58:56.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:56.163+0000 7f5f8ab39700 1 -- 172.21.15.52:0/287511306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f841008b0 msgr2=0x7f5f84100210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:56.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:56.163+0000 7f5f8ab39700 1 --2- 172.21.15.52:0/287511306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f841008b0 0x7f5f84100210 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5f78005b40 tx=0x7f5f78004740 comp rx=0 tx=0).stop 2024-04-27T01:58:56.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:56.163+0000 7f5f8ab39700 1 -- 172.21.15.52:0/287511306 shutdown_connections 2024-04-27T01:58:56.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:56.163+0000 7f5f8ab39700 1 --2- 172.21.15.52:0/287511306 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f5f740384e0 0x7f5f7403a990 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:56.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:56.163+0000 7f5f8ab39700 1 --2- 172.21.15.52:0/287511306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f841008b0 0x7f5f84100210 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:56.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:56.163+0000 7f5f8ab39700 1 -- 172.21.15.52:0/287511306 >> 172.21.15.52:0/287511306 conn(0x7f5f840fa2b0 msgr2=0x7f5f840faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:56.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:56.163+0000 7f5f8ab39700 1 -- 172.21.15.52:0/287511306 shutdown_connections 2024-04-27T01:58:56.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:56.163+0000 7f5f8ab39700 1 -- 172.21.15.52:0/287511306 wait complete. 2024-04-27T01:58:56.145 INFO:teuthology.orchestra.run.smithi052.stderr:dumped monmap epoch 1 2024-04-27T01:58:57.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:56 smithi029 ceph-mon[26769]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:58:57.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:56 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/287511306' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:58:57.721 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:58:57.721 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:58:57.928 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:58:57.975 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:58:59.019 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:58 smithi029 ceph-mon[26769]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:58:59.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.492+0000 7f6e92b59700 1 -- 172.21.15.52:0/878893243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e8c0ffe20 msgr2=0x7f6e8c100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:59.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.492+0000 7f6e92b59700 1 --2- 172.21.15.52:0/878893243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e8c0ffe20 0x7f6e8c100230 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6e80009910 tx=0x7f6e80009c20 comp rx=0 tx=0).stop 2024-04-27T01:58:59.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.492+0000 7f6e92b59700 1 -- 172.21.15.52:0/878893243 shutdown_connections 2024-04-27T01:58:59.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.492+0000 7f6e92b59700 1 --2- 172.21.15.52:0/878893243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e8c0ffe20 0x7f6e8c100230 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:59.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.492+0000 7f6e92b59700 1 -- 172.21.15.52:0/878893243 >> 172.21.15.52:0/878893243 conn(0x7f6e8c0fb3d0 msgr2=0x7f6e8c0fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:59.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.492+0000 7f6e92b59700 1 -- 172.21.15.52:0/878893243 shutdown_connections 2024-04-27T01:58:59.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.492+0000 7f6e92b59700 1 -- 172.21.15.52:0/878893243 wait complete. 2024-04-27T01:58:59.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.493+0000 7f6e92b59700 1 Processor -- start 2024-04-27T01:58:59.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.493+0000 7f6e92b59700 1 -- start start 2024-04-27T01:58:59.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.493+0000 7f6e92b59700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e8c0ffe20 0x7f6e8c194f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:59.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.493+0000 7f6e92b59700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8c1954a0 con 0x7f6e8c0ffe20 2024-04-27T01:58:59.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.494+0000 7f6e908f5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e8c0ffe20 0x7f6e8c194f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:59.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.494+0000 7f6e908f5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e8c0ffe20 0x7f6e8c194f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.52:38594/0 (socket says 172.21.15.52:38594) 2024-04-27T01:58:59.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.494+0000 7f6e908f5700 1 -- 172.21.15.52:0/540317060 learned_addr learned my addr 172.21.15.52:0/540317060 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T01:58:59.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.494+0000 7f6e908f5700 1 -- 172.21.15.52:0/540317060 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e800095f0 con 0x7f6e8c0ffe20 2024-04-27T01:58:59.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.494+0000 7f6e908f5700 1 --2- 172.21.15.52:0/540317060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e8c0ffe20 0x7f6e8c194f60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6e800092e0 tx=0x7f6e80009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:59.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.494+0000 7f6e89ffb700 1 -- 172.21.15.52:0/540317060 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e8001d070 con 0x7f6e8c0ffe20 2024-04-27T01:58:59.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.494+0000 7f6e89ffb700 1 -- 172.21.15.52:0/540317060 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6e8000be10 con 0x7f6e8c0ffe20 2024-04-27T01:58:59.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.494+0000 7f6e92b59700 1 -- 172.21.15.52:0/540317060 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e8c1956a0 con 0x7f6e8c0ffe20 2024-04-27T01:58:59.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.494+0000 7f6e89ffb700 1 -- 172.21.15.52:0/540317060 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e8000f460 con 0x7f6e8c0ffe20 2024-04-27T01:58:59.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.495+0000 7f6e92b59700 1 -- 172.21.15.52:0/540317060 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e8c195b40 con 0x7f6e8c0ffe20 2024-04-27T01:58:59.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.495+0000 7f6e89ffb700 1 -- 172.21.15.52:0/540317060 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f6e8000b960 con 0x7f6e8c0ffe20 2024-04-27T01:58:59.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.495+0000 7f6e92b59700 1 -- 172.21.15.52:0/540317060 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e8c18ea80 con 0x7f6e8c0ffe20 2024-04-27T01:58:59.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.496+0000 7f6e89ffb700 1 --2- 172.21.15.52:0/540317060 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f6e7c0384e0 0x7f6e7c03a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:58:59.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.496+0000 7f6e89ffb700 1 -- 172.21.15.52:0/540317060 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6e8004c340 con 0x7f6e8c0ffe20 2024-04-27T01:58:59.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.496+0000 7f6e8bfff700 1 --2- 172.21.15.52:0/540317060 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f6e7c0384e0 0x7f6e7c03a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:58:59.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.496+0000 7f6e8bfff700 1 --2- 172.21.15.52:0/540317060 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f6e7c0384e0 0x7f6e7c03a990 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6e78006fd0 tx=0x7f6e78006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:58:59.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.501+0000 7f6e89ffb700 1 -- 172.21.15.52:0/540317060 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6e80022790 con 0x7f6e8c0ffe20 2024-04-27T01:58:59.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.781+0000 7f6e92b59700 1 -- 172.21.15.52:0/540317060 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e8c02cc30 con 0x7f6e8c0ffe20 2024-04-27T01:58:59.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.782+0000 7f6e89ffb700 1 -- 172.21.15.52:0/540317060 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f6e80027030 con 0x7f6e8c0ffe20 2024-04-27T01:58:59.763 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:58:59.763 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":1,"fsid":"4c587076-0439-11ef-bc93-c7b262605968","modified":"2024-04-27T01:56:49.232926Z","created":"2024-04-27T01:56:49.232926Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:58:59.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.784+0000 7f6e92b59700 1 -- 172.21.15.52:0/540317060 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f6e7c0384e0 msgr2=0x7f6e7c03a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:59.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.784+0000 7f6e92b59700 1 --2- 172.21.15.52:0/540317060 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f6e7c0384e0 0x7f6e7c03a990 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6e78006fd0 tx=0x7f6e78006e40 comp rx=0 tx=0).stop 2024-04-27T01:58:59.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.784+0000 7f6e92b59700 1 -- 172.21.15.52:0/540317060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e8c0ffe20 msgr2=0x7f6e8c194f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:58:59.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.784+0000 7f6e92b59700 1 --2- 172.21.15.52:0/540317060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e8c0ffe20 0x7f6e8c194f60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6e800092e0 tx=0x7f6e80009e60 comp rx=0 tx=0).stop 2024-04-27T01:58:59.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.784+0000 7f6e92b59700 1 -- 172.21.15.52:0/540317060 shutdown_connections 2024-04-27T01:58:59.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.784+0000 7f6e92b59700 1 --2- 172.21.15.52:0/540317060 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f6e7c0384e0 0x7f6e7c03a990 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:59.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.784+0000 7f6e92b59700 1 --2- 172.21.15.52:0/540317060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e8c0ffe20 0x7f6e8c194f60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:58:59.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.784+0000 7f6e92b59700 1 -- 172.21.15.52:0/540317060 >> 172.21.15.52:0/540317060 conn(0x7f6e8c0fb3d0 msgr2=0x7f6e8c0fc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:58:59.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.784+0000 7f6e92b59700 1 -- 172.21.15.52:0/540317060 shutdown_connections 2024-04-27T01:58:59.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:58:59.784+0000 7f6e92b59700 1 -- 172.21.15.52:0/540317060 wait complete. 2024-04-27T01:58:59.766 INFO:teuthology.orchestra.run.smithi052.stderr:dumped monmap epoch 1 2024-04-27T01:58:59.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:59 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/540317060' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:58:59.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:59 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:58:59.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:58:59 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:00.867 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:00 smithi029 ceph-mon[26769]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:00.867 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:00 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:00.867 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:00 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-27T01:59:01.339 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:59:01.340 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:59:01.557 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:59:01.607 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:59:01.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:01 smithi029 ceph-mon[26769]: from='mgr.14164 172.21.15.29:0/4183685817' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-27T01:59:01.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:01 smithi029 ceph-mon[26769]: mgrmap e14: smithi029.ewvdzs(active, since 63s) 2024-04-27T01:59:02.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.006+0000 7fa09575a700 1 -- 172.21.15.52:0/2438567325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0900ffe20 msgr2=0x7fa090100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:02.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.006+0000 7fa09575a700 1 --2- 172.21.15.52:0/2438567325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0900ffe20 0x7fa090100230 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa078009910 tx=0x7fa078009c20 comp rx=0 tx=0).stop 2024-04-27T01:59:02.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.006+0000 7fa09575a700 1 -- 172.21.15.52:0/2438567325 shutdown_connections 2024-04-27T01:59:02.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.006+0000 7fa09575a700 1 --2- 172.21.15.52:0/2438567325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0900ffe20 0x7fa090100230 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:02.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.006+0000 7fa09575a700 1 -- 172.21.15.52:0/2438567325 >> 172.21.15.52:0/2438567325 conn(0x7fa0900fb3d0 msgr2=0x7fa0900fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:02.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.006+0000 7fa09575a700 1 -- 172.21.15.52:0/2438567325 shutdown_connections 2024-04-27T01:59:02.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.006+0000 7fa09575a700 1 -- 172.21.15.52:0/2438567325 wait complete. 2024-04-27T01:59:02.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.007+0000 7fa09575a700 1 Processor -- start 2024-04-27T01:59:02.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.007+0000 7fa09575a700 1 -- start start 2024-04-27T01:59:02.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.007+0000 7fa09575a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0900ffe20 0x7fa090194f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:02.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.007+0000 7fa09575a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0901954a0 con 0x7fa0900ffe20 2024-04-27T01:59:02.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.008+0000 7fa08effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0900ffe20 0x7fa090194f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:02.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.008+0000 7fa08effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0900ffe20 0x7fa090194f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.52:38610/0 (socket says 172.21.15.52:38610) 2024-04-27T01:59:02.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.008+0000 7fa08effd700 1 -- 172.21.15.52:0/3275110341 learned_addr learned my addr 172.21.15.52:0/3275110341 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T01:59:02.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.008+0000 7fa08effd700 1 -- 172.21.15.52:0/3275110341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0780095f0 con 0x7fa0900ffe20 2024-04-27T01:59:02.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.008+0000 7fa08effd700 1 --2- 172.21.15.52:0/3275110341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0900ffe20 0x7fa090194f60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa078005b40 tx=0x7fa078004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:59:02.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.008+0000 7fa087fff700 1 -- 172.21.15.52:0/3275110341 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa07801d070 con 0x7fa0900ffe20 2024-04-27T01:59:02.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.008+0000 7fa087fff700 1 -- 172.21.15.52:0/3275110341 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa07800be10 con 0x7fa0900ffe20 2024-04-27T01:59:02.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.008+0000 7fa09575a700 1 -- 172.21.15.52:0/3275110341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0901956a0 con 0x7fa0900ffe20 2024-04-27T01:59:02.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.008+0000 7fa087fff700 1 -- 172.21.15.52:0/3275110341 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa07800f460 con 0x7fa0900ffe20 2024-04-27T01:59:02.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.008+0000 7fa09575a700 1 -- 172.21.15.52:0/3275110341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa090195b40 con 0x7fa0900ffe20 2024-04-27T01:59:02.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.009+0000 7fa087fff700 1 -- 172.21.15.52:0/3275110341 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 14) v1 ==== 45154+0+0 (secure 0 0 0) 0x7fa07800f6d0 con 0x7fa0900ffe20 2024-04-27T01:59:02.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.009+0000 7fa087fff700 1 --2- 172.21.15.52:0/3275110341 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fa07c0384e0 0x7fa07c03a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:02.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.009+0000 7fa087fff700 1 -- 172.21.15.52:0/3275110341 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa07804c3b0 con 0x7fa0900ffe20 2024-04-27T01:59:02.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.009+0000 7fa09575a700 1 -- 172.21.15.52:0/3275110341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0901008b0 con 0x7fa0900ffe20 2024-04-27T01:59:02.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.009+0000 7fa08e7fc700 1 -- 172.21.15.52:0/3275110341 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fa07c0384e0 msgr2=0x7fa07c03a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/2469926221 2024-04-27T01:59:02.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.009+0000 7fa08e7fc700 1 --2- 172.21.15.52:0/3275110341 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fa07c0384e0 0x7fa07c03a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-27T01:59:02.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.014+0000 7fa087fff700 1 -- 172.21.15.52:0/3275110341 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa0901008b0 con 0x7fa0900ffe20 2024-04-27T01:59:03.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.210+0000 7fa08e7fc700 1 -- 172.21.15.52:0/3275110341 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fa07c0384e0 msgr2=0x7fa07c03a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/2469926221 2024-04-27T01:59:03.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.210+0000 7fa08e7fc700 1 --2- 172.21.15.52:0/3275110341 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fa07c0384e0 0x7fa07c03a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-27T01:59:03.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.292+0000 7fa09575a700 1 -- 172.21.15.52:0/3275110341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0901008b0 con 0x7fa0900ffe20 2024-04-27T01:59:03.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.293+0000 7fa087fff700 1 -- 172.21.15.52:0/3275110341 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fa0901008b0 con 0x7fa0900ffe20 2024-04-27T01:59:03.274 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:59:03.274 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":1,"fsid":"4c587076-0439-11ef-bc93-c7b262605968","modified":"2024-04-27T01:56:49.232926Z","created":"2024-04-27T01:56:49.232926Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:59:03.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.295+0000 7fa09575a700 1 -- 172.21.15.52:0/3275110341 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fa07c0384e0 msgr2=0x7fa07c03a990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-27T01:59:03.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.295+0000 7fa09575a700 1 --2- 172.21.15.52:0/3275110341 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fa07c0384e0 0x7fa07c03a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:03.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.295+0000 7fa09575a700 1 -- 172.21.15.52:0/3275110341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0900ffe20 msgr2=0x7fa090194f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:03.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.295+0000 7fa09575a700 1 --2- 172.21.15.52:0/3275110341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0900ffe20 0x7fa090194f60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa078005b40 tx=0x7fa078004740 comp rx=0 tx=0).stop 2024-04-27T01:59:03.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.295+0000 7fa09575a700 1 -- 172.21.15.52:0/3275110341 shutdown_connections 2024-04-27T01:59:03.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.295+0000 7fa09575a700 1 --2- 172.21.15.52:0/3275110341 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7fa07c0384e0 0x7fa07c03a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:03.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.295+0000 7fa09575a700 1 --2- 172.21.15.52:0/3275110341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0900ffe20 0x7fa090194f60 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:03.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.295+0000 7fa09575a700 1 -- 172.21.15.52:0/3275110341 >> 172.21.15.52:0/3275110341 conn(0x7fa0900fb3d0 msgr2=0x7fa0900fc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:03.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.295+0000 7fa09575a700 1 -- 172.21.15.52:0/3275110341 shutdown_connections 2024-04-27T01:59:03.277 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:03.295+0000 7fa09575a700 1 -- 172.21.15.52:0/3275110341 wait complete. 2024-04-27T01:59:03.277 INFO:teuthology.orchestra.run.smithi052.stderr:dumped monmap epoch 1 2024-04-27T01:59:03.576 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:03 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/3275110341' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:59:04.868 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:59:04.868 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:59:05.076 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:59:05.123 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:59:06.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.604+0000 7f8e33b05700 1 -- 172.21.15.52:0/4074657180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e2c0ffe20 msgr2=0x7f8e2c100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:06.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.604+0000 7f8e33b05700 1 --2- 172.21.15.52:0/4074657180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e2c0ffe20 0x7f8e2c100230 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8e28009910 tx=0x7f8e28009c20 comp rx=0 tx=0).stop 2024-04-27T01:59:06.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.604+0000 7f8e33b05700 1 -- 172.21.15.52:0/4074657180 shutdown_connections 2024-04-27T01:59:06.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.604+0000 7f8e33b05700 1 --2- 172.21.15.52:0/4074657180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e2c0ffe20 0x7f8e2c100230 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:06.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.604+0000 7f8e33b05700 1 -- 172.21.15.52:0/4074657180 >> 172.21.15.52:0/4074657180 conn(0x7f8e2c0fb3d0 msgr2=0x7f8e2c0fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:06.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.604+0000 7f8e33b05700 1 -- 172.21.15.52:0/4074657180 shutdown_connections 2024-04-27T01:59:06.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.604+0000 7f8e33b05700 1 -- 172.21.15.52:0/4074657180 wait complete. 2024-04-27T01:59:06.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.605+0000 7f8e33b05700 1 Processor -- start 2024-04-27T01:59:06.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.605+0000 7f8e33b05700 1 -- start start 2024-04-27T01:59:06.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.605+0000 7f8e33b05700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e2c0ffe20 0x7f8e2c194f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:06.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.605+0000 7f8e33b05700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c1954a0 con 0x7f8e2c0ffe20 2024-04-27T01:59:06.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.605+0000 7f8e318a1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e2c0ffe20 0x7f8e2c194f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:06.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.605+0000 7f8e318a1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e2c0ffe20 0x7f8e2c194f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.52:51676/0 (socket says 172.21.15.52:51676) 2024-04-27T01:59:06.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.605+0000 7f8e318a1700 1 -- 172.21.15.52:0/2231167328 learned_addr learned my addr 172.21.15.52:0/2231167328 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T01:59:06.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.606+0000 7f8e318a1700 1 -- 172.21.15.52:0/2231167328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e280095f0 con 0x7f8e2c0ffe20 2024-04-27T01:59:06.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.606+0000 7f8e318a1700 1 --2- 172.21.15.52:0/2231167328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e2c0ffe20 0x7f8e2c194f60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8e280092e0 tx=0x7f8e28009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:59:06.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.606+0000 7f8e22ffd700 1 -- 172.21.15.52:0/2231167328 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e2801d070 con 0x7f8e2c0ffe20 2024-04-27T01:59:06.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.606+0000 7f8e33b05700 1 -- 172.21.15.52:0/2231167328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e2c1956a0 con 0x7f8e2c0ffe20 2024-04-27T01:59:06.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.606+0000 7f8e22ffd700 1 -- 172.21.15.52:0/2231167328 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8e2800be10 con 0x7f8e2c0ffe20 2024-04-27T01:59:06.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.606+0000 7f8e22ffd700 1 -- 172.21.15.52:0/2231167328 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e2800f460 con 0x7f8e2c0ffe20 2024-04-27T01:59:06.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.606+0000 7f8e33b05700 1 -- 172.21.15.52:0/2231167328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e2c195b40 con 0x7f8e2c0ffe20 2024-04-27T01:59:06.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.607+0000 7f8e22ffd700 1 -- 172.21.15.52:0/2231167328 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 14) v1 ==== 45154+0+0 (secure 0 0 0) 0x7f8e2800b960 con 0x7f8e2c0ffe20 2024-04-27T01:59:06.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.607+0000 7f8e33b05700 1 -- 172.21.15.52:0/2231167328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e2c18ea80 con 0x7f8e2c0ffe20 2024-04-27T01:59:06.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.607+0000 7f8e22ffd700 1 --2- 172.21.15.52:0/2231167328 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f8e180384e0 0x7f8e1803a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:06.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.607+0000 7f8e22ffd700 1 -- 172.21.15.52:0/2231167328 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8e2804c420 con 0x7f8e2c0ffe20 2024-04-27T01:59:06.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.607+0000 7f8e310a0700 1 -- 172.21.15.52:0/2231167328 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f8e180384e0 msgr2=0x7f8e1803a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/2469926221 2024-04-27T01:59:06.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.607+0000 7f8e310a0700 1 --2- 172.21.15.52:0/2231167328 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f8e180384e0 0x7f8e1803a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-27T01:59:06.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.616+0000 7f8e22ffd700 1 -- 172.21.15.52:0/2231167328 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8e28022790 con 0x7f8e2c0ffe20 2024-04-27T01:59:06.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.808+0000 7f8e310a0700 1 -- 172.21.15.52:0/2231167328 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f8e180384e0 msgr2=0x7f8e1803a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/2469926221 2024-04-27T01:59:06.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.808+0000 7f8e310a0700 1 --2- 172.21.15.52:0/2231167328 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f8e180384e0 0x7f8e1803a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-27T01:59:06.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.899+0000 7f8e33b05700 1 -- 172.21.15.52:0/2231167328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8e2c02cc30 con 0x7f8e2c0ffe20 2024-04-27T01:59:06.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.899+0000 7f8e22ffd700 1 -- 172.21.15.52:0/2231167328 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f8e28027070 con 0x7f8e2c0ffe20 2024-04-27T01:59:06.881 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:59:06.881 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":1,"fsid":"4c587076-0439-11ef-bc93-c7b262605968","modified":"2024-04-27T01:56:49.232926Z","created":"2024-04-27T01:56:49.232926Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:59:06.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.901+0000 7f8e33b05700 1 -- 172.21.15.52:0/2231167328 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f8e180384e0 msgr2=0x7f8e1803a990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-27T01:59:06.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.901+0000 7f8e33b05700 1 --2- 172.21.15.52:0/2231167328 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f8e180384e0 0x7f8e1803a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:06.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.901+0000 7f8e33b05700 1 -- 172.21.15.52:0/2231167328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e2c0ffe20 msgr2=0x7f8e2c194f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:06.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.901+0000 7f8e33b05700 1 --2- 172.21.15.52:0/2231167328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e2c0ffe20 0x7f8e2c194f60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8e280092e0 tx=0x7f8e28009e60 comp rx=0 tx=0).stop 2024-04-27T01:59:06.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.901+0000 7f8e33b05700 1 -- 172.21.15.52:0/2231167328 shutdown_connections 2024-04-27T01:59:06.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.901+0000 7f8e33b05700 1 --2- 172.21.15.52:0/2231167328 >> [v2:172.21.15.29:6800/2469926221,v1:172.21.15.29:6801/2469926221] conn(0x7f8e180384e0 0x7f8e1803a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:06.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.901+0000 7f8e33b05700 1 --2- 172.21.15.52:0/2231167328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e2c0ffe20 0x7f8e2c194f60 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:06.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.901+0000 7f8e33b05700 1 -- 172.21.15.52:0/2231167328 >> 172.21.15.52:0/2231167328 conn(0x7f8e2c0fb3d0 msgr2=0x7f8e2c0fc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:06.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.901+0000 7f8e33b05700 1 -- 172.21.15.52:0/2231167328 shutdown_connections 2024-04-27T01:59:06.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:06.901+0000 7f8e33b05700 1 -- 172.21.15.52:0/2231167328 wait complete. 2024-04-27T01:59:06.884 INFO:teuthology.orchestra.run.smithi052.stderr:dumped monmap epoch 1 2024-04-27T01:59:07.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:06 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/2231167328' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:59:08.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:07 smithi029 ceph-mon[26769]: Active manager daemon smithi029.ewvdzs restarted 2024-04-27T01:59:08.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:07 smithi029 ceph-mon[26769]: Activating manager daemon smithi029.ewvdzs 2024-04-27T01:59:08.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:07 smithi029 ceph-mon[26769]: osdmap e5: 0 total, 0 up, 0 in 2024-04-27T01:59:08.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:07 smithi029 ceph-mon[26769]: mgrmap e15: smithi029.ewvdzs(active, starting, since 0.00320033s) 2024-04-27T01:59:08.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:07 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-04-27T01:59:08.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:07 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mgr metadata", "who": "smithi029.ewvdzs", "id": "smithi029.ewvdzs"}]: dispatch 2024-04-27T01:59:08.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:07 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T01:59:08.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:07 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T01:59:08.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:07 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T01:59:08.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:07 smithi029 ceph-mon[26769]: Manager daemon smithi029.ewvdzs is now available 2024-04-27T01:59:08.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:07 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:59:08.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:07 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:59:08.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:07 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:08.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:07 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:59:08.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:07 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.ewvdzs/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:59:08.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:07 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.ewvdzs/trash_purge_schedule"}]: dispatch 2024-04-27T01:59:08.497 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:59:08.497 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:59:08.706 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:59:08.764 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:59:09.697 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:09 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:09.698 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:09 smithi029 ceph-mon[26769]: [27/Apr/2024:01:59:08] ENGINE Bus STARTING 2024-04-27T01:59:09.698 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:09 smithi029 ceph-mon[26769]: mgrmap e16: smithi029.ewvdzs(active, since 1.00628s) 2024-04-27T01:59:09.698 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:09 smithi029 ceph-mon[26769]: [27/Apr/2024:01:59:08] ENGINE Serving on http://172.21.15.29:8765 2024-04-27T01:59:09.698 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:09 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:10.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.362+0000 7f63b19e4700 1 -- 172.21.15.52:0/2559609499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63ac0ffe20 msgr2=0x7f63ac100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:10.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.362+0000 7f63b19e4700 1 --2- 172.21.15.52:0/2559609499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63ac0ffe20 0x7f63ac100230 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f639c009910 tx=0x7f639c009c20 comp rx=0 tx=0).stop 2024-04-27T01:59:10.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.362+0000 7f63b19e4700 1 -- 172.21.15.52:0/2559609499 shutdown_connections 2024-04-27T01:59:10.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.362+0000 7f63b19e4700 1 --2- 172.21.15.52:0/2559609499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63ac0ffe20 0x7f63ac100230 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:10.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.362+0000 7f63b19e4700 1 -- 172.21.15.52:0/2559609499 >> 172.21.15.52:0/2559609499 conn(0x7f63ac0fb3d0 msgr2=0x7f63ac0fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:10.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.362+0000 7f63b19e4700 1 -- 172.21.15.52:0/2559609499 shutdown_connections 2024-04-27T01:59:10.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.362+0000 7f63b19e4700 1 -- 172.21.15.52:0/2559609499 wait complete. 2024-04-27T01:59:10.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.363+0000 7f63b19e4700 1 Processor -- start 2024-04-27T01:59:10.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.363+0000 7f63b19e4700 1 -- start start 2024-04-27T01:59:10.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.363+0000 7f63b19e4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63ac0ffe20 0x7f63ac194f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:10.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.363+0000 7f63b19e4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac1954a0 con 0x7f63ac0ffe20 2024-04-27T01:59:10.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.363+0000 7f63aaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63ac0ffe20 0x7f63ac194f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:10.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.363+0000 7f63aaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63ac0ffe20 0x7f63ac194f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.52:51690/0 (socket says 172.21.15.52:51690) 2024-04-27T01:59:10.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.363+0000 7f63aaffd700 1 -- 172.21.15.52:0/1790440584 learned_addr learned my addr 172.21.15.52:0/1790440584 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T01:59:10.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.364+0000 7f63aaffd700 1 -- 172.21.15.52:0/1790440584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639c0095f0 con 0x7f63ac0ffe20 2024-04-27T01:59:10.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.364+0000 7f63aaffd700 1 --2- 172.21.15.52:0/1790440584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63ac0ffe20 0x7f63ac194f60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f639c0092e0 tx=0x7f639c009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:59:10.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.364+0000 7f63b09e2700 1 -- 172.21.15.52:0/1790440584 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f639c01d070 con 0x7f63ac0ffe20 2024-04-27T01:59:10.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.364+0000 7f63b09e2700 1 -- 172.21.15.52:0/1790440584 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f639c00be10 con 0x7f63ac0ffe20 2024-04-27T01:59:10.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.364+0000 7f63b19e4700 1 -- 172.21.15.52:0/1790440584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63ac1956a0 con 0x7f63ac0ffe20 2024-04-27T01:59:10.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.364+0000 7f63b09e2700 1 -- 172.21.15.52:0/1790440584 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f639c00f460 con 0x7f63ac0ffe20 2024-04-27T01:59:10.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.364+0000 7f63b19e4700 1 -- 172.21.15.52:0/1790440584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63ac195b40 con 0x7f63ac0ffe20 2024-04-27T01:59:10.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.365+0000 7f63b09e2700 1 -- 172.21.15.52:0/1790440584 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 45005+0+0 (secure 0 0 0) 0x7f639c00b960 con 0x7f63ac0ffe20 2024-04-27T01:59:10.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.365+0000 7f63b19e4700 1 -- 172.21.15.52:0/1790440584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63ac18ea80 con 0x7f63ac0ffe20 2024-04-27T01:59:10.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.365+0000 7f63b09e2700 1 --2- 172.21.15.52:0/1790440584 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f6394038380 0x7f639403a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:10.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.365+0000 7f63b09e2700 1 -- 172.21.15.52:0/1790440584 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f639c04c1d0 con 0x7f63ac0ffe20 2024-04-27T01:59:10.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.365+0000 7f63aa7fc700 1 --2- 172.21.15.52:0/1790440584 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f6394038380 0x7f639403a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:10.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.366+0000 7f63aa7fc700 1 --2- 172.21.15.52:0/1790440584 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f6394038380 0x7f639403a830 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f63a0006fd0 tx=0x7f63a0006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:59:10.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.371+0000 7f63b09e2700 1 -- 172.21.15.52:0/1790440584 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f639c022790 con 0x7f63ac0ffe20 2024-04-27T01:59:10.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.463+0000 7f63b09e2700 1 -- 172.21.15.52:0/1790440584 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 17) v1 ==== 45197+0+0 (secure 0 0 0) 0x7f639c017d20 con 0x7f63ac0ffe20 2024-04-27T01:59:10.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.656+0000 7f63b19e4700 1 -- 172.21.15.52:0/1790440584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63ac02cc30 con 0x7f63ac0ffe20 2024-04-27T01:59:10.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.657+0000 7f63b09e2700 1 -- 172.21.15.52:0/1790440584 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f639c027090 con 0x7f63ac0ffe20 2024-04-27T01:59:10.638 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:59:10.638 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":1,"fsid":"4c587076-0439-11ef-bc93-c7b262605968","modified":"2024-04-27T01:56:49.232926Z","created":"2024-04-27T01:56:49.232926Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:59:10.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.659+0000 7f63b19e4700 1 -- 172.21.15.52:0/1790440584 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f6394038380 msgr2=0x7f639403a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:10.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.659+0000 7f63b19e4700 1 --2- 172.21.15.52:0/1790440584 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f6394038380 0x7f639403a830 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f63a0006fd0 tx=0x7f63a0006e40 comp rx=0 tx=0).stop 2024-04-27T01:59:10.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.659+0000 7f63b19e4700 1 -- 172.21.15.52:0/1790440584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63ac0ffe20 msgr2=0x7f63ac194f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:10.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.659+0000 7f63b19e4700 1 --2- 172.21.15.52:0/1790440584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63ac0ffe20 0x7f63ac194f60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f639c0092e0 tx=0x7f639c009e60 comp rx=0 tx=0).stop 2024-04-27T01:59:10.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.659+0000 7f63b19e4700 1 -- 172.21.15.52:0/1790440584 shutdown_connections 2024-04-27T01:59:10.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.659+0000 7f63b19e4700 1 --2- 172.21.15.52:0/1790440584 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f6394038380 0x7f639403a830 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:10.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.659+0000 7f63b19e4700 1 --2- 172.21.15.52:0/1790440584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63ac0ffe20 0x7f63ac194f60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:10.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.659+0000 7f63b19e4700 1 -- 172.21.15.52:0/1790440584 >> 172.21.15.52:0/1790440584 conn(0x7f63ac0fb3d0 msgr2=0x7f63ac0fc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:10.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.659+0000 7f63b19e4700 1 -- 172.21.15.52:0/1790440584 shutdown_connections 2024-04-27T01:59:10.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:10.660+0000 7f63b19e4700 1 -- 172.21.15.52:0/1790440584 wait complete. 2024-04-27T01:59:10.641 INFO:teuthology.orchestra.run.smithi052.stderr:dumped monmap epoch 1 2024-04-27T01:59:10.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:10 smithi029 ceph-mon[26769]: [27/Apr/2024:01:59:08] ENGINE Serving on https://172.21.15.29:7150 2024-04-27T01:59:10.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:10 smithi029 ceph-mon[26769]: [27/Apr/2024:01:59:08] ENGINE Bus STARTED 2024-04-27T01:59:11.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:11 smithi029 ceph-mon[26769]: mgrmap e17: smithi029.ewvdzs(active, since 2s) 2024-04-27T01:59:11.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:11 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:11.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:11 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/1790440584' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:59:12.240 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:59:12.241 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:59:12.736 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:59:12.791 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-27T01:59:14.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:14 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:14.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:14 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:14.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:14 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:14.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:14 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:14.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:14 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:59:14.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:14 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:14.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:14 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:14.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.012+0000 7f9548c10700 1 -- 172.21.15.52:0/782208849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95440ffe20 msgr2=0x7f9544100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:14.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.012+0000 7f9548c10700 1 --2- 172.21.15.52:0/782208849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95440ffe20 0x7f9544100230 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9534009910 tx=0x7f9534009c20 comp rx=0 tx=0).stop 2024-04-27T01:59:14.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.012+0000 7f9548c10700 1 -- 172.21.15.52:0/782208849 shutdown_connections 2024-04-27T01:59:14.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.012+0000 7f9548c10700 1 --2- 172.21.15.52:0/782208849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95440ffe20 0x7f9544100230 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:14.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.012+0000 7f9548c10700 1 -- 172.21.15.52:0/782208849 >> 172.21.15.52:0/782208849 conn(0x7f95440fb3d0 msgr2=0x7f95440fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:14.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.012+0000 7f9548c10700 1 -- 172.21.15.52:0/782208849 shutdown_connections 2024-04-27T01:59:14.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.012+0000 7f9548c10700 1 -- 172.21.15.52:0/782208849 wait complete. 2024-04-27T01:59:14.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.013+0000 7f9548c10700 1 Processor -- start 2024-04-27T01:59:14.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.013+0000 7f9548c10700 1 -- start start 2024-04-27T01:59:14.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.013+0000 7f9548c10700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95440ffe20 0x7f9544197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:14.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.013+0000 7f9548c10700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95441976a0 con 0x7f95440ffe20 2024-04-27T01:59:14.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.013+0000 7f954259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95440ffe20 0x7f9544197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:14.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.013+0000 7f954259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95440ffe20 0x7f9544197160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.52:51698/0 (socket says 172.21.15.52:51698) 2024-04-27T01:59:14.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.013+0000 7f954259c700 1 -- 172.21.15.52:0/178700363 learned_addr learned my addr 172.21.15.52:0/178700363 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T01:59:14.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.014+0000 7f954259c700 1 -- 172.21.15.52:0/178700363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95340095f0 con 0x7f95440ffe20 2024-04-27T01:59:14.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.014+0000 7f954259c700 1 --2- 172.21.15.52:0/178700363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95440ffe20 0x7f9544197160 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9534005b40 tx=0x7f9534004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:59:14.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.014+0000 7f95337fe700 1 -- 172.21.15.52:0/178700363 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f953401d070 con 0x7f95440ffe20 2024-04-27T01:59:14.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.014+0000 7f95337fe700 1 -- 172.21.15.52:0/178700363 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f953400be10 con 0x7f95440ffe20 2024-04-27T01:59:14.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.014+0000 7f9548c10700 1 -- 172.21.15.52:0/178700363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95441978a0 con 0x7f95440ffe20 2024-04-27T01:59:14.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.014+0000 7f95337fe700 1 -- 172.21.15.52:0/178700363 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f953400f460 con 0x7f95440ffe20 2024-04-27T01:59:14.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.014+0000 7f9548c10700 1 -- 172.21.15.52:0/178700363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9544197d40 con 0x7f95440ffe20 2024-04-27T01:59:14.998 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.015+0000 7f95337fe700 1 -- 172.21.15.52:0/178700363 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 45197+0+0 (secure 0 0 0) 0x7f953400b960 con 0x7f95440ffe20 2024-04-27T01:59:14.998 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.015+0000 7f9548c10700 1 -- 172.21.15.52:0/178700363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9544190e80 con 0x7f95440ffe20 2024-04-27T01:59:14.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.015+0000 7f95337fe700 1 --2- 172.21.15.52:0/178700363 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f952c0385d0 0x7f952c03aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:14.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.015+0000 7f95337fe700 1 -- 172.21.15.52:0/178700363 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f953404c530 con 0x7f95440ffe20 2024-04-27T01:59:14.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.015+0000 7f9541d9b700 1 --2- 172.21.15.52:0/178700363 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f952c0385d0 0x7f952c03aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:14.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.016+0000 7f9541d9b700 1 --2- 172.21.15.52:0/178700363 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f952c0385d0 0x7f952c03aa80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9538006fd0 tx=0x7f9538006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:59:15.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.020+0000 7f95337fe700 1 -- 172.21.15.52:0/178700363 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f953402bba0 con 0x7f95440ffe20 2024-04-27T01:59:15.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.306+0000 7f9548c10700 1 -- 172.21.15.52:0/178700363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f954402cc30 con 0x7f95440ffe20 2024-04-27T01:59:15.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.307+0000 7f95337fe700 1 -- 172.21.15.52:0/178700363 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f9534027070 con 0x7f95440ffe20 2024-04-27T01:59:15.288 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:59:15.288 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":1,"fsid":"4c587076-0439-11ef-bc93-c7b262605968","modified":"2024-04-27T01:56:49.232926Z","created":"2024-04-27T01:56:49.232926Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:59:15.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.309+0000 7f9548c10700 1 -- 172.21.15.52:0/178700363 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f952c0385d0 msgr2=0x7f952c03aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:15.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.309+0000 7f9548c10700 1 --2- 172.21.15.52:0/178700363 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f952c0385d0 0x7f952c03aa80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9538006fd0 tx=0x7f9538006e40 comp rx=0 tx=0).stop 2024-04-27T01:59:15.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.309+0000 7f9548c10700 1 -- 172.21.15.52:0/178700363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95440ffe20 msgr2=0x7f9544197160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:15.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.309+0000 7f9548c10700 1 --2- 172.21.15.52:0/178700363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95440ffe20 0x7f9544197160 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9534005b40 tx=0x7f9534004740 comp rx=0 tx=0).stop 2024-04-27T01:59:15.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.309+0000 7f9548c10700 1 -- 172.21.15.52:0/178700363 shutdown_connections 2024-04-27T01:59:15.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.309+0000 7f9548c10700 1 --2- 172.21.15.52:0/178700363 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f952c0385d0 0x7f952c03aa80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:15.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.309+0000 7f9548c10700 1 --2- 172.21.15.52:0/178700363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95440ffe20 0x7f9544197160 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:15.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.309+0000 7f9548c10700 1 -- 172.21.15.52:0/178700363 >> 172.21.15.52:0/178700363 conn(0x7f95440fb3d0 msgr2=0x7f95440fc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:15.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.309+0000 7f9548c10700 1 -- 172.21.15.52:0/178700363 shutdown_connections 2024-04-27T01:59:15.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:15.309+0000 7f9548c10700 1 -- 172.21.15.52:0/178700363 wait complete. 2024-04-27T01:59:15.291 INFO:teuthology.orchestra.run.smithi052.stderr:dumped monmap epoch 1 2024-04-27T01:59:16.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:16 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:16.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:16 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:16.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:16 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd/host:smithi029", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:59:16.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:16 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:59:16.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:16 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:59:16.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:16 smithi029 ceph-mon[26769]: Updating smithi029:/etc/ceph/ceph.conf 2024-04-27T01:59:16.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:16 smithi029 ceph-mon[26769]: Updating smithi052:/etc/ceph/ceph.conf 2024-04-27T01:59:16.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:16 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/178700363' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:59:16.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:16 smithi029 ceph-mon[26769]: Updating smithi052:/var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:59:16.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:16 smithi029 ceph-mon[26769]: Updating smithi029:/var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:59:16.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:16 smithi029 ceph-mon[26769]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:59:16.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:16 smithi029 ceph-mon[26769]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:59:17.001 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:59:17.002 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:59:17.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:17 smithi029 ceph-mon[26769]: Updating smithi052:/var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:59:17.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:17 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:17.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:17 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:17.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:17 smithi029 ceph-mon[26769]: Updating smithi029:/var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:59:17.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:17 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:17.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:17 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:17.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:17 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:17.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:17 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi052", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-27T01:59:17.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:17 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi052", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-27T01:59:17.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:17 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:59:17.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:17 smithi029 ceph-mon[26769]: Deploying daemon ceph-exporter.smithi052 on smithi052 2024-04-27T01:59:17.911 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:59:19.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.773+0000 7f98a2218700 1 -- 172.21.15.52:0/2415685881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c0fefa0 msgr2=0x7f989c0ff370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:19.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.773+0000 7f98a2218700 1 --2- 172.21.15.52:0/2415685881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c0fefa0 0x7f989c0ff370 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f988c009910 tx=0x7f988c009c20 comp rx=0 tx=0).stop 2024-04-27T01:59:19.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.773+0000 7f98a2218700 1 -- 172.21.15.52:0/2415685881 shutdown_connections 2024-04-27T01:59:19.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.773+0000 7f98a2218700 1 --2- 172.21.15.52:0/2415685881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c0fefa0 0x7f989c0ff370 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:19.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.773+0000 7f98a2218700 1 -- 172.21.15.52:0/2415685881 >> 172.21.15.52:0/2415685881 conn(0x7f989c0fa860 msgr2=0x7f989c0fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:19.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.774+0000 7f98a2218700 1 -- 172.21.15.52:0/2415685881 shutdown_connections 2024-04-27T01:59:19.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.774+0000 7f98a2218700 1 -- 172.21.15.52:0/2415685881 wait complete. 2024-04-27T01:59:19.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.774+0000 7f98a2218700 1 Processor -- start 2024-04-27T01:59:19.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.774+0000 7f98a2218700 1 -- start start 2024-04-27T01:59:19.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.775+0000 7f98a2218700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c0fefa0 0x7f989c197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:19.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.775+0000 7f98a2218700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989c197aa0 con 0x7f989c0fefa0 2024-04-27T01:59:19.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.775+0000 7f989b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c0fefa0 0x7f989c197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:19.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.775+0000 7f989b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c0fefa0 0x7f989c197560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.52:36288/0 (socket says 172.21.15.52:36288) 2024-04-27T01:59:19.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.775+0000 7f989b7fe700 1 -- 172.21.15.52:0/313018166 learned_addr learned my addr 172.21.15.52:0/313018166 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T01:59:19.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.775+0000 7f989b7fe700 1 -- 172.21.15.52:0/313018166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c0095f0 con 0x7f989c0fefa0 2024-04-27T01:59:19.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.775+0000 7f989b7fe700 1 --2- 172.21.15.52:0/313018166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c0fefa0 0x7f989c197560 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f988c005b40 tx=0x7f988c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:59:19.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.776+0000 7f9898ff9700 1 -- 172.21.15.52:0/313018166 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f988c01d070 con 0x7f989c0fefa0 2024-04-27T01:59:19.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.776+0000 7f9898ff9700 1 -- 172.21.15.52:0/313018166 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f988c0044d0 con 0x7f989c0fefa0 2024-04-27T01:59:19.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.776+0000 7f98a2218700 1 -- 172.21.15.52:0/313018166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f989c197ca0 con 0x7f989c0fefa0 2024-04-27T01:59:19.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.776+0000 7f9898ff9700 1 -- 172.21.15.52:0/313018166 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f988c003930 con 0x7f989c0fefa0 2024-04-27T01:59:19.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.776+0000 7f98a2218700 1 -- 172.21.15.52:0/313018166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f989c198060 con 0x7f989c0fefa0 2024-04-27T01:59:19.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.777+0000 7f9898ff9700 1 -- 172.21.15.52:0/313018166 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 45197+0+0 (secure 0 0 0) 0x7f988c003bf0 con 0x7f989c0fefa0 2024-04-27T01:59:19.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.777+0000 7f98a2218700 1 -- 172.21.15.52:0/313018166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f989c062380 con 0x7f989c0fefa0 2024-04-27T01:59:19.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.777+0000 7f9898ff9700 1 --2- 172.21.15.52:0/313018166 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f9884038580 0x7f988403aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:19.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.777+0000 7f9898ff9700 1 -- 172.21.15.52:0/313018166 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f988c04c660 con 0x7f989c0fefa0 2024-04-27T01:59:19.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.777+0000 7f989affd700 1 --2- 172.21.15.52:0/313018166 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f9884038580 0x7f988403aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:19.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.778+0000 7f989affd700 1 --2- 172.21.15.52:0/313018166 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f9884038580 0x7f988403aa30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9890006fd0 tx=0x7f9890006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:59:19.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:19.782+0000 7f9898ff9700 1 -- 172.21.15.52:0/313018166 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f988c02a950 con 0x7f989c0fefa0 2024-04-27T01:59:20.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:20.085+0000 7f98a2218700 1 -- 172.21.15.52:0/313018166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f989c105d30 con 0x7f989c0fefa0 2024-04-27T01:59:20.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:20.086+0000 7f9898ff9700 1 -- 172.21.15.52:0/313018166 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f988c02a360 con 0x7f989c0fefa0 2024-04-27T01:59:20.067 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:59:20.067 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":1,"fsid":"4c587076-0439-11ef-bc93-c7b262605968","modified":"2024-04-27T01:56:49.232926Z","created":"2024-04-27T01:56:49.232926Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:59:20.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:20.088+0000 7f98a2218700 1 -- 172.21.15.52:0/313018166 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f9884038580 msgr2=0x7f988403aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:20.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:20.088+0000 7f98a2218700 1 --2- 172.21.15.52:0/313018166 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f9884038580 0x7f988403aa30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9890006fd0 tx=0x7f9890006e40 comp rx=0 tx=0).stop 2024-04-27T01:59:20.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:20.088+0000 7f98a2218700 1 -- 172.21.15.52:0/313018166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c0fefa0 msgr2=0x7f989c197560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:20.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:20.088+0000 7f98a2218700 1 --2- 172.21.15.52:0/313018166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c0fefa0 0x7f989c197560 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f988c005b40 tx=0x7f988c004c80 comp rx=0 tx=0).stop 2024-04-27T01:59:20.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:20.088+0000 7f98a2218700 1 -- 172.21.15.52:0/313018166 shutdown_connections 2024-04-27T01:59:20.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:20.088+0000 7f98a2218700 1 --2- 172.21.15.52:0/313018166 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f9884038580 0x7f988403aa30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:20.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:20.088+0000 7f98a2218700 1 --2- 172.21.15.52:0/313018166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c0fefa0 0x7f989c197560 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:20.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:20.089+0000 7f98a2218700 1 -- 172.21.15.52:0/313018166 >> 172.21.15.52:0/313018166 conn(0x7f989c0fa860 msgr2=0x7f989c069110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:20.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:20.089+0000 7f98a2218700 1 -- 172.21.15.52:0/313018166 shutdown_connections 2024-04-27T01:59:20.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:20.089+0000 7f98a2218700 1 -- 172.21.15.52:0/313018166 wait complete. 2024-04-27T01:59:20.071 INFO:teuthology.orchestra.run.smithi052.stderr:dumped monmap epoch 1 2024-04-27T01:59:20.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:20 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/313018166' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:59:21.711 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:59:21.712 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:59:22.326 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:59:22.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:21 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:22.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:21 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:22.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:21 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:22.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:21 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:22.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:21 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi052", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T01:59:22.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:21 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi052", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-27T01:59:22.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:21 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:59:22.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:21 smithi029 ceph-mon[26769]: Deploying daemon crash.smithi052 on smithi052 2024-04-27T01:59:23.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.893+0000 7fa6fe3df700 1 -- 172.21.15.52:0/3135670270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6f8100f30 msgr2=0x7fa6f8101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:23.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.893+0000 7fa6fe3df700 1 --2- 172.21.15.52:0/3135670270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6f8100f30 0x7fa6f8101300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8009910 tx=0x7fa6e8009c20 comp rx=0 tx=0).stop 2024-04-27T01:59:23.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.893+0000 7fa6fe3df700 1 -- 172.21.15.52:0/3135670270 shutdown_connections 2024-04-27T01:59:23.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.893+0000 7fa6fe3df700 1 --2- 172.21.15.52:0/3135670270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6f8100f30 0x7fa6f8101300 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:23.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.893+0000 7fa6fe3df700 1 -- 172.21.15.52:0/3135670270 >> 172.21.15.52:0/3135670270 conn(0x7fa6f8074af0 msgr2=0x7fa6f8074ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:23.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.893+0000 7fa6fe3df700 1 -- 172.21.15.52:0/3135670270 shutdown_connections 2024-04-27T01:59:23.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.893+0000 7fa6fe3df700 1 -- 172.21.15.52:0/3135670270 wait complete. 2024-04-27T01:59:23.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.893+0000 7fa6fe3df700 1 Processor -- start 2024-04-27T01:59:23.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.893+0000 7fa6fe3df700 1 -- start start 2024-04-27T01:59:23.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.894+0000 7fa6fe3df700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6f8100f30 0x7fa6f80fed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:23.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.894+0000 7fa6fe3df700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f8100ce0 con 0x7fa6f8100f30 2024-04-27T01:59:23.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.894+0000 7fa6f7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6f8100f30 0x7fa6f80fed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:23.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.894+0000 7fa6f7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6f8100f30 0x7fa6f80fed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.52:36312/0 (socket says 172.21.15.52:36312) 2024-04-27T01:59:23.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.894+0000 7fa6f7fff700 1 -- 172.21.15.52:0/2119066721 learned_addr learned my addr 172.21.15.52:0/2119066721 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T01:59:23.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.894+0000 7fa6f7fff700 1 -- 172.21.15.52:0/2119066721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e80095f0 con 0x7fa6f8100f30 2024-04-27T01:59:23.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.895+0000 7fa6f7fff700 1 --2- 172.21.15.52:0/2119066721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6f8100f30 0x7fa6f80fed60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8005b40 tx=0x7fa6e8004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:59:23.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.895+0000 7fa6f57fa700 1 -- 172.21.15.52:0/2119066721 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6e801d070 con 0x7fa6f8100f30 2024-04-27T01:59:23.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.895+0000 7fa6fe3df700 1 -- 172.21.15.52:0/2119066721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f80ff2a0 con 0x7fa6f8100f30 2024-04-27T01:59:23.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.895+0000 7fa6f57fa700 1 -- 172.21.15.52:0/2119066721 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa6e80044d0 con 0x7fa6f8100f30 2024-04-27T01:59:23.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.895+0000 7fa6f57fa700 1 -- 172.21.15.52:0/2119066721 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6e8003930 con 0x7fa6f8100f30 2024-04-27T01:59:23.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.895+0000 7fa6fe3df700 1 -- 172.21.15.52:0/2119066721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f80ff720 con 0x7fa6f8100f30 2024-04-27T01:59:23.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.896+0000 7fa6f57fa700 1 -- 172.21.15.52:0/2119066721 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 45197+0+0 (secure 0 0 0) 0x7fa6e8003bf0 con 0x7fa6f8100f30 2024-04-27T01:59:23.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.896+0000 7fa6f57fa700 1 --2- 172.21.15.52:0/2119066721 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa6e4038580 0x7fa6e403aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:23.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.896+0000 7fa6f57fa700 1 -- 172.21.15.52:0/2119066721 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fa6e804c500 con 0x7fa6f8100f30 2024-04-27T01:59:23.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.896+0000 7fa6f77fe700 1 --2- 172.21.15.52:0/2119066721 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa6e4038580 0x7fa6e403aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:23.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.896+0000 7fa6fe3df700 1 -- 172.21.15.52:0/2119066721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6f8107610 con 0x7fa6f8100f30 2024-04-27T01:59:23.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.897+0000 7fa6f77fe700 1 --2- 172.21.15.52:0/2119066721 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa6e4038580 0x7fa6e403aa30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0006fd0 tx=0x7fa6e0006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:59:23.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:23.902+0000 7fa6f57fa700 1 -- 172.21.15.52:0/2119066721 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa6e802a950 con 0x7fa6f8100f30 2024-04-27T01:59:24.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:24.186+0000 7fa6fe3df700 1 -- 172.21.15.52:0/2119066721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6f8062380 con 0x7fa6f8100f30 2024-04-27T01:59:24.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:24.187+0000 7fa6f57fa700 1 -- 172.21.15.52:0/2119066721 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fa6e802a360 con 0x7fa6f8100f30 2024-04-27T01:59:24.170 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:59:24.170 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":1,"fsid":"4c587076-0439-11ef-bc93-c7b262605968","modified":"2024-04-27T01:56:49.232926Z","created":"2024-04-27T01:56:49.232926Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:59:24.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:24.189+0000 7fa6fe3df700 1 -- 172.21.15.52:0/2119066721 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa6e4038580 msgr2=0x7fa6e403aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:24.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:24.189+0000 7fa6fe3df700 1 --2- 172.21.15.52:0/2119066721 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa6e4038580 0x7fa6e403aa30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0006fd0 tx=0x7fa6e0006e40 comp rx=0 tx=0).stop 2024-04-27T01:59:24.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:24.189+0000 7fa6fe3df700 1 -- 172.21.15.52:0/2119066721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6f8100f30 msgr2=0x7fa6f80fed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:24.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:24.189+0000 7fa6fe3df700 1 --2- 172.21.15.52:0/2119066721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6f8100f30 0x7fa6f80fed60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8005b40 tx=0x7fa6e8004c80 comp rx=0 tx=0).stop 2024-04-27T01:59:24.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:24.189+0000 7fa6fe3df700 1 -- 172.21.15.52:0/2119066721 shutdown_connections 2024-04-27T01:59:24.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:24.189+0000 7fa6fe3df700 1 --2- 172.21.15.52:0/2119066721 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa6e4038580 0x7fa6e403aa30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:24.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:24.189+0000 7fa6fe3df700 1 --2- 172.21.15.52:0/2119066721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6f8100f30 0x7fa6f80fed60 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:24.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:24.189+0000 7fa6fe3df700 1 -- 172.21.15.52:0/2119066721 >> 172.21.15.52:0/2119066721 conn(0x7fa6f8074af0 msgr2=0x7fa6f8104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:24.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:24.189+0000 7fa6fe3df700 1 -- 172.21.15.52:0/2119066721 shutdown_connections 2024-04-27T01:59:24.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:24.189+0000 7fa6fe3df700 1 -- 172.21.15.52:0/2119066721 wait complete. 2024-04-27T01:59:24.173 INFO:teuthology.orchestra.run.smithi052.stderr:dumped monmap epoch 1 2024-04-27T01:59:24.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:23 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:24.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:23 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:59:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:24 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/2119066721' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:59:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:24 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:24 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:25.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:24 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:25.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:24 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:25.873 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:59:25.873 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:59:25.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:25 smithi029 ceph-mon[26769]: Deploying daemon node-exporter.smithi052 on smithi052 2024-04-27T01:59:26.079 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:59:28.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.972+0000 7fecf38b7700 1 -- 172.21.15.52:0/382526612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecec100090 msgr2=0x7fecec100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:28.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.972+0000 7fecf38b7700 1 --2- 172.21.15.52:0/382526612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecec100090 0x7fecec100460 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fecdc00b0a0 tx=0x7fecdc00b3b0 comp rx=0 tx=0).stop 2024-04-27T01:59:28.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.972+0000 7fecf38b7700 1 -- 172.21.15.52:0/382526612 shutdown_connections 2024-04-27T01:59:28.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.972+0000 7fecf38b7700 1 --2- 172.21.15.52:0/382526612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecec100090 0x7fecec100460 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:28.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.972+0000 7fecf38b7700 1 -- 172.21.15.52:0/382526612 >> 172.21.15.52:0/382526612 conn(0x7fecec0fb970 msgr2=0x7fecec0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:28.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.972+0000 7fecf38b7700 1 -- 172.21.15.52:0/382526612 shutdown_connections 2024-04-27T01:59:28.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.972+0000 7fecf38b7700 1 -- 172.21.15.52:0/382526612 wait complete. 2024-04-27T01:59:28.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.973+0000 7fecf38b7700 1 Processor -- start 2024-04-27T01:59:28.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.973+0000 7fecf38b7700 1 -- start start 2024-04-27T01:59:28.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.973+0000 7fecf38b7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecec100090 0x7fecec195320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:28.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.973+0000 7fecf38b7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecec195860 con 0x7fecec100090 2024-04-27T01:59:28.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.973+0000 7fecf1653700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecec100090 0x7fecec195320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:28.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.973+0000 7fecf1653700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecec100090 0x7fecec195320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.52:54306/0 (socket says 172.21.15.52:54306) 2024-04-27T01:59:28.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.973+0000 7fecf1653700 1 -- 172.21.15.52:0/2427532392 learned_addr learned my addr 172.21.15.52:0/2427532392 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T01:59:28.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.974+0000 7fecf1653700 1 -- 172.21.15.52:0/2427532392 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecdc009d00 con 0x7fecec100090 2024-04-27T01:59:28.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.974+0000 7fecf1653700 1 --2- 172.21.15.52:0/2427532392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecec100090 0x7fecec195320 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fecdc0062a0 tx=0x7fecdc003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:59:28.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.974+0000 7fecea7fc700 1 -- 172.21.15.52:0/2427532392 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecdc00e040 con 0x7fecec100090 2024-04-27T01:59:28.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.974+0000 7fecea7fc700 1 -- 172.21.15.52:0/2427532392 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fecdc00bdd0 con 0x7fecec100090 2024-04-27T01:59:28.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.974+0000 7fecf38b7700 1 -- 172.21.15.52:0/2427532392 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecec195a60 con 0x7fecec100090 2024-04-27T01:59:28.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.974+0000 7fecea7fc700 1 -- 172.21.15.52:0/2427532392 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecdc01d940 con 0x7fecec100090 2024-04-27T01:59:28.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.974+0000 7fecf38b7700 1 -- 172.21.15.52:0/2427532392 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecec195ee0 con 0x7fecec100090 2024-04-27T01:59:28.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.975+0000 7fecf38b7700 1 -- 172.21.15.52:0/2427532392 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecec062380 con 0x7fecec100090 2024-04-27T01:59:28.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.975+0000 7fecea7fc700 1 -- 172.21.15.52:0/2427532392 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 45197+0+0 (secure 0 0 0) 0x7fecdc004250 con 0x7fecec100090 2024-04-27T01:59:28.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.975+0000 7fecea7fc700 1 --2- 172.21.15.52:0/2427532392 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fecd4038580 0x7fecd403aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:28.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.975+0000 7fecea7fc700 1 -- 172.21.15.52:0/2427532392 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fecdc04c4b0 con 0x7fecec100090 2024-04-27T01:59:28.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.975+0000 7fecf0e52700 1 --2- 172.21.15.52:0/2427532392 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fecd4038580 0x7fecd403aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:28.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.976+0000 7fecf0e52700 1 --2- 172.21.15.52:0/2427532392 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fecd4038580 0x7fecd403aa30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fece0006fd0 tx=0x7fece0006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:59:28.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:28.980+0000 7fecea7fc700 1 -- 172.21.15.52:0/2427532392 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fecdc011570 con 0x7fecec100090 2024-04-27T01:59:29.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:28 smithi029 ceph-mon[26769]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:29.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:28 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:29.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:29.273+0000 7fecf38b7700 1 -- 172.21.15.52:0/2427532392 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fecec196680 con 0x7fecec100090 2024-04-27T01:59:29.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:29.273+0000 7fecea7fc700 1 -- 172.21.15.52:0/2427532392 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fecdc011390 con 0x7fecec100090 2024-04-27T01:59:29.255 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:59:29.255 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":1,"fsid":"4c587076-0439-11ef-bc93-c7b262605968","modified":"2024-04-27T01:56:49.232926Z","created":"2024-04-27T01:56:49.232926Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:59:29.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:29.275+0000 7fecf38b7700 1 -- 172.21.15.52:0/2427532392 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fecd4038580 msgr2=0x7fecd403aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:29.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:29.275+0000 7fecf38b7700 1 --2- 172.21.15.52:0/2427532392 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fecd4038580 0x7fecd403aa30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fece0006fd0 tx=0x7fece0006e40 comp rx=0 tx=0).stop 2024-04-27T01:59:29.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:29.275+0000 7fecf38b7700 1 -- 172.21.15.52:0/2427532392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecec100090 msgr2=0x7fecec195320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:29.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:29.275+0000 7fecf38b7700 1 --2- 172.21.15.52:0/2427532392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecec100090 0x7fecec195320 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fecdc0062a0 tx=0x7fecdc003ce0 comp rx=0 tx=0).stop 2024-04-27T01:59:29.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:29.275+0000 7fecf38b7700 1 -- 172.21.15.52:0/2427532392 shutdown_connections 2024-04-27T01:59:29.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:29.275+0000 7fecf38b7700 1 --2- 172.21.15.52:0/2427532392 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fecd4038580 0x7fecd403aa30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:29.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:29.275+0000 7fecf38b7700 1 --2- 172.21.15.52:0/2427532392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecec100090 0x7fecec195320 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:29.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:29.275+0000 7fecf38b7700 1 -- 172.21.15.52:0/2427532392 >> 172.21.15.52:0/2427532392 conn(0x7fecec0fb970 msgr2=0x7fecec104270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:29.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:29.275+0000 7fecf38b7700 1 -- 172.21.15.52:0/2427532392 shutdown_connections 2024-04-27T01:59:29.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:29.275+0000 7fecf38b7700 1 -- 172.21.15.52:0/2427532392 wait complete. 2024-04-27T01:59:29.257 INFO:teuthology.orchestra.run.smithi052.stderr:dumped monmap epoch 1 2024-04-27T01:59:30.644 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:30 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:30.645 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:30 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:30.645 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:30 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:30.645 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:30 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:30.645 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:30 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi052.gbsftx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:59:30.645 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:30 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi052.gbsftx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-27T01:59:30.645 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:30 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:59:30.645 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:30 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:59:30.645 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:30 smithi029 ceph-mon[26769]: Deploying daemon mgr.smithi052.gbsftx on smithi052 2024-04-27T01:59:30.645 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:30 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/2427532392' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:59:30.645 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:30 smithi029 ceph-mon[26769]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:30.822 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:59:30.823 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:59:31.270 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:59:32.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.912+0000 7f494aabe700 1 -- 172.21.15.52:0/4207254406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4944100090 msgr2=0x7f4944100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:32.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.912+0000 7f494aabe700 1 --2- 172.21.15.52:0/4207254406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4944100090 0x7f4944100460 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4938009910 tx=0x7f4938009c20 comp rx=0 tx=0).stop 2024-04-27T01:59:32.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.912+0000 7f494aabe700 1 -- 172.21.15.52:0/4207254406 shutdown_connections 2024-04-27T01:59:32.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.912+0000 7f494aabe700 1 --2- 172.21.15.52:0/4207254406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4944100090 0x7f4944100460 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:32.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.912+0000 7f494aabe700 1 -- 172.21.15.52:0/4207254406 >> 172.21.15.52:0/4207254406 conn(0x7f49440fb970 msgr2=0x7f49440fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:32.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.912+0000 7f494aabe700 1 -- 172.21.15.52:0/4207254406 shutdown_connections 2024-04-27T01:59:32.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.912+0000 7f494aabe700 1 -- 172.21.15.52:0/4207254406 wait complete. 2024-04-27T01:59:32.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.912+0000 7f494aabe700 1 Processor -- start 2024-04-27T01:59:32.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.913+0000 7f494aabe700 1 -- start start 2024-04-27T01:59:32.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.913+0000 7f494aabe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4944100090 0x7f49441953a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:32.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.913+0000 7f494aabe700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49441958e0 con 0x7f4944100090 2024-04-27T01:59:32.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.913+0000 7f494885a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4944100090 0x7f49441953a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:32.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.913+0000 7f494885a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4944100090 0x7f49441953a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.52:54322/0 (socket says 172.21.15.52:54322) 2024-04-27T01:59:32.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.913+0000 7f494885a700 1 -- 172.21.15.52:0/2758429480 learned_addr learned my addr 172.21.15.52:0/2758429480 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T01:59:32.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.913+0000 7f494885a700 1 -- 172.21.15.52:0/2758429480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49380095f0 con 0x7f4944100090 2024-04-27T01:59:32.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.914+0000 7f494885a700 1 --2- 172.21.15.52:0/2758429480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4944100090 0x7f49441953a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4938005b40 tx=0x7f4938004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:59:32.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.914+0000 7f4941ffb700 1 -- 172.21.15.52:0/2758429480 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f493801d070 con 0x7f4944100090 2024-04-27T01:59:32.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.914+0000 7f494aabe700 1 -- 172.21.15.52:0/2758429480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4944195ae0 con 0x7f4944100090 2024-04-27T01:59:32.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.914+0000 7f4941ffb700 1 -- 172.21.15.52:0/2758429480 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f49380044d0 con 0x7f4944100090 2024-04-27T01:59:32.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.914+0000 7f4941ffb700 1 -- 172.21.15.52:0/2758429480 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4938003930 con 0x7f4944100090 2024-04-27T01:59:32.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.914+0000 7f494aabe700 1 -- 172.21.15.52:0/2758429480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4944195ea0 con 0x7f4944100090 2024-04-27T01:59:32.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.915+0000 7f4941ffb700 1 -- 172.21.15.52:0/2758429480 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 45197+0+0 (secure 0 0 0) 0x7f493800b960 con 0x7f4944100090 2024-04-27T01:59:32.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.915+0000 7f494aabe700 1 -- 172.21.15.52:0/2758429480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4944062380 con 0x7f4944100090 2024-04-27T01:59:32.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.915+0000 7f4941ffb700 1 --2- 172.21.15.52:0/2758429480 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f49340385d0 0x7f493403aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:32.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.915+0000 7f4941ffb700 1 -- 172.21.15.52:0/2758429480 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f493804c4b0 con 0x7f4944100090 2024-04-27T01:59:32.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.915+0000 7f4943fff700 1 --2- 172.21.15.52:0/2758429480 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f49340385d0 0x7f493403aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:32.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.916+0000 7f4943fff700 1 --2- 172.21.15.52:0/2758429480 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f49340385d0 0x7f493403aa80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4930006fd0 tx=0x7f4930006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:59:32.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:32.921+0000 7f4941ffb700 1 -- 172.21.15.52:0/2758429480 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f493802a540 con 0x7f4944100090 2024-04-27T01:59:33.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:33.210+0000 7f494aabe700 1 -- 172.21.15.52:0/2758429480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4944107d90 con 0x7f4944100090 2024-04-27T01:59:33.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:33.211+0000 7f4941ffb700 1 -- 172.21.15.52:0/2758429480 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f493802a360 con 0x7f4944100090 2024-04-27T01:59:33.192 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:59:33.192 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":1,"fsid":"4c587076-0439-11ef-bc93-c7b262605968","modified":"2024-04-27T01:56:49.232926Z","created":"2024-04-27T01:56:49.232926Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:59:33.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:33.213+0000 7f494aabe700 1 -- 172.21.15.52:0/2758429480 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f49340385d0 msgr2=0x7f493403aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:33.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:33.213+0000 7f494aabe700 1 --2- 172.21.15.52:0/2758429480 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f49340385d0 0x7f493403aa80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4930006fd0 tx=0x7f4930006e40 comp rx=0 tx=0).stop 2024-04-27T01:59:33.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:33.213+0000 7f494aabe700 1 -- 172.21.15.52:0/2758429480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4944100090 msgr2=0x7f49441953a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:33.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:33.213+0000 7f494aabe700 1 --2- 172.21.15.52:0/2758429480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4944100090 0x7f49441953a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4938005b40 tx=0x7f4938004c80 comp rx=0 tx=0).stop 2024-04-27T01:59:33.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:33.213+0000 7f494aabe700 1 -- 172.21.15.52:0/2758429480 shutdown_connections 2024-04-27T01:59:33.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:33.213+0000 7f494aabe700 1 --2- 172.21.15.52:0/2758429480 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f49340385d0 0x7f493403aa80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:33.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:33.213+0000 7f494aabe700 1 --2- 172.21.15.52:0/2758429480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4944100090 0x7f49441953a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:33.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:33.213+0000 7f494aabe700 1 -- 172.21.15.52:0/2758429480 >> 172.21.15.52:0/2758429480 conn(0x7f49440fb970 msgr2=0x7f4944104270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:33.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:33.213+0000 7f494aabe700 1 -- 172.21.15.52:0/2758429480 shutdown_connections 2024-04-27T01:59:33.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:33.213+0000 7f494aabe700 1 -- 172.21.15.52:0/2758429480 wait complete. 2024-04-27T01:59:33.196 INFO:teuthology.orchestra.run.smithi052.stderr:dumped monmap epoch 1 2024-04-27T01:59:33.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:32 smithi029 ceph-mon[26769]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:34.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:33 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:34.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:33 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/2758429480' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:59:34.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:33 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:34.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:33 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:34.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:33 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:34.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:33 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:34.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:33 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:59:34.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:33 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:59:34.886 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:59:34.886 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:59:35.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:34 smithi029 ceph-mon[26769]: Deploying daemon mon.smithi052 on smithi052 2024-04-27T01:59:35.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:34 smithi029 ceph-mon[26769]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:35.843 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:59:37.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:36 smithi029 ceph-mon[26769]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:37.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.977+0000 7f6ac835d700 1 -- 172.21.15.52:0/59005519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ac00fef90 msgr2=0x7f6ac00ff360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:37.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.977+0000 7f6ac835d700 1 --2- 172.21.15.52:0/59005519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ac00fef90 0x7f6ac00ff360 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6abc009910 tx=0x7f6abc009c20 comp rx=0 tx=0).stop 2024-04-27T01:59:37.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.977+0000 7f6ac835d700 1 -- 172.21.15.52:0/59005519 shutdown_connections 2024-04-27T01:59:37.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.977+0000 7f6ac835d700 1 --2- 172.21.15.52:0/59005519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ac00fef90 0x7f6ac00ff360 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:37.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.977+0000 7f6ac835d700 1 -- 172.21.15.52:0/59005519 >> 172.21.15.52:0/59005519 conn(0x7f6ac00fa850 msgr2=0x7f6ac00fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:37.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.977+0000 7f6ac835d700 1 -- 172.21.15.52:0/59005519 shutdown_connections 2024-04-27T01:59:37.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.977+0000 7f6ac835d700 1 -- 172.21.15.52:0/59005519 wait complete. 2024-04-27T01:59:37.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.978+0000 7f6ac835d700 1 Processor -- start 2024-04-27T01:59:37.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.978+0000 7f6ac835d700 1 -- start start 2024-04-27T01:59:37.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.978+0000 7f6ac835d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ac00fef90 0x7f6ac0195350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:37.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.978+0000 7f6ac835d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac0195890 con 0x7f6ac00fef90 2024-04-27T01:59:37.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.978+0000 7f6ac60f9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ac00fef90 0x7f6ac0195350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:37.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.978+0000 7f6ac60f9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ac00fef90 0x7f6ac0195350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.52:56290/0 (socket says 172.21.15.52:56290) 2024-04-27T01:59:37.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.978+0000 7f6ac60f9700 1 -- 172.21.15.52:0/2757727190 learned_addr learned my addr 172.21.15.52:0/2757727190 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T01:59:37.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.979+0000 7f6ac60f9700 1 -- 172.21.15.52:0/2757727190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc0095f0 con 0x7f6ac00fef90 2024-04-27T01:59:37.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.979+0000 7f6ac60f9700 1 --2- 172.21.15.52:0/2757727190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ac00fef90 0x7f6ac0195350 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6abc005b40 tx=0x7f6abc004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:59:37.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.979+0000 7f6ab77fe700 1 -- 172.21.15.52:0/2757727190 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6abc01d070 con 0x7f6ac00fef90 2024-04-27T01:59:37.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.979+0000 7f6ac835d700 1 -- 172.21.15.52:0/2757727190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac0195a90 con 0x7f6ac00fef90 2024-04-27T01:59:37.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.979+0000 7f6ab77fe700 1 -- 172.21.15.52:0/2757727190 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6abc0044d0 con 0x7f6ac00fef90 2024-04-27T01:59:37.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.979+0000 7f6ab77fe700 1 -- 172.21.15.52:0/2757727190 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6abc003930 con 0x7f6ac00fef90 2024-04-27T01:59:37.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.979+0000 7f6ac835d700 1 -- 172.21.15.52:0/2757727190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac0195e50 con 0x7f6ac00fef90 2024-04-27T01:59:37.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.980+0000 7f6ab77fe700 1 -- 172.21.15.52:0/2757727190 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 45197+0+0 (secure 0 0 0) 0x7f6abc003ba0 con 0x7f6ac00fef90 2024-04-27T01:59:37.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.980+0000 7f6ab77fe700 1 --2- 172.21.15.52:0/2757727190 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f6aac0385d0 0x7f6aac03aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:37.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.980+0000 7f6ab77fe700 1 -- 172.21.15.52:0/2757727190 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f6abc04c460 con 0x7f6ac00fef90 2024-04-27T01:59:37.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.980+0000 7f6ac835d700 1 -- 172.21.15.52:0/2757727190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ac0062380 con 0x7f6ac00fef90 2024-04-27T01:59:37.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.980+0000 7f6ac58f8700 1 --2- 172.21.15.52:0/2757727190 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f6aac0385d0 0x7f6aac03aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:37.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.981+0000 7f6ac58f8700 1 --2- 172.21.15.52:0/2757727190 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f6aac0385d0 0x7f6aac03aa80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0006fd0 tx=0x7f6ab0006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:59:37.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:37.986+0000 7f6ab77fe700 1 -- 172.21.15.52:0/2757727190 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6abc02a540 con 0x7f6ac00fef90 2024-04-27T01:59:38.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:37 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:59:38.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:38.304+0000 7f6ac835d700 1 -- 172.21.15.52:0/2757727190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6ac0199d00 con 0x7f6ac00fef90 2024-04-27T01:59:38.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:38.304+0000 7f6ab77fe700 1 -- 172.21.15.52:0/2757727190 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f6abc02a360 con 0x7f6ac00fef90 2024-04-27T01:59:38.286 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:59:38.286 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":1,"fsid":"4c587076-0439-11ef-bc93-c7b262605968","modified":"2024-04-27T01:56:49.232926Z","created":"2024-04-27T01:56:49.232926Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:59:38.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:38.307+0000 7f6ac835d700 1 -- 172.21.15.52:0/2757727190 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f6aac0385d0 msgr2=0x7f6aac03aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:38.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:38.307+0000 7f6ac835d700 1 --2- 172.21.15.52:0/2757727190 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f6aac0385d0 0x7f6aac03aa80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0006fd0 tx=0x7f6ab0006e40 comp rx=0 tx=0).stop 2024-04-27T01:59:38.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:38.307+0000 7f6ac835d700 1 -- 172.21.15.52:0/2757727190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ac00fef90 msgr2=0x7f6ac0195350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:38.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:38.307+0000 7f6ac835d700 1 --2- 172.21.15.52:0/2757727190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ac00fef90 0x7f6ac0195350 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6abc005b40 tx=0x7f6abc004c80 comp rx=0 tx=0).stop 2024-04-27T01:59:38.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:38.307+0000 7f6ac835d700 1 -- 172.21.15.52:0/2757727190 shutdown_connections 2024-04-27T01:59:38.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:38.307+0000 7f6ac835d700 1 --2- 172.21.15.52:0/2757727190 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f6aac0385d0 0x7f6aac03aa80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:38.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:38.307+0000 7f6ac835d700 1 --2- 172.21.15.52:0/2757727190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ac00fef90 0x7f6ac0195350 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:38.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:38.307+0000 7f6ac835d700 1 -- 172.21.15.52:0/2757727190 >> 172.21.15.52:0/2757727190 conn(0x7f6ac00fa850 msgr2=0x7f6ac00687c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:38.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:38.307+0000 7f6ac835d700 1 -- 172.21.15.52:0/2757727190 shutdown_connections 2024-04-27T01:59:38.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:38.307+0000 7f6ac835d700 1 -- 172.21.15.52:0/2757727190 wait complete. 2024-04-27T01:59:38.289 INFO:teuthology.orchestra.run.smithi052.stderr:dumped monmap epoch 1 2024-04-27T01:59:40.300 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:59:40.300 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:59:40.505 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi052/config 2024-04-27T01:59:44.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:43 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:44.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-04-27T01:59:44.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: mon.smithi029 calling monitor election 2024-04-27T01:59:44.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2024-04-27T01:59:44.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2024-04-27T01:59:44.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:44.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: from='mgr.? 172.21.15.52:0/2405228458' entity='mgr.smithi052.gbsftx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi052.gbsftx/crt"}]: dispatch 2024-04-27T01:59:44.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2024-04-27T01:59:44.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: mon.smithi052 calling monitor election 2024-04-27T01:59:44.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2024-04-27T01:59:44.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:44.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2024-04-27T01:59:44.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2024-04-27T01:59:44.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: mon.smithi029 is new leader, mons smithi029,smithi052 in quorum (ranks 0,1) 2024-04-27T01:59:44.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: monmap e2: 2 mons at {smithi029=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],smithi052=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0]} removed_ranks: {} 2024-04-27T01:59:44.203 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: fsmap 2024-04-27T01:59:44.203 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: osdmap e5: 0 total, 0 up, 0 in 2024-04-27T01:59:44.203 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: mgrmap e17: smithi029.ewvdzs(active, since 35s) 2024-04-27T01:59:44.203 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: Standby manager daemon smithi052.gbsftx started 2024-04-27T01:59:44.203 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: overall HEALTH_OK 2024-04-27T01:59:44.203 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: from='mgr.? 172.21.15.52:0/2405228458' entity='mgr.smithi052.gbsftx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T01:59:44.203 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:44.203 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: from='mgr.? 172.21.15.52:0/2405228458' entity='mgr.smithi052.gbsftx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi052.gbsftx/key"}]: dispatch 2024-04-27T01:59:44.203 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: from='mgr.? 172.21.15.52:0/2405228458' entity='mgr.smithi052.gbsftx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T01:59:44.203 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:44.203 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:44.203 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:44.204 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:43 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:44.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.966+0000 7f3d1f84c700 1 -- 172.21.15.52:0/3008245219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d040035c0 msgr2=0x7f3d04005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:44.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.966+0000 7f3d1f84c700 1 --2- 172.21.15.52:0/3008245219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d040035c0 0x7f3d04005a40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3d180ffea0 tx=0x7f3d0800feb0 comp rx=0 tx=0).stop 2024-04-27T01:59:44.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.966+0000 7f3d1f84c700 1 -- 172.21.15.52:0/3008245219 shutdown_connections 2024-04-27T01:59:44.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.966+0000 7f3d1f84c700 1 --2- 172.21.15.52:0/3008245219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d040035c0 0x7f3d04005a40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:44.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.966+0000 7f3d1f84c700 1 -- 172.21.15.52:0/3008245219 >> 172.21.15.52:0/3008245219 conn(0x7f3d180fba50 msgr2=0x7f3d180fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:44.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.966+0000 7f3d1f84c700 1 -- 172.21.15.52:0/3008245219 shutdown_connections 2024-04-27T01:59:44.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.966+0000 7f3d1f84c700 1 -- 172.21.15.52:0/3008245219 wait complete. 2024-04-27T01:59:44.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.967+0000 7f3d1f84c700 1 Processor -- start 2024-04-27T01:59:44.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.967+0000 7f3d1f84c700 1 -- start start 2024-04-27T01:59:44.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.967+0000 7f3d1f84c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d040035c0 0x7f3d18195600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:44.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.967+0000 7f3d1f84c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d18195b40 0x7f3d18199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:44.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.967+0000 7f3d1f84c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d18196150 con 0x7f3d040035c0 2024-04-27T01:59:44.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.967+0000 7f3d1f84c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d181962c0 con 0x7f3d18195b40 2024-04-27T01:59:44.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.967+0000 7f3d1cde7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d18195b40 0x7f3d18199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:44.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.967+0000 7f3d1cde7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d18195b40 0x7f3d18199fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47420/0 (socket says 172.21.15.52:47420) 2024-04-27T01:59:44.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.967+0000 7f3d1cde7700 1 -- 172.21.15.52:0/3029093039 learned_addr learned my addr 172.21.15.52:0/3029093039 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T01:59:44.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.968+0000 7f3d1cde7700 1 -- 172.21.15.52:0/3029093039 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d18195b40 msgr2=0x7f3d18199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-27T01:59:44.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.968+0000 7f3d1d5e8700 1 --2- 172.21.15.52:0/3029093039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d040035c0 0x7f3d18195600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:44.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.968+0000 7f3d1cde7700 1 -- 172.21.15.52:0/3029093039 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d18195b40 msgr2=0x7f3d18199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-27T01:59:44.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.968+0000 7f3d1cde7700 1 --2- 172.21.15.52:0/3029093039 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d18195b40 0x7f3d18199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T01:59:44.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.968+0000 7f3d1cde7700 1 --2- 172.21.15.52:0/3029093039 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d18195b40 0x7f3d18199fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-27T01:59:44.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.968+0000 7f3d1d5e8700 1 -- 172.21.15.52:0/3029093039 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d18195b40 msgr2=0x7f3d18199fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-27T01:59:44.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.968+0000 7f3d1d5e8700 1 --2- 172.21.15.52:0/3029093039 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d18195b40 0x7f3d18199fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:44.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.968+0000 7f3d1d5e8700 1 -- 172.21.15.52:0/3029093039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d0800f900 con 0x7f3d040035c0 2024-04-27T01:59:44.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.968+0000 7f3d1d5e8700 1 --2- 172.21.15.52:0/3029093039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d040035c0 0x7f3d18195600 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3d180ffea0 tx=0x7f3d08004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:59:44.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.968+0000 7f3d0e7fc700 1 -- 172.21.15.52:0/3029093039 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d08013b30 con 0x7f3d040035c0 2024-04-27T01:59:44.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.968+0000 7f3d0e7fc700 1 -- 172.21.15.52:0/3029093039 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3d080044e0 con 0x7f3d040035c0 2024-04-27T01:59:44.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.968+0000 7f3d1f84c700 1 -- 172.21.15.52:0/3029093039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d1819a5b0 con 0x7f3d040035c0 2024-04-27T01:59:44.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.969+0000 7f3d0e7fc700 1 -- 172.21.15.52:0/3029093039 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d08003e20 con 0x7f3d040035c0 2024-04-27T01:59:44.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.969+0000 7f3d1f84c700 1 -- 172.21.15.52:0/3029093039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d1819ab00 con 0x7f3d040035c0 2024-04-27T01:59:44.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.969+0000 7f3d1f84c700 1 -- 172.21.15.52:0/3029093039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d18062380 con 0x7f3d040035c0 2024-04-27T01:59:44.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.970+0000 7f3d0e7fc700 1 -- 172.21.15.52:0/3029093039 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89858+0+0 (secure 0 0 0) 0x7f3d08013c90 con 0x7f3d040035c0 2024-04-27T01:59:44.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.970+0000 7f3d0e7fc700 1 --2- 172.21.15.52:0/3029093039 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3cfc06c700 0x7f3cfc06ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:44.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.971+0000 7f3d0e7fc700 1 -- 172.21.15.52:0/3029093039 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f3d0808d240 con 0x7f3d040035c0 2024-04-27T01:59:44.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.971+0000 7f3d1cde7700 1 --2- 172.21.15.52:0/3029093039 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3cfc06c700 0x7f3cfc06ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:44.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.971+0000 7f3d1cde7700 1 --2- 172.21.15.52:0/3029093039 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3cfc06c700 0x7f3cfc06ebb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3d14009180 tx=0x7f3d14008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:59:44.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:44.975+0000 7f3d0e7fc700 1 -- 172.21.15.52:0/3029093039 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3d08058960 con 0x7f3d040035c0 2024-04-27T01:59:45.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:44 smithi052 ceph-mon[32586]: mgrmap e18: smithi029.ewvdzs(active, since 36s), standbys: smithi052.gbsftx 2024-04-27T01:59:45.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:44 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mgr metadata", "who": "smithi052.gbsftx", "id": "smithi052.gbsftx"}]: dispatch 2024-04-27T01:59:45.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:44 smithi052 ceph-mon[32586]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:45.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:44 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:59:45.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:44 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2024-04-27T01:59:45.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:44 smithi029 ceph-mon[26769]: mgrmap e18: smithi029.ewvdzs(active, since 36s), standbys: smithi052.gbsftx 2024-04-27T01:59:45.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:44 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mgr metadata", "who": "smithi052.gbsftx", "id": "smithi052.gbsftx"}]: dispatch 2024-04-27T01:59:45.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:44 smithi029 ceph-mon[26769]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:45.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:44 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:59:45.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:44 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2024-04-27T01:59:45.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:45.267+0000 7f3d1f84c700 1 -- 172.21.15.52:0/3029093039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d181a0250 con 0x7f3d040035c0 2024-04-27T01:59:45.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:45.268+0000 7f3d0e7fc700 1 -- 172.21.15.52:0/3029093039 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1018 (secure 0 0 0) 0x7f3d08031090 con 0x7f3d040035c0 2024-04-27T01:59:45.249 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-27T01:59:45.249 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":2,"fsid":"4c587076-0439-11ef-bc93-c7b262605968","modified":"2024-04-27T01:59:38.700788Z","created":"2024-04-27T01:56:49.232926Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-27T01:59:45.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:45.270+0000 7f3d1f84c700 1 -- 172.21.15.52:0/3029093039 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3cfc06c700 msgr2=0x7f3cfc06ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:45.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:45.270+0000 7f3d1f84c700 1 --2- 172.21.15.52:0/3029093039 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3cfc06c700 0x7f3cfc06ebb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3d14009180 tx=0x7f3d14008040 comp rx=0 tx=0).stop 2024-04-27T01:59:45.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:45.270+0000 7f3d1f84c700 1 -- 172.21.15.52:0/3029093039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d040035c0 msgr2=0x7f3d18195600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:45.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:45.270+0000 7f3d1f84c700 1 --2- 172.21.15.52:0/3029093039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d040035c0 0x7f3d18195600 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3d180ffea0 tx=0x7f3d08004f90 comp rx=0 tx=0).stop 2024-04-27T01:59:45.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:45.270+0000 7f3d1f84c700 1 -- 172.21.15.52:0/3029093039 shutdown_connections 2024-04-27T01:59:45.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:45.270+0000 7f3d1f84c700 1 --2- 172.21.15.52:0/3029093039 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3cfc06c700 0x7f3cfc06ebb0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:45.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:45.270+0000 7f3d1f84c700 1 --2- 172.21.15.52:0/3029093039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d040035c0 0x7f3d18195600 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:45.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:45.270+0000 7f3d1f84c700 1 --2- 172.21.15.52:0/3029093039 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d18195b40 0x7f3d18199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:45.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:45.270+0000 7f3d1f84c700 1 -- 172.21.15.52:0/3029093039 >> 172.21.15.52:0/3029093039 conn(0x7f3d180fba50 msgr2=0x7f3d1818d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:45.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:45.270+0000 7f3d1f84c700 1 -- 172.21.15.52:0/3029093039 shutdown_connections 2024-04-27T01:59:45.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T01:59:45.270+0000 7f3d1f84c700 1 -- 172.21.15.52:0/3029093039 wait complete. 2024-04-27T01:59:45.252 INFO:teuthology.orchestra.run.smithi052.stderr:dumped monmap epoch 2 2024-04-27T01:59:45.877 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:45 smithi052 ceph-mon[32586]: from='client.? 172.21.15.52:0/3029093039' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:59:45.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:45 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/3029093039' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:59:46.233 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-27T01:59:46.233 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-27T01:59:46.466 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T01:59:46.862 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:46 smithi029 ceph-mon[26769]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:46.863 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:46 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:46.863 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:46 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:46.863 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:46 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:59:46.863 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:46 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:59:47.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:46 smithi052 ceph-mon[32586]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:47.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:46 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:47.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:46 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:47.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:46 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:59:47.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:46 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:59:47.924 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:47 smithi029 ceph-mon[26769]: Updating smithi029:/etc/ceph/ceph.conf 2024-04-27T01:59:47.924 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:47 smithi029 ceph-mon[26769]: Updating smithi052:/etc/ceph/ceph.conf 2024-04-27T01:59:47.924 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:47 smithi029 ceph-mon[26769]: Updating smithi052:/var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:59:47.924 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:47 smithi029 ceph-mon[26769]: Updating smithi029:/var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:59:47.924 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:47 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:47.924 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:47 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:47.924 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:47 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:47.924 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:47 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:47.925 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:47 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:47.925 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:47 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:59:47.925 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:47 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:59:47.925 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:47 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:59:48.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:47 smithi052 ceph-mon[32586]: Updating smithi029:/etc/ceph/ceph.conf 2024-04-27T01:59:48.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:47 smithi052 ceph-mon[32586]: Updating smithi052:/etc/ceph/ceph.conf 2024-04-27T01:59:48.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:47 smithi052 ceph-mon[32586]: Updating smithi052:/var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:59:48.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:47 smithi052 ceph-mon[32586]: Updating smithi029:/var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:59:48.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:47 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:48.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:47 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:48.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:47 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:48.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:47 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:48.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:47 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:48.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:47 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:59:48.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:47 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:59:48.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:47 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:59:48.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.795+0000 7f393d7e3700 1 -- 172.21.15.29:0/963938557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39380fefc0 msgr2=0x7f39380ff390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:48.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.795+0000 7f393d7e3700 1 --2- 172.21.15.29:0/963938557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39380fefc0 0x7f39380ff390 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3928009910 tx=0x7f3928009c20 comp rx=0 tx=0).stop 2024-04-27T01:59:48.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.795+0000 7f393d7e3700 1 -- 172.21.15.29:0/963938557 shutdown_connections 2024-04-27T01:59:48.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.795+0000 7f393d7e3700 1 --2- 172.21.15.29:0/963938557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39380fefc0 0x7f39380ff390 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:48.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.795+0000 7f393d7e3700 1 -- 172.21.15.29:0/963938557 >> 172.21.15.29:0/963938557 conn(0x7f3938074af0 msgr2=0x7f3938074ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:48.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.795+0000 7f393d7e3700 1 -- 172.21.15.29:0/963938557 shutdown_connections 2024-04-27T01:59:48.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.795+0000 7f393d7e3700 1 -- 172.21.15.29:0/963938557 wait complete. 2024-04-27T01:59:48.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.795+0000 7f393d7e3700 1 Processor -- start 2024-04-27T01:59:48.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.795+0000 7f393d7e3700 1 -- start start 2024-04-27T01:59:48.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.796+0000 7f393d7e3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39380fefc0 0x7f3938194430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:48.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.796+0000 7f393d7e3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3938194970 0x7f3938198de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:48.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.796+0000 7f393d7e3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3938194f80 con 0x7f39380fefc0 2024-04-27T01:59:48.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.796+0000 7f393d7e3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39381950f0 con 0x7f3938194970 2024-04-27T01:59:48.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.796+0000 7f3936ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39380fefc0 0x7f3938194430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:48.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.796+0000 7f3936ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39380fefc0 0x7f3938194430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43906/0 (socket says 172.21.15.29:43906) 2024-04-27T01:59:48.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.796+0000 7f3936ffd700 1 -- 172.21.15.29:0/916953536 learned_addr learned my addr 172.21.15.29:0/916953536 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:59:48.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.796+0000 7f3936ffd700 1 -- 172.21.15.29:0/916953536 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3938194970 msgr2=0x7f3938198de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:48.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.796+0000 7f3936ffd700 1 --2- 172.21.15.29:0/916953536 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3938194970 0x7f3938198de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:48.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.796+0000 7f3936ffd700 1 -- 172.21.15.29:0/916953536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39280095f0 con 0x7f39380fefc0 2024-04-27T01:59:48.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.796+0000 7f3936ffd700 1 --2- 172.21.15.29:0/916953536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39380fefc0 0x7f3938194430 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f39280092e0 tx=0x7f392800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:59:48.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.796+0000 7f3934ff9700 1 -- 172.21.15.29:0/916953536 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f392801f070 con 0x7f39380fefc0 2024-04-27T01:59:48.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.796+0000 7f3934ff9700 1 -- 172.21.15.29:0/916953536 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3928005140 con 0x7f39380fefc0 2024-04-27T01:59:48.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.796+0000 7f393d7e3700 1 -- 172.21.15.29:0/916953536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3938199380 con 0x7f39380fefc0 2024-04-27T01:59:48.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.796+0000 7f3934ff9700 1 -- 172.21.15.29:0/916953536 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f392800f690 con 0x7f39380fefc0 2024-04-27T01:59:48.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.797+0000 7f393d7e3700 1 -- 172.21.15.29:0/916953536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39381998d0 con 0x7f39380fefc0 2024-04-27T01:59:48.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.798+0000 7f393d7e3700 1 -- 172.21.15.29:0/916953536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3938199c00 con 0x7f39380fefc0 2024-04-27T01:59:48.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.798+0000 7f3934ff9700 1 -- 172.21.15.29:0/916953536 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89858+0+0 (secure 0 0 0) 0x7f3928019650 con 0x7f39380fefc0 2024-04-27T01:59:48.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.798+0000 7f3934ff9700 1 --2- 172.21.15.29:0/916953536 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f392406c760 0x7f392406ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:48.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.799+0000 7f3934ff9700 1 -- 172.21.15.29:0/916953536 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f392808dd00 con 0x7f39380fefc0 2024-04-27T01:59:48.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.799+0000 7f392ffff700 1 --2- 172.21.15.29:0/916953536 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f392406c760 0x7f392406ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:48.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.799+0000 7f392ffff700 1 --2- 172.21.15.29:0/916953536 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f392406c760 0x7f392406ec10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f39200098e0 tx=0x7f3920006dd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:59:48.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:48.803+0000 7f3934ff9700 1 -- 172.21.15.29:0/916953536 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3928059540 con 0x7f39380fefc0 2024-04-27T01:59:48.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:49.006+0000 7f393d7e3700 1 -- 172.21.15.29:0/916953536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f3938067f50 con 0x7f39380fefc0 2024-04-27T01:59:48.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:49.006+0000 7f3934ff9700 1 -- 172.21.15.29:0/916953536 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+223 (secure 0 0 0) 0x7f39280280f0 con 0x7f39380fefc0 2024-04-27T01:59:48.987 INFO:teuthology.orchestra.run.smithi029.stdout:# minimal ceph.conf for 4c587076-0439-11ef-bc93-c7b262605968 2024-04-27T01:59:48.988 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2024-04-27T01:59:48.988 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = 4c587076-0439-11ef-bc93-c7b262605968 2024-04-27T01:59:48.988 INFO:teuthology.orchestra.run.smithi029.stdout: mon_host = [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-04-27T01:59:48.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:49.008+0000 7f393d7e3700 1 -- 172.21.15.29:0/916953536 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f392406c760 msgr2=0x7f392406ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:48.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:49.008+0000 7f393d7e3700 1 --2- 172.21.15.29:0/916953536 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f392406c760 0x7f392406ec10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f39200098e0 tx=0x7f3920006dd0 comp rx=0 tx=0).stop 2024-04-27T01:59:48.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:49.008+0000 7f393d7e3700 1 -- 172.21.15.29:0/916953536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39380fefc0 msgr2=0x7f3938194430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:48.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:49.008+0000 7f393d7e3700 1 --2- 172.21.15.29:0/916953536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39380fefc0 0x7f3938194430 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f39280092e0 tx=0x7f392800bda0 comp rx=0 tx=0).stop 2024-04-27T01:59:48.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:49.008+0000 7f393d7e3700 1 -- 172.21.15.29:0/916953536 shutdown_connections 2024-04-27T01:59:48.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:49.008+0000 7f393d7e3700 1 --2- 172.21.15.29:0/916953536 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f392406c760 0x7f392406ec10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:48.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:49.008+0000 7f393d7e3700 1 --2- 172.21.15.29:0/916953536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39380fefc0 0x7f3938194430 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:48.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:49.008+0000 7f393d7e3700 1 --2- 172.21.15.29:0/916953536 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3938194970 0x7f3938198de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:48.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:49.009+0000 7f393d7e3700 1 -- 172.21.15.29:0/916953536 >> 172.21.15.29:0/916953536 conn(0x7f3938074af0 msgr2=0x7f3938104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:48.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:49.009+0000 7f393d7e3700 1 -- 172.21.15.29:0/916953536 shutdown_connections 2024-04-27T01:59:48.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:49.009+0000 7f393d7e3700 1 -- 172.21.15.29:0/916953536 wait complete. 2024-04-27T01:59:49.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:48 smithi029 ceph-mon[26769]: Reconfiguring mon.smithi029 (unknown last config time)... 2024-04-27T01:59:49.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:48 smithi029 ceph-mon[26769]: Reconfiguring daemon mon.smithi029 on smithi029 2024-04-27T01:59:49.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:48 smithi029 ceph-mon[26769]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:49.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:48 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:49.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:48 smithi052 ceph-mon[32586]: Reconfiguring mon.smithi029 (unknown last config time)... 2024-04-27T01:59:49.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:48 smithi052 ceph-mon[32586]: Reconfiguring daemon mon.smithi029 on smithi029 2024-04-27T01:59:49.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:48 smithi052 ceph-mon[32586]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:49.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:48 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:49.768 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-27T01:59:49.768 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-27T01:59:49.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.conf 2024-04-27T01:59:49.795 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-27T01:59:49.795 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:59:49.859 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-27T01:59:49.860 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.conf 2024-04-27T01:59:49.893 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-27T01:59:49.893 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:59:49.928 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:49 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/916953536' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:59:49.929 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:49 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:49.929 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:49 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:49.929 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:49 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi029.ewvdzs", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:59:49.929 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:49 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:59:49.929 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:49 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:59:49.958 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:49 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/916953536' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:59:49.958 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:49 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:49.958 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:49 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:49.958 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:49 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi029.ewvdzs", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:59:49.958 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:49 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:59:49.958 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:49 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:59:49.962 INFO:tasks.cephadm:Deploying OSDs... 2024-04-27T01:59:49.962 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-27T01:59:49.963 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2024-04-27T01:59:49.978 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-27T01:59:49.978 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2024-04-27T01:59:50.034 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-27T01:59:50.034 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:59:50.034 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-27T01:59:50.034 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:59:50.034 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:59:50.034 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-04-27 01:58:59.158574474 +0000 2024-04-27T01:59:50.034 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-04-27 01:53:01.580219232 +0000 2024-04-27T01:59:50.034 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-04-27 01:53:01.580219232 +0000 2024-04-27T01:59:50.034 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-04-27 01:53:01.580219232 +0000 2024-04-27T01:59:50.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-27T01:59:50.101 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-04-27T01:59:50.101 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-04-27T01:59:50.101 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000106122 s, 4.8 MB/s 2024-04-27T01:59:50.102 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-27T01:59:50.159 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2024-04-27T01:59:50.214 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-27T01:59:50.215 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:59:50.215 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-04-27T01:59:50.215 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:59:50.215 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:59:50.215 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-04-27 01:58:59.159574458 +0000 2024-04-27T01:59:50.215 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-04-27 01:53:01.915212137 +0000 2024-04-27T01:59:50.215 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-04-27 01:53:01.915212137 +0000 2024-04-27T01:59:50.215 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-04-27 01:53:01.915212137 +0000 2024-04-27T01:59:50.215 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-27T01:59:50.278 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-04-27T01:59:50.278 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-04-27T01:59:50.278 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000113824 s, 4.5 MB/s 2024-04-27T01:59:50.279 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-27T01:59:50.335 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2024-04-27T01:59:50.391 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-27T01:59:50.391 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:59:50.391 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-27T01:59:50.391 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:59:50.391 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:59:50.391 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-04-27 01:58:59.159574458 +0000 2024-04-27T01:59:50.391 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-04-27 01:53:02.244205170 +0000 2024-04-27T01:59:50.391 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-04-27 01:53:02.244205170 +0000 2024-04-27T01:59:50.391 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-04-27 01:53:02.244205170 +0000 2024-04-27T01:59:50.392 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-27T01:59:50.459 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-04-27T01:59:50.460 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-04-27T01:59:50.460 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.2545e-05 s, 5.5 MB/s 2024-04-27T01:59:50.461 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-27T01:59:50.517 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2024-04-27T01:59:50.573 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-27T01:59:50.573 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:59:50.573 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-27T01:59:50.573 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:59:50.573 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:59:50.573 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-04-27 01:58:59.159574458 +0000 2024-04-27T01:59:50.574 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-04-27 01:53:02.571198245 +0000 2024-04-27T01:59:50.574 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-04-27 01:53:02.571198245 +0000 2024-04-27T01:59:50.574 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-04-27 01:53:02.571198245 +0000 2024-04-27T01:59:50.574 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-27T01:59:50.637 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-04-27T01:59:50.637 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-04-27T01:59:50.637 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000110786 s, 4.6 MB/s 2024-04-27T01:59:50.638 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-27T01:59:50.695 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-27T01:59:50.695 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2024-04-27T01:59:50.715 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-27T01:59:50.715 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_1 2024-04-27T01:59:50.773 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-27T01:59:50.773 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:59:50.773 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-27T01:59:50.773 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:59:50.773 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:59:50.773 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-04-27 01:59:37.650822898 +0000 2024-04-27T01:59:50.773 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-04-27 01:53:00.700925341 +0000 2024-04-27T01:59:50.774 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-04-27 01:53:00.700925341 +0000 2024-04-27T01:59:50.774 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-04-27 01:53:00.700925341 +0000 2024-04-27T01:59:50.774 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-27T01:59:50.841 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-04-27T01:59:50.842 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-04-27T01:59:50.842 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000123033 s, 4.2 MB/s 2024-04-27T01:59:50.843 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-27T01:59:50.902 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_2 2024-04-27T01:59:50.961 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-27T01:59:50.961 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:59:50.962 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-27T01:59:50.962 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:59:50.962 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:59:50.962 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-04-27 01:59:37.651822887 +0000 2024-04-27T01:59:50.962 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-04-27 01:53:01.030921088 +0000 2024-04-27T01:59:50.962 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-04-27 01:53:01.030921088 +0000 2024-04-27T01:59:50.962 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-04-27 01:53:01.030921088 +0000 2024-04-27T01:59:50.962 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-27T01:59:51.028 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-04-27T01:59:51.028 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-04-27T01:59:51.028 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000127821 s, 4.0 MB/s 2024-04-27T01:59:51.030 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-27T01:59:51.089 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_3 2024-04-27T01:59:51.089 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:50 smithi029 ceph-mon[26769]: Reconfiguring mgr.smithi029.ewvdzs (unknown last config time)... 2024-04-27T01:59:51.089 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:50 smithi029 ceph-mon[26769]: Reconfiguring daemon mgr.smithi029.ewvdzs on smithi029 2024-04-27T01:59:51.089 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:50 smithi029 ceph-mon[26769]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:51.147 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-27T01:59:51.147 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:59:51.147 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-27T01:59:51.147 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:59:51.147 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:59:51.147 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-04-27 01:59:37.651822887 +0000 2024-04-27T01:59:51.147 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-04-27 01:53:01.428915959 +0000 2024-04-27T01:59:51.147 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-04-27 01:53:01.428915959 +0000 2024-04-27T01:59:51.147 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-04-27 01:53:01.428915959 +0000 2024-04-27T01:59:51.148 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-27T01:59:51.211 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:50 smithi052 ceph-mon[32586]: Reconfiguring mgr.smithi029.ewvdzs (unknown last config time)... 2024-04-27T01:59:51.211 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:50 smithi052 ceph-mon[32586]: Reconfiguring daemon mgr.smithi029.ewvdzs on smithi029 2024-04-27T01:59:51.211 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:50 smithi052 ceph-mon[32586]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:51.213 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-04-27T01:59:51.213 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-04-27T01:59:51.213 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000124824 s, 4.1 MB/s 2024-04-27T01:59:51.215 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-27T01:59:51.273 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_4 2024-04-27T01:59:51.331 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-27T01:59:51.331 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:59:51.332 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-27T01:59:51.332 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:59:51.332 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:59:51.332 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-04-27 01:59:37.651822887 +0000 2024-04-27T01:59:51.332 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-04-27 01:53:01.775911487 +0000 2024-04-27T01:59:51.332 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-04-27 01:53:01.775911487 +0000 2024-04-27T01:59:51.332 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-04-27 01:53:01.775911487 +0000 2024-04-27T01:59:51.332 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-27T01:59:51.398 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-04-27T01:59:51.398 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-04-27T01:59:51.398 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000126002 s, 4.1 MB/s 2024-04-27T01:59:51.399 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-27T01:59:51.459 INFO:tasks.cephadm:Deploying osd.0 on smithi029 with /dev/vg_nvme/lv_4... 2024-04-27T01:59:51.459 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-27T01:59:52.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:52 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:52.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:52 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:52.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:52 smithi029 ceph-mon[26769]: Reconfiguring ceph-exporter.smithi029 (monmap changed)... 2024-04-27T01:59:52.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:52 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi029", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-27T01:59:52.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:52 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:59:52.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:52 smithi029 ceph-mon[26769]: Reconfiguring daemon ceph-exporter.smithi029 on smithi029 2024-04-27T01:59:52.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:52 smithi029 ceph-mon[26769]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:52.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:52 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:52.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:52 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:52.505 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:52 smithi052 ceph-mon[32586]: Reconfiguring ceph-exporter.smithi029 (monmap changed)... 2024-04-27T01:59:52.505 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:52 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi029", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-27T01:59:52.505 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:52 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:59:52.505 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:52 smithi052 ceph-mon[32586]: Reconfiguring daemon ceph-exporter.smithi029 on smithi029 2024-04-27T01:59:52.505 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:52 smithi052 ceph-mon[32586]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:52.640 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T01:59:53.885 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:53 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:53.885 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:53 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:53.885 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:53 smithi029 ceph-mon[26769]: Reconfiguring crash.smithi029 (monmap changed)... 2024-04-27T01:59:53.885 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:53 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi029", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T01:59:53.886 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:53 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:59:53.886 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:53 smithi029 ceph-mon[26769]: Reconfiguring daemon crash.smithi029 on smithi029 2024-04-27T01:59:53.886 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:53 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:59:54.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:53 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:54.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:53 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:54.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:53 smithi052 ceph-mon[32586]: Reconfiguring crash.smithi029 (monmap changed)... 2024-04-27T01:59:54.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:53 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi029", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T01:59:54.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:53 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:59:54.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:53 smithi052 ceph-mon[32586]: Reconfiguring daemon crash.smithi029 on smithi029 2024-04-27T01:59:54.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:53 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:59:54.790 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:54 smithi029 ceph-mon[26769]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:54.897 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-27T01:59:54.897 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-27T01:59:54.898 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-04-27T01:59:54.898 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-04-27T01:59:54.898 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00850662 s, 1.2 GB/s 2024-04-27T01:59:54.898 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-04-27T01:59:55.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:54 smithi052 ceph-mon[32586]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:55.578 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi029:vg_nvme/lv_4 2024-04-27T01:59:56.144 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T01:59:57.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:56 smithi029 ceph-mon[26769]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:57.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:56 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:57.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:56 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:57.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:56 smithi052 ceph-mon[32586]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:57.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:56 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:57.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:56 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T01:59:58.077 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:57 smithi029 ceph-mon[26769]: Reconfiguring alertmanager.smithi029 (dependencies changed)... 2024-04-27T01:59:58.077 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:57 smithi029 ceph-mon[26769]: Reconfiguring daemon alertmanager.smithi029 on smithi029 2024-04-27T01:59:58.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:57 smithi052 ceph-mon[32586]: Reconfiguring alertmanager.smithi029 (dependencies changed)... 2024-04-27T01:59:58.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:57 smithi052 ceph-mon[32586]: Reconfiguring daemon alertmanager.smithi029 on smithi029 2024-04-27T01:59:59.013 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 01:59:58 smithi029 ceph-mon[26769]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:59.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 01:59:58 smithi052 ceph-mon[32586]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:59:59.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.870+0000 7fe838620700 1 -- 172.21.15.29:0/4259479423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8300fde90 msgr2=0x7fe8300fe280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:59.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.870+0000 7fe838620700 1 --2- 172.21.15.29:0/4259479423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8300fde90 0x7fe8300fe280 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe82c009910 tx=0x7fe82c009c20 comp rx=0 tx=0).stop 2024-04-27T01:59:59.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.871+0000 7fe838620700 1 -- 172.21.15.29:0/4259479423 shutdown_connections 2024-04-27T01:59:59.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.871+0000 7fe838620700 1 --2- 172.21.15.29:0/4259479423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe8300fe850 0x7fe8301059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:59.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.871+0000 7fe838620700 1 --2- 172.21.15.29:0/4259479423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8300fde90 0x7fe8300fe280 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:59.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.871+0000 7fe838620700 1 -- 172.21.15.29:0/4259479423 >> 172.21.15.29:0/4259479423 conn(0x7fe8300f9a50 msgr2=0x7fe8300fbe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:59:59.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.871+0000 7fe838620700 1 -- 172.21.15.29:0/4259479423 shutdown_connections 2024-04-27T01:59:59.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.871+0000 7fe838620700 1 -- 172.21.15.29:0/4259479423 wait complete. 2024-04-27T01:59:59.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.871+0000 7fe838620700 1 Processor -- start 2024-04-27T01:59:59.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.871+0000 7fe838620700 1 -- start start 2024-04-27T01:59:59.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.872+0000 7fe838620700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8300fde90 0x7fe830195b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:59.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.872+0000 7fe838620700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe8300fe850 0x7fe830196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:59.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.872+0000 7fe838620700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe830196660 con 0x7fe8300fde90 2024-04-27T01:59:59.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.872+0000 7fe838620700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8301967a0 con 0x7fe8300fe850 2024-04-27T01:59:59.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.872+0000 7fe8363bc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8300fde90 0x7fe830195b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:59.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.872+0000 7fe8363bc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8300fde90 0x7fe830195b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59422/0 (socket says 172.21.15.29:59422) 2024-04-27T01:59:59.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.872+0000 7fe8363bc700 1 -- 172.21.15.29:0/3670788442 learned_addr learned my addr 172.21.15.29:0/3670788442 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T01:59:59.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.872+0000 7fe835bbb700 1 --2- 172.21.15.29:0/3670788442 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe8300fe850 0x7fe830196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:59.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.872+0000 7fe8363bc700 1 -- 172.21.15.29:0/3670788442 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe8300fe850 msgr2=0x7fe830196040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:59:59.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.872+0000 7fe8363bc700 1 --2- 172.21.15.29:0/3670788442 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe8300fe850 0x7fe830196040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T01:59:59.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.872+0000 7fe8363bc700 1 -- 172.21.15.29:0/3670788442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe82c0095f0 con 0x7fe8300fde90 2024-04-27T01:59:59.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.872+0000 7fe8363bc700 1 --2- 172.21.15.29:0/3670788442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8300fde90 0x7fe830195b00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe82c005620 tx=0x7fe82c005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:59:59.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.872+0000 7fe8277fe700 1 -- 172.21.15.29:0/3670788442 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe82c01d070 con 0x7fe8300fde90 2024-04-27T01:59:59.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.872+0000 7fe8277fe700 1 -- 172.21.15.29:0/3670788442 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe82c00f460 con 0x7fe8300fde90 2024-04-27T01:59:59.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.872+0000 7fe838620700 1 -- 172.21.15.29:0/3670788442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe83019b1f0 con 0x7fe8300fde90 2024-04-27T01:59:59.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.872+0000 7fe8277fe700 1 -- 172.21.15.29:0/3670788442 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe82c021640 con 0x7fe8300fde90 2024-04-27T01:59:59.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.873+0000 7fe838620700 1 -- 172.21.15.29:0/3670788442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe83019b6e0 con 0x7fe8300fde90 2024-04-27T01:59:59.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.873+0000 7fe838620700 1 -- 172.21.15.29:0/3670788442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe83018fc90 con 0x7fe8300fde90 2024-04-27T01:59:59.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.874+0000 7fe8277fe700 1 -- 172.21.15.29:0/3670788442 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89858+0+0 (secure 0 0 0) 0x7fe82c00fad0 con 0x7fe8300fde90 2024-04-27T01:59:59.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.874+0000 7fe8277fe700 1 --2- 172.21.15.29:0/3670788442 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fe81c06c600 0x7fe81c06eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T01:59:59.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.874+0000 7fe8277fe700 1 -- 172.21.15.29:0/3670788442 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fe82c08ba00 con 0x7fe8300fde90 2024-04-27T01:59:59.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.874+0000 7fe835bbb700 1 --2- 172.21.15.29:0/3670788442 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fe81c06c600 0x7fe81c06eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T01:59:59.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.875+0000 7fe835bbb700 1 --2- 172.21.15.29:0/3670788442 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fe81c06c600 0x7fe81c06eab0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe820005950 tx=0x7fe8200058e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T01:59:59.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T01:59:59.879+0000 7fe8277fe700 1 -- 172.21.15.29:0/3670788442 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe82c05ac30 con 0x7fe8300fde90 2024-04-27T02:00:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:00.093+0000 7fe838620700 1 -- 172.21.15.29:0/3670788442 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fe830061190 con 0x7fe81c06c600 2024-04-27T02:00:00.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:00 smithi029 ceph-mon[26769]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:00.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:00 smithi029 ceph-mon[26769]: overall HEALTH_OK 2024-04-27T02:00:00.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:00 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:00:00.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:00 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:00:00.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:00 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:00:01.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:00 smithi052 ceph-mon[32586]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:01.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:00 smithi052 ceph-mon[32586]: overall HEALTH_OK 2024-04-27T02:00:01.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:00 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:00:01.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:00 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:00:01.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:00 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:00:01.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:01 smithi029 ceph-mon[26769]: from='client.14254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:00:02.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:01 smithi052 ceph-mon[32586]: from='client.14254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:00:02.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:02 smithi029 ceph-mon[26769]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:03.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:02 smithi052 ceph-mon[32586]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:04.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:04 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:04.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:04 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:04.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:04 smithi052 ceph-mon[32586]: Reconfiguring grafana.smithi029 (dependencies changed)... 2024-04-27T02:00:04.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:04 smithi052 ceph-mon[32586]: Reconfiguring daemon grafana.smithi029 on smithi029 2024-04-27T02:00:04.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:04 smithi052 ceph-mon[32586]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:04.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:04 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:04.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:04 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:04.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:04 smithi029 ceph-mon[26769]: Reconfiguring grafana.smithi029 (dependencies changed)... 2024-04-27T02:00:04.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:04 smithi029 ceph-mon[26769]: Reconfiguring daemon grafana.smithi029 on smithi029 2024-04-27T02:00:04.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:04 smithi029 ceph-mon[26769]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:05.692 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:05 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/1213873317' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3686f584-4bb4-4d6a-8daa-93b9184e4bc0"}]: dispatch 2024-04-27T02:00:05.692 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:05 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/1213873317' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3686f584-4bb4-4d6a-8daa-93b9184e4bc0"}]': finished 2024-04-27T02:00:05.692 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:05 smithi029 ceph-mon[26769]: osdmap e6: 1 total, 0 up, 1 in 2024-04-27T02:00:05.692 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:05 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:00:05.692 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:05 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/410295385' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:00:05.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:05 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/1213873317' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3686f584-4bb4-4d6a-8daa-93b9184e4bc0"}]: dispatch 2024-04-27T02:00:05.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:05 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/1213873317' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3686f584-4bb4-4d6a-8daa-93b9184e4bc0"}]': finished 2024-04-27T02:00:05.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:05 smithi052 ceph-mon[32586]: osdmap e6: 1 total, 0 up, 1 in 2024-04-27T02:00:05.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:05 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:00:05.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:05 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/410295385' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:00:07.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:06 smithi052 ceph-mon[32586]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:07.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:06 smithi029 ceph-mon[26769]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:08.905 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:08 smithi029 ceph-mon[26769]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:08.906 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:08 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:00:09.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:08 smithi052 ceph-mon[32586]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:09.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:08 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:00:10.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:10 smithi029 ceph-mon[26769]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:11.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:10 smithi052 ceph-mon[32586]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:13.175 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:12 smithi029 ceph-mon[26769]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:13.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:12 smithi052 ceph-mon[32586]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:15.170 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:14 smithi029 ceph-mon[26769]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:15.170 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:14 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:15.170 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:14 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:15.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:14 smithi052 ceph-mon[32586]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:15.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:14 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:15.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:14 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:16.115 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:15 smithi029 ceph-mon[26769]: Reconfiguring prometheus.smithi029 (dependencies changed)... 2024-04-27T02:00:16.116 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:15 smithi029 ceph-mon[26769]: Reconfiguring daemon prometheus.smithi029 on smithi029 2024-04-27T02:00:16.116 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:15 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-27T02:00:16.116 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:15 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:00:16.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:15 smithi052 ceph-mon[32586]: Reconfiguring prometheus.smithi029 (dependencies changed)... 2024-04-27T02:00:16.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:15 smithi052 ceph-mon[32586]: Reconfiguring daemon prometheus.smithi029 on smithi029 2024-04-27T02:00:16.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:15 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-27T02:00:16.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:15 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:00:16.851 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:16 smithi029 ceph-mon[26769]: Deploying daemon osd.0 on smithi029 2024-04-27T02:00:16.851 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:16 smithi029 ceph-mon[26769]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:17.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:16 smithi052 ceph-mon[32586]: Deploying daemon osd.0 on smithi029 2024-04-27T02:00:17.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:16 smithi052 ceph-mon[32586]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:18.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:18 smithi029 ceph-mon[26769]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:19.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:18 smithi052 ceph-mon[32586]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:20.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:20 smithi029 ceph-mon[26769]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:20.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:20 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:20.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:20 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:20.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:20 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi052", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-27T02:00:20.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:20 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:00:21.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:20 smithi052 ceph-mon[32586]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:21.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:20 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:21.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:20 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:21.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:20 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi052", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-27T02:00:21.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:20 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:00:22.091 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:21 smithi029 ceph-mon[26769]: Reconfiguring ceph-exporter.smithi052 (monmap changed)... 2024-04-27T02:00:22.091 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:21 smithi029 ceph-mon[26769]: Reconfiguring daemon ceph-exporter.smithi052 on smithi052 2024-04-27T02:00:22.092 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:21 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:22.092 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:21 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:22.092 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:21 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi052", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T02:00:22.092 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:21 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:00:22.146 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:21 smithi052 ceph-mon[32586]: Reconfiguring ceph-exporter.smithi052 (monmap changed)... 2024-04-27T02:00:22.146 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:21 smithi052 ceph-mon[32586]: Reconfiguring daemon ceph-exporter.smithi052 on smithi052 2024-04-27T02:00:22.146 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:21 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:22.146 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:21 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:22.146 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:21 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi052", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T02:00:22.146 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:21 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:00:22.961 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:22 smithi052 ceph-mon[32586]: Reconfiguring crash.smithi052 (monmap changed)... 2024-04-27T02:00:22.962 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:22 smithi052 ceph-mon[32586]: Reconfiguring daemon crash.smithi052 on smithi052 2024-04-27T02:00:22.962 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:22 smithi052 ceph-mon[32586]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:22.962 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:22 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:00:23.053 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:22 smithi029 ceph-mon[26769]: Reconfiguring crash.smithi052 (monmap changed)... 2024-04-27T02:00:23.053 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:22 smithi029 ceph-mon[26769]: Reconfiguring daemon crash.smithi052 on smithi052 2024-04-27T02:00:23.053 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:22 smithi029 ceph-mon[26769]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:23.053 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:22 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:00:24.123 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:24 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:24.123 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:24 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:24.123 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:24 smithi052 ceph-mon[32586]: Reconfiguring mgr.smithi052.gbsftx (monmap changed)... 2024-04-27T02:00:24.123 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:24 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi052.gbsftx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T02:00:24.123 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:24 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T02:00:24.123 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:24 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:00:24.124 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:24 smithi052 ceph-mon[32586]: Reconfiguring daemon mgr.smithi052.gbsftx on smithi052 2024-04-27T02:00:24.124 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:24 smithi052 ceph-mon[32586]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:24.124 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:24 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:24.124 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:24 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:24.124 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:24 smithi029 ceph-mon[26769]: Reconfiguring mgr.smithi052.gbsftx (monmap changed)... 2024-04-27T02:00:24.124 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:24 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi052.gbsftx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T02:00:24.124 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:24 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T02:00:24.125 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:24 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:00:24.125 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:24 smithi029 ceph-mon[26769]: Reconfiguring daemon mgr.smithi052.gbsftx on smithi052 2024-04-27T02:00:24.125 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:24 smithi029 ceph-mon[26769]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:25.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:25 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:25.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:25 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:25.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:25 smithi029 ceph-mon[26769]: Reconfiguring mon.smithi052 (monmap changed)... 2024-04-27T02:00:25.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:25 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T02:00:25.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:25 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T02:00:25.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:25 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:00:25.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:25 smithi029 ceph-mon[26769]: Reconfiguring daemon mon.smithi052 on smithi052 2024-04-27T02:00:25.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:25 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:25.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:25 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:25.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:25 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:25.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:25 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:25.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:25 smithi052 ceph-mon[32586]: Reconfiguring mon.smithi052 (monmap changed)... 2024-04-27T02:00:25.755 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:25 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T02:00:25.755 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:25 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T02:00:25.755 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:25 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:00:25.755 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:25 smithi052 ceph-mon[32586]: Reconfiguring daemon mon.smithi052 on smithi052 2024-04-27T02:00:25.755 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:25 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:25.755 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:25 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:27.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:26 smithi052 ceph-mon[32586]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:27.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:26 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:27.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:26 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:27.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:26 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T02:00:27.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:26 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029:9093"}]: dispatch 2024-04-27T02:00:27.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:26 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:27.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:26 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T02:00:27.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:26 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi029:3000"}]: dispatch 2024-04-27T02:00:27.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:26 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:27.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:26 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T02:00:27.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:26 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi029:9095"}]: dispatch 2024-04-27T02:00:27.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:26 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:27.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:26 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:00:27.059 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:26 smithi029 ceph-mon[26769]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:27.060 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:26 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:27.060 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:26 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:27.060 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:26 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T02:00:27.060 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:26 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029:9093"}]: dispatch 2024-04-27T02:00:27.060 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:26 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:27.060 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:26 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T02:00:27.060 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:26 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi029:3000"}]: dispatch 2024-04-27T02:00:27.060 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:26 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:27.060 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:26 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T02:00:27.060 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:26 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi029:9095"}]: dispatch 2024-04-27T02:00:27.060 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:26 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:27.060 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:26 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:00:28.099 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:27 smithi029 ceph-mon[26769]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T02:00:28.099 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:27 smithi029 ceph-mon[26769]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029:9093"}]: dispatch 2024-04-27T02:00:28.099 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:27 smithi029 ceph-mon[26769]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T02:00:28.099 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:27 smithi029 ceph-mon[26769]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi029:3000"}]: dispatch 2024-04-27T02:00:28.099 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:27 smithi029 ceph-mon[26769]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T02:00:28.099 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:27 smithi029 ceph-mon[26769]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi029:9095"}]: dispatch 2024-04-27T02:00:28.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:27 smithi052 ceph-mon[32586]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T02:00:28.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:27 smithi052 ceph-mon[32586]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029:9093"}]: dispatch 2024-04-27T02:00:28.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:27 smithi052 ceph-mon[32586]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T02:00:28.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:27 smithi052 ceph-mon[32586]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi029:3000"}]: dispatch 2024-04-27T02:00:28.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:27 smithi052 ceph-mon[32586]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T02:00:28.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:27 smithi052 ceph-mon[32586]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi029:9095"}]: dispatch 2024-04-27T02:00:28.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:28.870+0000 7fe8277fe700 1 -- 172.21.15.29:0/3670788442 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe830061190 con 0x7fe81c06c600 2024-04-27T02:00:28.852 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 0 on host 'smithi029' 2024-04-27T02:00:28.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:28.873+0000 7fe838620700 1 -- 172.21.15.29:0/3670788442 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fe81c06c600 msgr2=0x7fe81c06eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:00:28.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:28.873+0000 7fe838620700 1 --2- 172.21.15.29:0/3670788442 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fe81c06c600 0x7fe81c06eab0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe820005950 tx=0x7fe8200058e0 comp rx=0 tx=0).stop 2024-04-27T02:00:28.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:28.873+0000 7fe838620700 1 -- 172.21.15.29:0/3670788442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8300fde90 msgr2=0x7fe830195b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:00:28.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:28.873+0000 7fe838620700 1 --2- 172.21.15.29:0/3670788442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8300fde90 0x7fe830195b00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe82c005620 tx=0x7fe82c005700 comp rx=0 tx=0).stop 2024-04-27T02:00:28.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:28.873+0000 7fe838620700 1 -- 172.21.15.29:0/3670788442 shutdown_connections 2024-04-27T02:00:28.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:28.873+0000 7fe838620700 1 --2- 172.21.15.29:0/3670788442 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fe81c06c600 0x7fe81c06eab0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:00:28.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:28.873+0000 7fe838620700 1 --2- 172.21.15.29:0/3670788442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8300fde90 0x7fe830195b00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:00:28.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:28.873+0000 7fe838620700 1 --2- 172.21.15.29:0/3670788442 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe8300fe850 0x7fe830196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:00:28.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:28.873+0000 7fe838620700 1 -- 172.21.15.29:0/3670788442 >> 172.21.15.29:0/3670788442 conn(0x7fe8300f9a50 msgr2=0x7fe830104250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:00:28.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:28.873+0000 7fe838620700 1 -- 172.21.15.29:0/3670788442 shutdown_connections 2024-04-27T02:00:28.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:28.873+0000 7fe838620700 1 -- 172.21.15.29:0/3670788442 wait complete. 2024-04-27T02:00:29.170 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:28 smithi029 ceph-mon[26769]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:29.170 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:28 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:29.170 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:28 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:29.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:28 smithi052 ceph-mon[32586]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:29.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:28 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:29.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:28 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:29.497 DEBUG:teuthology.orchestra.run.smithi029:osd.0> sudo journalctl -f -n 0 -u ceph-4c587076-0439-11ef-bc93-c7b262605968@osd.0.service 2024-04-27T02:00:29.500 INFO:tasks.cephadm:Deploying osd.1 on smithi029 with /dev/vg_nvme/lv_3... 2024-04-27T02:00:29.500 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-27T02:00:30.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:29 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:30.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:29 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:30.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:29 smithi029 ceph-mon[26769]: from='osd.0 [v2:172.21.15.29:6802/1138569897,v1:172.21.15.29:6803/1138569897]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-27T02:00:30.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:29 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:30.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:29 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:30.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:29 smithi052 ceph-mon[32586]: from='osd.0 [v2:172.21.15.29:6802/1138569897,v1:172.21.15.29:6803/1138569897]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-27T02:00:31.024 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:00:31.151 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:30 smithi029 ceph-mon[26769]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:31.151 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:30 smithi029 ceph-mon[26769]: from='osd.0 [v2:172.21.15.29:6802/1138569897,v1:172.21.15.29:6803/1138569897]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-27T02:00:31.151 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:30 smithi029 ceph-mon[26769]: osdmap e7: 1 total, 0 up, 1 in 2024-04-27T02:00:31.151 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:30 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:00:31.151 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:30 smithi029 ceph-mon[26769]: from='osd.0 [v2:172.21.15.29:6802/1138569897,v1:172.21.15.29:6803/1138569897]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-27T02:00:31.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:00:30.857+0000 7f3a1d954700 -1 osd.0 0 waiting for initial osdmap 2024-04-27T02:00:31.152 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:00:30.861+0000 7f3a19749700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T02:00:31.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:30 smithi052 ceph-mon[32586]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:31.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:30 smithi052 ceph-mon[32586]: from='osd.0 [v2:172.21.15.29:6802/1138569897,v1:172.21.15.29:6803/1138569897]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-27T02:00:31.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:30 smithi052 ceph-mon[32586]: osdmap e7: 1 total, 0 up, 1 in 2024-04-27T02:00:31.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:30 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:00:31.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:30 smithi052 ceph-mon[32586]: from='osd.0 [v2:172.21.15.29:6802/1138569897,v1:172.21.15.29:6803/1138569897]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-27T02:00:32.178 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:31 smithi029 ceph-mon[26769]: from='osd.0 [v2:172.21.15.29:6802/1138569897,v1:172.21.15.29:6803/1138569897]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-04-27T02:00:32.178 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:31 smithi029 ceph-mon[26769]: osdmap e8: 1 total, 0 up, 1 in 2024-04-27T02:00:32.178 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:31 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:00:32.178 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:31 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:00:32.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:31 smithi052 ceph-mon[32586]: from='osd.0 [v2:172.21.15.29:6802/1138569897,v1:172.21.15.29:6803/1138569897]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-04-27T02:00:32.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:31 smithi052 ceph-mon[32586]: osdmap e8: 1 total, 0 up, 1 in 2024-04-27T02:00:32.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:31 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:00:32.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:31 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:00:33.096 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:32 smithi029 ceph-mon[26769]: purged_snaps scrub starts 2024-04-27T02:00:33.096 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:32 smithi029 ceph-mon[26769]: purged_snaps scrub ok 2024-04-27T02:00:33.096 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:32 smithi029 ceph-mon[26769]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:33.096 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:32 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:00:33.096 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:32 smithi029 ceph-mon[26769]: osd.0 [v2:172.21.15.29:6802/1138569897,v1:172.21.15.29:6803/1138569897] boot 2024-04-27T02:00:33.096 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:32 smithi029 ceph-mon[26769]: osdmap e9: 1 total, 1 up, 1 in 2024-04-27T02:00:33.096 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:32 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:00:33.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:32 smithi052 ceph-mon[32586]: purged_snaps scrub starts 2024-04-27T02:00:33.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:32 smithi052 ceph-mon[32586]: purged_snaps scrub ok 2024-04-27T02:00:33.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:32 smithi052 ceph-mon[32586]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:00:33.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:32 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:00:33.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:32 smithi052 ceph-mon[32586]: osd.0 [v2:172.21.15.29:6802/1138569897,v1:172.21.15.29:6803/1138569897] boot 2024-04-27T02:00:33.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:32 smithi052 ceph-mon[32586]: osdmap e9: 1 total, 1 up, 1 in 2024-04-27T02:00:33.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:32 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:00:33.605 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-27T02:00:33.605 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-27T02:00:33.605 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-04-27T02:00:33.605 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-04-27T02:00:33.605 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00978165 s, 1.1 GB/s 2024-04-27T02:00:33.605 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-04-27T02:00:34.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:33 smithi029 ceph-mon[26769]: osdmap e10: 1 total, 1 up, 1 in 2024-04-27T02:00:34.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:33 smithi052 ceph-mon[32586]: osdmap e10: 1 total, 1 up, 1 in 2024-04-27T02:00:34.863 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi029:vg_nvme/lv_3 2024-04-27T02:00:35.094 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:00:35.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:34 smithi029 ceph-mon[26769]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:35.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:34 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:35.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:34 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:35.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:34 smithi052 ceph-mon[32586]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:35.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:34 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:35.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:34 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.051+0000 7f0a77b0e700 1 -- 172.21.15.29:0/1594513212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a70103550 msgr2=0x7f0a70105930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:00:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.051+0000 7f0a77b0e700 1 --2- 172.21.15.29:0/1594513212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a70103550 0x7f0a70105930 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0a64009910 tx=0x7f0a64009c20 comp rx=0 tx=0).stop 2024-04-27T02:00:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.051+0000 7f0a77b0e700 1 -- 172.21.15.29:0/1594513212 shutdown_connections 2024-04-27T02:00:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.051+0000 7f0a77b0e700 1 --2- 172.21.15.29:0/1594513212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a70103550 0x7f0a70105930 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:00:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.051+0000 7f0a77b0e700 1 --2- 172.21.15.29:0/1594513212 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a70100c30 0x7f0a70103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:00:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.051+0000 7f0a77b0e700 1 -- 172.21.15.29:0/1594513212 >> 172.21.15.29:0/1594513212 conn(0x7f0a700fa610 msgr2=0x7f0a700fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:00:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.051+0000 7f0a77b0e700 1 -- 172.21.15.29:0/1594513212 shutdown_connections 2024-04-27T02:00:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.051+0000 7f0a77b0e700 1 -- 172.21.15.29:0/1594513212 wait complete. 2024-04-27T02:00:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.052+0000 7f0a77b0e700 1 Processor -- start 2024-04-27T02:00:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.052+0000 7f0a77b0e700 1 -- start start 2024-04-27T02:00:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.052+0000 7f0a77b0e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a70100c30 0x7f0a70195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:00:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.052+0000 7f0a77b0e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a70103550 0x7f0a70195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:00:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.052+0000 7f0a77b0e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a701964d0 con 0x7f0a70100c30 2024-04-27T02:00:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.052+0000 7f0a77b0e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a70196610 con 0x7f0a70103550 2024-04-27T02:00:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.052+0000 7f0a758aa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a70100c30 0x7f0a70195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:00:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.052+0000 7f0a758aa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a70100c30 0x7f0a70195970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46320/0 (socket says 172.21.15.29:46320) 2024-04-27T02:00:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.052+0000 7f0a758aa700 1 -- 172.21.15.29:0/2165222011 learned_addr learned my addr 172.21.15.29:0/2165222011 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:00:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.052+0000 7f0a750a9700 1 --2- 172.21.15.29:0/2165222011 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a70103550 0x7f0a70195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:00:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.052+0000 7f0a758aa700 1 -- 172.21.15.29:0/2165222011 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a70103550 msgr2=0x7f0a70195eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:00:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.052+0000 7f0a758aa700 1 --2- 172.21.15.29:0/2165222011 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a70103550 0x7f0a70195eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:00:37.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.052+0000 7f0a758aa700 1 -- 172.21.15.29:0/2165222011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a640095f0 con 0x7f0a70100c30 2024-04-27T02:00:37.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.052+0000 7f0a758aa700 1 --2- 172.21.15.29:0/2165222011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a70100c30 0x7f0a70195970 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0a60007e90 tx=0x7f0a6000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:00:37.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.053+0000 7f0a6effd700 1 -- 172.21.15.29:0/2165222011 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a6000dda0 con 0x7f0a70100c30 2024-04-27T02:00:37.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.053+0000 7f0a6effd700 1 -- 172.21.15.29:0/2165222011 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0a6000f040 con 0x7f0a70100c30 2024-04-27T02:00:37.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.053+0000 7f0a77b0e700 1 -- 172.21.15.29:0/2165222011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a7019b070 con 0x7f0a70100c30 2024-04-27T02:00:37.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.053+0000 7f0a6effd700 1 -- 172.21.15.29:0/2165222011 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a60012610 con 0x7f0a70100c30 2024-04-27T02:00:37.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.053+0000 7f0a77b0e700 1 -- 172.21.15.29:0/2165222011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a7019b5c0 con 0x7f0a70100c30 2024-04-27T02:00:37.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.054+0000 7f0a77b0e700 1 -- 172.21.15.29:0/2165222011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a7018fab0 con 0x7f0a70100c30 2024-04-27T02:00:37.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.054+0000 7f0a6effd700 1 -- 172.21.15.29:0/2165222011 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89858+0+0 (secure 0 0 0) 0x7f0a600128d0 con 0x7f0a70100c30 2024-04-27T02:00:37.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.055+0000 7f0a6effd700 1 --2- 172.21.15.29:0/2165222011 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f0a5806c600 0x7f0a5806eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:00:37.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.055+0000 7f0a6effd700 1 -- 172.21.15.29:0/2165222011 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1982+0+0 (secure 0 0 0) 0x7f0a6008a320 con 0x7f0a70100c30 2024-04-27T02:00:37.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.055+0000 7f0a750a9700 1 --2- 172.21.15.29:0/2165222011 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f0a5806c600 0x7f0a5806eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:00:37.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.055+0000 7f0a750a9700 1 --2- 172.21.15.29:0/2165222011 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f0a5806c600 0x7f0a5806eab0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0a64005b40 tx=0x7f0a64005a90 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:00:37.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.059+0000 7f0a6effd700 1 -- 172.21.15.29:0/2165222011 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0a60059310 con 0x7f0a70100c30 2024-04-27T02:00:37.134 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:36 smithi029 ceph-mon[26769]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:37.155 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:36 smithi052 ceph-mon[32586]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:37.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:37.272+0000 7f0a77b0e700 1 -- 172.21.15.29:0/2165222011 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a70061190 con 0x7f0a5806c600 2024-04-27T02:00:38.174 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:37 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:00:38.174 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:37 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:00:38.174 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:37 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:00:38.174 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:37 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:00:38.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:37 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:00:38.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:37 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:00:38.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:37 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:00:38.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:37 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:00:39.075 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:38 smithi029 ceph-mon[26769]: from='client.14272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:00:39.075 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:38 smithi029 ceph-mon[26769]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:39.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:38 smithi052 ceph-mon[32586]: from='client.14272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:00:39.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:38 smithi052 ceph-mon[32586]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:41.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:40 smithi029 ceph-mon[26769]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:41.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:40 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:41.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:40 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:41.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:40 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:00:41.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:40 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:41.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:40 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:00:41.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:40 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:00:41.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:40 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:41.037 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:40 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:00:41.122 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:40 smithi052 ceph-mon[32586]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:41.123 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:40 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:41.123 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:40 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:41.123 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:40 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:00:41.123 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:40 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:41.123 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:40 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:00:41.123 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:40 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:00:41.123 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:40 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:41.123 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:40 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:00:42.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:41 smithi029 ceph-mon[26769]: Detected new or changed devices on smithi029 2024-04-27T02:00:42.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:41 smithi029 ceph-mon[26769]: Adjusting osd_memory_target on smithi029 to 11845M 2024-04-27T02:00:42.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:41 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/354353632' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad9bc7ef-74a2-46c2-a201-a2820c9c8923"}]: dispatch 2024-04-27T02:00:42.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:41 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/354353632' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad9bc7ef-74a2-46c2-a201-a2820c9c8923"}]': finished 2024-04-27T02:00:42.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:41 smithi029 ceph-mon[26769]: osdmap e11: 2 total, 1 up, 2 in 2024-04-27T02:00:42.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:41 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:00:42.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:41 smithi052 ceph-mon[32586]: Detected new or changed devices on smithi029 2024-04-27T02:00:42.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:41 smithi052 ceph-mon[32586]: Adjusting osd_memory_target on smithi029 to 11845M 2024-04-27T02:00:42.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:41 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/354353632' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad9bc7ef-74a2-46c2-a201-a2820c9c8923"}]: dispatch 2024-04-27T02:00:42.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:41 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/354353632' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad9bc7ef-74a2-46c2-a201-a2820c9c8923"}]': finished 2024-04-27T02:00:42.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:41 smithi052 ceph-mon[32586]: osdmap e11: 2 total, 1 up, 2 in 2024-04-27T02:00:42.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:41 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:00:43.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:42 smithi029 ceph-mon[26769]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:43.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:42 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:43.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:42 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:43.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:42 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/2407929099' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:00:43.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:42 smithi052 ceph-mon[32586]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:43.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:42 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:43.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:42 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:43.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:42 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/2407929099' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:00:45.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:44 smithi029 ceph-mon[26769]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:45.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:44 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:45.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:44 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:45.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:44 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:00:45.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:44 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:00:45.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:44 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:45.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:44 smithi052 ceph-mon[32586]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:45.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:44 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:45.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:44 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:45.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:44 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:00:45.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:44 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:00:45.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:44 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:47.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:46 smithi029 ceph-mon[26769]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:47.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:46 smithi052 ceph-mon[32586]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:49.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:48 smithi029 ceph-mon[26769]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:49.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:48 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-27T02:00:49.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:48 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:00:49.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:48 smithi052 ceph-mon[32586]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:49.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:48 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-27T02:00:49.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:48 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:00:50.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:49 smithi029 ceph-mon[26769]: Deploying daemon osd.1 on smithi029 2024-04-27T02:00:50.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:49 smithi052 ceph-mon[32586]: Deploying daemon osd.1 on smithi029 2024-04-27T02:00:50.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:50 smithi029 ceph-mon[26769]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:51.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:50 smithi052 ceph-mon[32586]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:53.182 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:52 smithi029 ceph-mon[26769]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:53.183 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:52 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:00:53.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:52 smithi052 ceph-mon[32586]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:53.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:52 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:00:55.130 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:54 smithi029 ceph-mon[26769]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:55.130 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:54 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:55.130 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:54 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:55.130 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:54 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:00:55.158 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:54 smithi052 ceph-mon[32586]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:55.159 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:54 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:55.159 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:54 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:55.159 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:54 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:00:57.133 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:56 smithi029 ceph-mon[26769]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:57.133 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:56 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:57.133 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:56 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:57.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:56 smithi052 ceph-mon[32586]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:57.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:56 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:57.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:56 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:58.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:58.882+0000 7f0a6effd700 1 -- 172.21.15.29:0/2165222011 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0a70061190 con 0x7f0a5806c600 2024-04-27T02:00:58.864 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 1 on host 'smithi029' 2024-04-27T02:00:58.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:58.885+0000 7f0a77b0e700 1 -- 172.21.15.29:0/2165222011 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f0a5806c600 msgr2=0x7f0a5806eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:00:58.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:58.885+0000 7f0a77b0e700 1 --2- 172.21.15.29:0/2165222011 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f0a5806c600 0x7f0a5806eab0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0a64005b40 tx=0x7f0a64005a90 comp rx=0 tx=0).stop 2024-04-27T02:00:58.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:58.885+0000 7f0a77b0e700 1 -- 172.21.15.29:0/2165222011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a70100c30 msgr2=0x7f0a70195970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:00:58.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:58.885+0000 7f0a77b0e700 1 --2- 172.21.15.29:0/2165222011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a70100c30 0x7f0a70195970 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0a60007e90 tx=0x7f0a6000d3b0 comp rx=0 tx=0).stop 2024-04-27T02:00:58.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:58.886+0000 7f0a77b0e700 1 -- 172.21.15.29:0/2165222011 shutdown_connections 2024-04-27T02:00:58.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:58.886+0000 7f0a77b0e700 1 --2- 172.21.15.29:0/2165222011 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f0a5806c600 0x7f0a5806eab0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:00:58.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:58.886+0000 7f0a77b0e700 1 --2- 172.21.15.29:0/2165222011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a70100c30 0x7f0a70195970 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:00:58.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:58.886+0000 7f0a77b0e700 1 --2- 172.21.15.29:0/2165222011 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a70103550 0x7f0a70195eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:00:58.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:58.886+0000 7f0a77b0e700 1 -- 172.21.15.29:0/2165222011 >> 172.21.15.29:0/2165222011 conn(0x7f0a700fa610 msgr2=0x7f0a700fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:00:58.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:58.886+0000 7f0a77b0e700 1 -- 172.21.15.29:0/2165222011 shutdown_connections 2024-04-27T02:00:58.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:00:58.886+0000 7f0a77b0e700 1 -- 172.21.15.29:0/2165222011 wait complete. 2024-04-27T02:00:59.181 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:58 smithi029 ceph-mon[26769]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:59.182 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:58 smithi029 ceph-mon[26769]: from='osd.1 [v2:172.21.15.29:6810/3705295680,v1:172.21.15.29:6811/3705295680]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-27T02:00:59.182 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:58 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:59.182 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:58 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:59.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:58 smithi052 ceph-mon[32586]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:00:59.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:58 smithi052 ceph-mon[32586]: from='osd.1 [v2:172.21.15.29:6810/3705295680,v1:172.21.15.29:6811/3705295680]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-27T02:00:59.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:58 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:59.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:58 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:00:59.503 DEBUG:teuthology.orchestra.run.smithi029:osd.1> sudo journalctl -f -n 0 -u ceph-4c587076-0439-11ef-bc93-c7b262605968@osd.1.service 2024-04-27T02:00:59.506 INFO:tasks.cephadm:Deploying osd.2 on smithi029 with /dev/vg_nvme/lv_2... 2024-04-27T02:00:59.506 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-27T02:01:00.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:59 smithi029 ceph-mon[26769]: from='osd.1 [v2:172.21.15.29:6810/3705295680,v1:172.21.15.29:6811/3705295680]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-27T02:01:00.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:59 smithi029 ceph-mon[26769]: osdmap e12: 2 total, 1 up, 2 in 2024-04-27T02:01:00.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:59 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:01:00.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:00:59 smithi029 ceph-mon[26769]: from='osd.1 [v2:172.21.15.29:6810/3705295680,v1:172.21.15.29:6811/3705295680]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-27T02:01:00.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:00:59.915+0000 7f49258be700 -1 osd.1 0 waiting for initial osdmap 2024-04-27T02:01:00.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:00:59.919+0000 7f49206b1700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T02:01:00.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:59 smithi052 ceph-mon[32586]: from='osd.1 [v2:172.21.15.29:6810/3705295680,v1:172.21.15.29:6811/3705295680]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-27T02:01:00.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:59 smithi052 ceph-mon[32586]: osdmap e12: 2 total, 1 up, 2 in 2024-04-27T02:01:00.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:59 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:01:00.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:00:59 smithi052 ceph-mon[32586]: from='osd.1 [v2:172.21.15.29:6810/3705295680,v1:172.21.15.29:6811/3705295680]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-27T02:01:00.277 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:01:01.005 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:00 smithi029 ceph-mon[26769]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:01:01.006 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:00 smithi029 ceph-mon[26769]: from='osd.1 [v2:172.21.15.29:6810/3705295680,v1:172.21.15.29:6811/3705295680]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-04-27T02:01:01.006 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:00 smithi029 ceph-mon[26769]: osdmap e13: 2 total, 1 up, 2 in 2024-04-27T02:01:01.006 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:00 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:01:01.006 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:00 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:01:01.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:00 smithi052 ceph-mon[32586]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:01:01.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:00 smithi052 ceph-mon[32586]: from='osd.1 [v2:172.21.15.29:6810/3705295680,v1:172.21.15.29:6811/3705295680]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-04-27T02:01:01.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:00 smithi052 ceph-mon[32586]: osdmap e13: 2 total, 1 up, 2 in 2024-04-27T02:01:01.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:00 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:01:01.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:00 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:01:02.000 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:01 smithi029 ceph-mon[26769]: purged_snaps scrub starts 2024-04-27T02:01:02.001 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:01 smithi029 ceph-mon[26769]: purged_snaps scrub ok 2024-04-27T02:01:02.001 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:01 smithi029 ceph-mon[26769]: osd.1 [v2:172.21.15.29:6810/3705295680,v1:172.21.15.29:6811/3705295680] boot 2024-04-27T02:01:02.001 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:01 smithi029 ceph-mon[26769]: osdmap e14: 2 total, 2 up, 2 in 2024-04-27T02:01:02.001 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:01 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:01:02.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:01 smithi052 ceph-mon[32586]: purged_snaps scrub starts 2024-04-27T02:01:02.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:01 smithi052 ceph-mon[32586]: purged_snaps scrub ok 2024-04-27T02:01:02.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:01 smithi052 ceph-mon[32586]: osd.1 [v2:172.21.15.29:6810/3705295680,v1:172.21.15.29:6811/3705295680] boot 2024-04-27T02:01:02.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:01 smithi052 ceph-mon[32586]: osdmap e14: 2 total, 2 up, 2 in 2024-04-27T02:01:02.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:01 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:01:02.997 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-27T02:01:02.997 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-27T02:01:02.997 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-04-27T02:01:02.997 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-04-27T02:01:02.997 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00871925 s, 1.2 GB/s 2024-04-27T02:01:02.997 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-04-27T02:01:03.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:02 smithi029 ceph-mon[26769]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:01:03.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:02 smithi029 ceph-mon[26769]: osdmap e15: 2 total, 2 up, 2 in 2024-04-27T02:01:03.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:02 smithi052 ceph-mon[32586]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:01:03.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:02 smithi052 ceph-mon[32586]: osdmap e15: 2 total, 2 up, 2 in 2024-04-27T02:01:03.705 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi029:vg_nvme/lv_2 2024-04-27T02:01:03.941 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:01:05.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:04 smithi029 ceph-mon[26769]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:05.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:04 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:05.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:04 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:05.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:04 smithi052 ceph-mon[32586]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:05.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:04 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:05.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:04 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:05.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.746+0000 7f72a3cde700 1 -- 172.21.15.29:0/214359831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f729c0ff130 msgr2=0x7f729c0ff540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:01:05.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.746+0000 7f72a3cde700 1 --2- 172.21.15.29:0/214359831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f729c0ff130 0x7f729c0ff540 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7298009910 tx=0x7f7298009c20 comp rx=0 tx=0).stop 2024-04-27T02:01:05.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.746+0000 7f72a3cde700 1 -- 172.21.15.29:0/214359831 shutdown_connections 2024-04-27T02:01:05.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.746+0000 7f72a3cde700 1 --2- 172.21.15.29:0/214359831 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f729c1003d0 0x7f729c100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:01:05.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.746+0000 7f72a3cde700 1 --2- 172.21.15.29:0/214359831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f729c0ff130 0x7f729c0ff540 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:01:05.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.746+0000 7f72a3cde700 1 -- 172.21.15.29:0/214359831 >> 172.21.15.29:0/214359831 conn(0x7f729c0fa780 msgr2=0x7f729c0fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:01:05.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.746+0000 7f72a3cde700 1 -- 172.21.15.29:0/214359831 shutdown_connections 2024-04-27T02:01:05.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.746+0000 7f72a3cde700 1 -- 172.21.15.29:0/214359831 wait complete. 2024-04-27T02:01:05.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.747+0000 7f72a3cde700 1 Processor -- start 2024-04-27T02:01:05.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.747+0000 7f72a3cde700 1 -- start start 2024-04-27T02:01:05.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.747+0000 7f72a3cde700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f729c0ff130 0x7f729c103160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:01:05.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.747+0000 7f72a3cde700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f729c1003d0 0x7f729c1017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:01:05.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.747+0000 7f72a3cde700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729c101cf0 con 0x7f729c1003d0 2024-04-27T02:01:05.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.747+0000 7f72a3cde700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729c101e30 con 0x7f729c0ff130 2024-04-27T02:01:05.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.747+0000 7f72a1279700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f729c1003d0 0x7f729c1017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:01:05.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.747+0000 7f72a1279700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f729c1003d0 0x7f729c1017b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49552/0 (socket says 172.21.15.29:49552) 2024-04-27T02:01:05.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.747+0000 7f72a1279700 1 -- 172.21.15.29:0/3034013333 learned_addr learned my addr 172.21.15.29:0/3034013333 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:01:05.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.748+0000 7f72a1a7a700 1 --2- 172.21.15.29:0/3034013333 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f729c0ff130 0x7f729c103160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:01:05.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.748+0000 7f72a1279700 1 -- 172.21.15.29:0/3034013333 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f729c0ff130 msgr2=0x7f729c103160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:01:05.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.748+0000 7f72a1279700 1 --2- 172.21.15.29:0/3034013333 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f729c0ff130 0x7f729c103160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:01:05.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.748+0000 7f72a1279700 1 -- 172.21.15.29:0/3034013333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72980095f0 con 0x7f729c1003d0 2024-04-27T02:01:05.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.748+0000 7f72a1279700 1 --2- 172.21.15.29:0/3034013333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f729c1003d0 0x7f729c1017b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f728c00d8d0 tx=0x7f728c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:01:05.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.748+0000 7f7292ffd700 1 -- 172.21.15.29:0/3034013333 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f728c0098a0 con 0x7f729c1003d0 2024-04-27T02:01:05.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.748+0000 7f7292ffd700 1 -- 172.21.15.29:0/3034013333 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f728c010460 con 0x7f729c1003d0 2024-04-27T02:01:05.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.748+0000 7f7292ffd700 1 -- 172.21.15.29:0/3034013333 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f728c00f5f0 con 0x7f729c1003d0 2024-04-27T02:01:05.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.748+0000 7f72a3cde700 1 -- 172.21.15.29:0/3034013333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f729c102110 con 0x7f729c1003d0 2024-04-27T02:01:05.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.748+0000 7f72a3cde700 1 -- 172.21.15.29:0/3034013333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f729c19db00 con 0x7f729c1003d0 2024-04-27T02:01:05.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.749+0000 7f72a3cde700 1 -- 172.21.15.29:0/3034013333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f729c107c10 con 0x7f729c1003d0 2024-04-27T02:01:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.750+0000 7f7292ffd700 1 -- 172.21.15.29:0/3034013333 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89858+0+0 (secure 0 0 0) 0x7f728c009a00 con 0x7f729c1003d0 2024-04-27T02:01:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.750+0000 7f7292ffd700 1 --2- 172.21.15.29:0/3034013333 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f728806c600 0x7f728806eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:01:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.750+0000 7f7292ffd700 1 -- 172.21.15.29:0/3034013333 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2414+0+0 (secure 0 0 0) 0x7f728c08a9b0 con 0x7f729c1003d0 2024-04-27T02:01:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.750+0000 7f72a1a7a700 1 --2- 172.21.15.29:0/3034013333 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f728806c600 0x7f728806eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:01:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.751+0000 7f72a1a7a700 1 --2- 172.21.15.29:0/3034013333 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f728806c600 0x7f728806eab0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7298006010 tx=0x7f729800b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:01:05.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.755+0000 7f7292ffd700 1 -- 172.21.15.29:0/3034013333 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f728c059870 con 0x7f729c1003d0 2024-04-27T02:01:05.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:05.981+0000 7f72a3cde700 1 -- 172.21.15.29:0/3034013333 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f729c068c50 con 0x7f728806c600 2024-04-27T02:01:06.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:06 smithi029 ceph-mon[26769]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:06.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:06 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:01:06.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:06 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:01:06.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:06 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:01:07.155 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:06 smithi052 ceph-mon[32586]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:07.155 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:06 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:01:07.155 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:06 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:01:07.155 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:06 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:01:08.147 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:07 smithi029 ceph-mon[26769]: from='client.14290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:01:08.148 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:07 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:01:08.174 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:07 smithi052 ceph-mon[32586]: from='client.14290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:01:08.174 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:07 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:01:08.877 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:08 smithi029 ceph-mon[26769]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:09.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:08 smithi052 ceph-mon[32586]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:10.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:09 smithi052 ceph-mon[32586]: Detected new or changed devices on smithi029 2024-04-27T02:01:10.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:09 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:10.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:09 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:10.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:09 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:01:10.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:09 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:01:10.006 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:09 smithi052 ceph-mon[32586]: Adjusting osd_memory_target on smithi029 to 5922M 2024-04-27T02:01:10.006 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:09 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:10.006 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:09 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:01:10.006 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:09 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:01:10.006 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:09 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:10.006 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:09 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:01:10.006 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:09 smithi052 ceph-mon[32586]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:10.052 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:09 smithi029 ceph-mon[26769]: Detected new or changed devices on smithi029 2024-04-27T02:01:10.052 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:09 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:10.052 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:09 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:10.052 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:09 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:01:10.052 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:09 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:01:10.052 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:09 smithi029 ceph-mon[26769]: Adjusting osd_memory_target on smithi029 to 5922M 2024-04-27T02:01:10.053 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:09 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:10.053 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:09 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:01:10.053 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:09 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:01:10.053 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:09 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:10.053 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:09 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:01:10.053 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:09 smithi029 ceph-mon[26769]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:11.061 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:10 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/3453475616' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "315cd205-cf98-4743-8b62-9a158d99a4e4"}]: dispatch 2024-04-27T02:01:11.061 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:10 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/3453475616' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "315cd205-cf98-4743-8b62-9a158d99a4e4"}]': finished 2024-04-27T02:01:11.061 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:10 smithi029 ceph-mon[26769]: osdmap e16: 3 total, 2 up, 3 in 2024-04-27T02:01:11.061 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:10 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:01:11.061 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:10 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/2760294891' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:01:11.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:10 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/3453475616' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "315cd205-cf98-4743-8b62-9a158d99a4e4"}]: dispatch 2024-04-27T02:01:11.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:10 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/3453475616' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "315cd205-cf98-4743-8b62-9a158d99a4e4"}]': finished 2024-04-27T02:01:11.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:10 smithi052 ceph-mon[32586]: osdmap e16: 3 total, 2 up, 3 in 2024-04-27T02:01:11.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:10 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:01:11.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:10 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/2760294891' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:01:12.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:11 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:12.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:11 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:12.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:11 smithi052 ceph-mon[32586]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:12.291 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:11 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:12.291 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:11 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:12.292 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:11 smithi029 ceph-mon[26769]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:14.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:14 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:14.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:14 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:14.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:14 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:01:14.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:14 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:01:14.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:14 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:14.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:14 smithi029 ceph-mon[26769]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:15.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:14 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:15.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:14 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:15.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:14 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:01:15.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:14 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:01:15.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:14 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:15.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:14 smithi052 ceph-mon[32586]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:16.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:16 smithi029 ceph-mon[26769]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:17.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:16 smithi052 ceph-mon[32586]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:18.152 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:17 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-27T02:01:18.152 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:17 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:01:18.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:17 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-27T02:01:18.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:17 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:01:19.083 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:18 smithi029 ceph-mon[26769]: Deploying daemon osd.2 on smithi029 2024-04-27T02:01:19.083 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:18 smithi029 ceph-mon[26769]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:19.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:18 smithi052 ceph-mon[32586]: Deploying daemon osd.2 on smithi029 2024-04-27T02:01:19.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:18 smithi052 ceph-mon[32586]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:21.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:20 smithi029 ceph-mon[26769]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:21.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:20 smithi052 ceph-mon[32586]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:22.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:22 smithi029 ceph-mon[26769]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:22.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:22 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:22.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:22 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:22.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:22 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:01:22.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:22 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:01:23.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:22 smithi052 ceph-mon[32586]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:23.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:22 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:23.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:22 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:23.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:22 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:01:23.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:22 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:01:25.063 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:24 smithi029 ceph-mon[26769]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:25.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:24 smithi052 ceph-mon[32586]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:27.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:27.101+0000 7f7292ffd700 1 -- 172.21.15.29:0/3034013333 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f729c068c50 con 0x7f728806c600 2024-04-27T02:01:27.082 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 2 on host 'smithi029' 2024-04-27T02:01:27.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:27.104+0000 7f72a3cde700 1 -- 172.21.15.29:0/3034013333 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f728806c600 msgr2=0x7f728806eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:01:27.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:27.104+0000 7f72a3cde700 1 --2- 172.21.15.29:0/3034013333 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f728806c600 0x7f728806eab0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7298006010 tx=0x7f729800b540 comp rx=0 tx=0).stop 2024-04-27T02:01:27.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:27.104+0000 7f72a3cde700 1 -- 172.21.15.29:0/3034013333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f729c1003d0 msgr2=0x7f729c1017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:01:27.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:27.104+0000 7f72a3cde700 1 --2- 172.21.15.29:0/3034013333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f729c1003d0 0x7f729c1017b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f728c00d8d0 tx=0x7f728c00dbe0 comp rx=0 tx=0).stop 2024-04-27T02:01:27.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:27.104+0000 7f72a3cde700 1 -- 172.21.15.29:0/3034013333 shutdown_connections 2024-04-27T02:01:27.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:27.104+0000 7f72a3cde700 1 --2- 172.21.15.29:0/3034013333 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f728806c600 0x7f728806eab0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:01:27.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:27.104+0000 7f72a3cde700 1 --2- 172.21.15.29:0/3034013333 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f729c0ff130 0x7f729c103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:01:27.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:27.104+0000 7f72a3cde700 1 --2- 172.21.15.29:0/3034013333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f729c1003d0 0x7f729c1017b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:01:27.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:27.104+0000 7f72a3cde700 1 -- 172.21.15.29:0/3034013333 >> 172.21.15.29:0/3034013333 conn(0x7f729c0fa780 msgr2=0x7f729c068540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:01:27.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:27.104+0000 7f72a3cde700 1 -- 172.21.15.29:0/3034013333 shutdown_connections 2024-04-27T02:01:27.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:01:27.104+0000 7f72a3cde700 1 -- 172.21.15.29:0/3034013333 wait complete. 2024-04-27T02:01:27.098 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:26 smithi029 ceph-mon[26769]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:27.098 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:26 smithi029 ceph-mon[26769]: from='osd.2 [v2:172.21.15.29:6818/3013669823,v1:172.21.15.29:6819/3013669823]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-27T02:01:27.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:26 smithi052 ceph-mon[32586]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:27.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:26 smithi052 ceph-mon[32586]: from='osd.2 [v2:172.21.15.29:6818/3013669823,v1:172.21.15.29:6819/3013669823]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-27T02:01:27.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:27 smithi029 ceph-mon[26769]: from='osd.2 [v2:172.21.15.29:6818/3013669823,v1:172.21.15.29:6819/3013669823]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-27T02:01:27.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:27 smithi029 ceph-mon[26769]: osdmap e17: 3 total, 2 up, 3 in 2024-04-27T02:01:27.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:27 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:01:27.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:27 smithi029 ceph-mon[26769]: from='osd.2 [v2:172.21.15.29:6818/3013669823,v1:172.21.15.29:6819/3013669823]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-27T02:01:27.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:27 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:27.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:27 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:28.099 DEBUG:teuthology.orchestra.run.smithi029:osd.2> sudo journalctl -f -n 0 -u ceph-4c587076-0439-11ef-bc93-c7b262605968@osd.2.service 2024-04-27T02:01:28.102 INFO:tasks.cephadm:Deploying osd.3 on smithi052 with /dev/vg_nvme/lv_4... 2024-04-27T02:01:28.102 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-27T02:01:28.130 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:27 smithi052 ceph-mon[32586]: from='osd.2 [v2:172.21.15.29:6818/3013669823,v1:172.21.15.29:6819/3013669823]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-27T02:01:28.130 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:27 smithi052 ceph-mon[32586]: osdmap e17: 3 total, 2 up, 3 in 2024-04-27T02:01:28.130 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:27 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:01:28.130 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:27 smithi052 ceph-mon[32586]: from='osd.2 [v2:172.21.15.29:6818/3013669823,v1:172.21.15.29:6819/3013669823]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-27T02:01:28.130 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:27 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:28.130 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:27 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:28.320 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi052/config 2024-04-27T02:01:28.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:28 smithi029 ceph-mon[26769]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:28.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:28 smithi029 ceph-mon[26769]: from='osd.2 [v2:172.21.15.29:6818/3013669823,v1:172.21.15.29:6819/3013669823]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-04-27T02:01:28.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:28 smithi029 ceph-mon[26769]: osdmap e18: 3 total, 2 up, 3 in 2024-04-27T02:01:28.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:28 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:01:28.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:28 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:01:29.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:28 smithi052 ceph-mon[32586]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:29.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:28 smithi052 ceph-mon[32586]: from='osd.2 [v2:172.21.15.29:6818/3013669823,v1:172.21.15.29:6819/3013669823]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-04-27T02:01:29.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:28 smithi052 ceph-mon[32586]: osdmap e18: 3 total, 2 up, 3 in 2024-04-27T02:01:29.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:28 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:01:29.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:28 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:01:30.328 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-27T02:01:30.328 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-27T02:01:30.328 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10+0 records in 2024-04-27T02:01:30.328 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records out 2024-04-27T02:01:30.328 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00822655 s, 1.3 GB/s 2024-04-27T02:01:30.328 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping successful for: 2024-04-27T02:01:30.626 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:30 smithi052 ceph-mon[32586]: purged_snaps scrub starts 2024-04-27T02:01:30.626 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:30 smithi052 ceph-mon[32586]: purged_snaps scrub ok 2024-04-27T02:01:30.626 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:30 smithi052 ceph-mon[32586]: osd.2 [v2:172.21.15.29:6818/3013669823,v1:172.21.15.29:6819/3013669823] boot 2024-04-27T02:01:30.626 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:30 smithi052 ceph-mon[32586]: osdmap e19: 3 total, 3 up, 3 in 2024-04-27T02:01:30.626 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:30 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:01:30.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:30 smithi029 ceph-mon[26769]: purged_snaps scrub starts 2024-04-27T02:01:30.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:30 smithi029 ceph-mon[26769]: purged_snaps scrub ok 2024-04-27T02:01:30.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:30 smithi029 ceph-mon[26769]: osd.2 [v2:172.21.15.29:6818/3013669823,v1:172.21.15.29:6819/3013669823] boot 2024-04-27T02:01:30.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:30 smithi029 ceph-mon[26769]: osdmap e19: 3 total, 3 up, 3 in 2024-04-27T02:01:30.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:30 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:01:30.956 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi052:vg_nvme/lv_4 2024-04-27T02:01:31.209 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi052/config 2024-04-27T02:01:31.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:31 smithi029 ceph-mon[26769]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:31.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:31 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-27T02:01:31.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:31 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-27T02:01:31.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:31 smithi029 ceph-mon[26769]: osdmap e20: 3 total, 3 up, 3 in 2024-04-27T02:01:31.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:31 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:31.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:31 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T02:01:31.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:31 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:31.571 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:31 smithi052 ceph-mon[32586]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:01:31.571 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:31 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-27T02:01:31.571 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:31 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-27T02:01:31.571 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:31 smithi052 ceph-mon[32586]: osdmap e20: 3 total, 3 up, 3 in 2024-04-27T02:01:31.571 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:31 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:31.571 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:31 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T02:01:31.571 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:31 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:32.050 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:01:31 smithi029 sudo[54706]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T02:01:32.050 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:01:31 smithi029 sudo[54706]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T02:01:32.051 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:01:31 smithi029 sudo[54706]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T02:01:32.051 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:01:31 smithi029 sudo[54706]: pam_unix(sudo:session): session closed for user root 2024-04-27T02:01:32.381 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:32 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-27T02:01:32.381 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:32 smithi029 ceph-mon[26769]: osdmap e21: 3 total, 3 up, 3 in 2024-04-27T02:01:32.381 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:32 smithi029 ceph-mon[26769]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:01:32.399 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:32 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-27T02:01:32.399 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:32 smithi052 ceph-mon[32586]: osdmap e21: 3 total, 3 up, 3 in 2024-04-27T02:01:32.399 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:32 smithi052 ceph-mon[32586]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:01:32.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:01:32 smithi029 sudo[54718]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T02:01:32.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:01:32 smithi029 sudo[54718]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T02:01:32.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:01:32 smithi029 sudo[54718]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T02:01:32.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:01:32 smithi029 sudo[54718]: pam_unix(sudo:session): session closed for user root 2024-04-27T02:01:32.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.880+0000 7f3ef18c5700 1 -- 172.21.15.52:0/1852248144 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3eec100240 msgr2=0x7f3eec100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:01:32.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.880+0000 7f3ef18c5700 1 --2- 172.21.15.52:0/1852248144 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3eec100240 0x7f3eec100650 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3edc009910 tx=0x7f3edc009c20 comp rx=0 tx=0).stop 2024-04-27T02:01:32.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.880+0000 7f3ef18c5700 1 -- 172.21.15.52:0/1852248144 shutdown_connections 2024-04-27T02:01:32.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.880+0000 7f3ef18c5700 1 --2- 172.21.15.52:0/1852248144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3eec101440 0x7f3eec101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:01:32.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.880+0000 7f3ef18c5700 1 --2- 172.21.15.52:0/1852248144 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3eec100240 0x7f3eec100650 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:01:32.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.880+0000 7f3ef18c5700 1 -- 172.21.15.52:0/1852248144 >> 172.21.15.52:0/1852248144 conn(0x7f3eec0fb7f0 msgr2=0x7f3eec0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:01:32.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.880+0000 7f3ef18c5700 1 -- 172.21.15.52:0/1852248144 shutdown_connections 2024-04-27T02:01:32.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.880+0000 7f3ef18c5700 1 -- 172.21.15.52:0/1852248144 wait complete. 2024-04-27T02:01:32.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.881+0000 7f3ef18c5700 1 Processor -- start 2024-04-27T02:01:32.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.881+0000 7f3ef18c5700 1 -- start start 2024-04-27T02:01:32.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.881+0000 7f3ef18c5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3eec100240 0x7f3eec074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:01:32.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.881+0000 7f3ef18c5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3eec101440 0x7f3eec073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:01:32.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.881+0000 7f3ef18c5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eec073680 con 0x7f3eec101440 2024-04-27T02:01:32.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.881+0000 7f3ef18c5700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eec0737c0 con 0x7f3eec100240 2024-04-27T02:01:32.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.881+0000 7f3eeaffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3eec100240 0x7f3eec074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:01:32.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.881+0000 7f3eeaffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3eec100240 0x7f3eec074af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55006/0 (socket says 172.21.15.52:55006) 2024-04-27T02:01:32.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.881+0000 7f3eeaffd700 1 -- 172.21.15.52:0/176944261 learned_addr learned my addr 172.21.15.52:0/176944261 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T02:01:32.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.881+0000 7f3eea7fc700 1 --2- 172.21.15.52:0/176944261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3eec101440 0x7f3eec073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:01:32.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.881+0000 7f3eeaffd700 1 -- 172.21.15.52:0/176944261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3eec101440 msgr2=0x7f3eec073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:01:32.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.881+0000 7f3eeaffd700 1 --2- 172.21.15.52:0/176944261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3eec101440 0x7f3eec073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:01:32.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.881+0000 7f3eeaffd700 1 -- 172.21.15.52:0/176944261 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3edc0095f0 con 0x7f3eec100240 2024-04-27T02:01:32.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.882+0000 7f3eeaffd700 1 --2- 172.21.15.52:0/176944261 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3eec100240 0x7f3eec074af0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3edc00bda0 tx=0x7f3edc00bdd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:01:32.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.882+0000 7f3ef08c3700 1 -- 172.21.15.52:0/176944261 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3edc01d070 con 0x7f3eec100240 2024-04-27T02:01:32.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.882+0000 7f3ef08c3700 1 -- 172.21.15.52:0/176944261 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3edc00f460 con 0x7f3eec100240 2024-04-27T02:01:32.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.882+0000 7f3ef08c3700 1 -- 172.21.15.52:0/176944261 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3edc021640 con 0x7f3eec100240 2024-04-27T02:01:32.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.882+0000 7f3ef18c5700 1 -- 172.21.15.52:0/176944261 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eec073a40 con 0x7f3eec100240 2024-04-27T02:01:32.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.882+0000 7f3ef18c5700 1 -- 172.21.15.52:0/176944261 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eec073f30 con 0x7f3eec100240 2024-04-27T02:01:32.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.883+0000 7f3ef18c5700 1 -- 172.21.15.52:0/176944261 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eec108d90 con 0x7f3eec100240 2024-04-27T02:01:32.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.883+0000 7f3ef08c3700 1 -- 172.21.15.52:0/176944261 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f3edc00f5d0 con 0x7f3eec100240 2024-04-27T02:01:32.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.884+0000 7f3ef08c3700 1 --2- 172.21.15.52:0/176944261 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3ed406c680 0x7f3ed406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:01:32.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.884+0000 7f3ef08c3700 1 -- 172.21.15.52:0/176944261 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3232+0+0 (secure 0 0 0) 0x7f3edc08c240 con 0x7f3eec100240 2024-04-27T02:01:32.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.884+0000 7f3eea7fc700 1 --2- 172.21.15.52:0/176944261 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3ed406c680 0x7f3ed406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:01:32.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.884+0000 7f3eea7fc700 1 --2- 172.21.15.52:0/176944261 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3ed406c680 0x7f3ed406eb30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0007be0 tx=0x7f3ee00073d0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:01:32.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:32.889+0000 7f3ef08c3700 1 -- 172.21.15.52:0/176944261 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3edc05ac70 con 0x7f3eec100240 2024-04-27T02:01:33.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:01:32 smithi029 sudo[55110]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T02:01:33.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:01:32 smithi029 sudo[55110]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T02:01:33.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:01:32 smithi029 sudo[55110]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T02:01:33.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:01:32 smithi029 sudo[55110]: pam_unix(sudo:session): session closed for user root 2024-04-27T02:01:33.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:33.096+0000 7f3ef18c5700 1 -- 172.21.15.52:0/176944261 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f3eec19df60 con 0x7f3ed406c680 2024-04-27T02:01:33.318 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:33 smithi029 ceph-mon[26769]: mgrmap e19: smithi029.ewvdzs(active, since 2m), standbys: smithi052.gbsftx 2024-04-27T02:01:33.318 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:33 smithi029 ceph-mon[26769]: osdmap e22: 3 total, 3 up, 3 in 2024-04-27T02:01:33.318 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:33 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:33.318 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:33 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:01:33.318 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:33 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:01:33.318 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:33 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:01:33.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:33 smithi052 ceph-mon[32586]: mgrmap e19: smithi029.ewvdzs(active, since 2m), standbys: smithi052.gbsftx 2024-04-27T02:01:33.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:33 smithi052 ceph-mon[32586]: osdmap e22: 3 total, 3 up, 3 in 2024-04-27T02:01:33.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:33 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:33.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:33 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:01:33.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:33 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:01:33.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:33 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:01:33.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:01:33 smithi029 sudo[55114]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T02:01:33.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:01:33 smithi029 sudo[55114]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T02:01:33.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:01:33 smithi029 sudo[55114]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T02:01:33.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:01:33 smithi029 sudo[55114]: pam_unix(sudo:session): session closed for user root 2024-04-27T02:01:34.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:01:33 smithi029 sudo[55215]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T02:01:34.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:01:33 smithi029 sudo[55215]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T02:01:34.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:01:33 smithi029 sudo[55215]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T02:01:34.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:01:33 smithi029 sudo[55215]: pam_unix(sudo:session): session closed for user root 2024-04-27T02:01:34.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:01:34 smithi029 sudo[55306]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T02:01:34.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:01:34 smithi029 sudo[55306]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T02:01:34.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:01:34 smithi029 sudo[55306]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T02:01:34.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:01:34 smithi029 sudo[55306]: pam_unix(sudo:session): session closed for user root 2024-04-27T02:01:34.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:34 smithi029 sudo[55327]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T02:01:34.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:34 smithi029 sudo[55327]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T02:01:34.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:34 smithi029 sudo[55327]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T02:01:34.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:34 smithi029 sudo[55327]: pam_unix(sudo:session): session closed for user root 2024-04-27T02:01:35.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:35 smithi052 ceph-mon[32586]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:01:35.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:35 smithi052 ceph-mon[32586]: Detected new or changed devices on smithi029 2024-04-27T02:01:35.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:35 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:35.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:35 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:35.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:35 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:01:35.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:35 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:01:35.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:35 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:01:35.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:35 smithi052 ceph-mon[32586]: Adjusting osd_memory_target on smithi029 to 3948M 2024-04-27T02:01:35.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:35 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:35.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:35 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:01:35.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:35 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:01:35.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:35 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:35.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:35 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:01:35.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:35 smithi052 ceph-mon[32586]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:01:35.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:35 smithi052 ceph-mon[32586]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T02:01:35.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:35 smithi029 sudo[55432]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T02:01:35.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:35 smithi029 sudo[55432]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T02:01:35.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:35 smithi029 sudo[55432]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T02:01:35.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:35 smithi029 sudo[55432]: pam_unix(sudo:session): session closed for user root 2024-04-27T02:01:35.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:35 smithi029 ceph-mon[26769]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:01:35.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:35 smithi029 ceph-mon[26769]: Detected new or changed devices on smithi029 2024-04-27T02:01:35.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:35 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:35.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:35 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:35.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:35 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:01:35.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:35 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:01:35.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:35 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:01:35.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:35 smithi029 ceph-mon[26769]: Adjusting osd_memory_target on smithi029 to 3948M 2024-04-27T02:01:35.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:35 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:35.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:35 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:01:35.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:35 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:01:35.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:35 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:35.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:35 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:01:35.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:35 smithi029 ceph-mon[26769]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:01:35.452 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:35 smithi029 ceph-mon[26769]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T02:01:36.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:35 smithi052 sudo[36554]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T02:01:36.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:35 smithi052 sudo[36554]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T02:01:36.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:35 smithi052 sudo[36554]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T02:01:36.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:35 smithi052 sudo[36554]: pam_unix(sudo:session): session closed for user root 2024-04-27T02:01:36.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:36 smithi029 ceph-mon[26769]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T02:01:36.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:36 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-04-27T02:01:36.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:36 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2024-04-27T02:01:36.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:36 smithi029 ceph-mon[26769]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T02:01:36.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:36 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-04-27T02:01:36.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:36 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2024-04-27T02:01:36.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:36 smithi029 ceph-mon[26769]: pgmap v84: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:01:36.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:36 smithi029 ceph-mon[26769]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T02:01:36.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:36 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:36.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:36 smithi052 sudo[36557]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T02:01:36.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:36 smithi052 sudo[36557]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T02:01:36.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:36 smithi052 sudo[36557]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T02:01:36.505 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:36 smithi052 sudo[36557]: pam_unix(sudo:session): session closed for user root 2024-04-27T02:01:36.505 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:36 smithi052 ceph-mon[32586]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T02:01:36.505 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:36 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-04-27T02:01:36.505 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:36 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2024-04-27T02:01:36.505 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:36 smithi052 ceph-mon[32586]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T02:01:36.505 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:36 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-04-27T02:01:36.506 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:36 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2024-04-27T02:01:36.506 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:36 smithi052 ceph-mon[32586]: pgmap v84: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:01:36.506 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:36 smithi052 ceph-mon[32586]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T02:01:36.506 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:36 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:37.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:37 smithi052 ceph-mon[32586]: from='client.? 172.21.15.52:0/1023307063' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59548c0e-dc14-4d71-940c-58802631725b"}]: dispatch 2024-04-27T02:01:37.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:37 smithi052 ceph-mon[32586]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59548c0e-dc14-4d71-940c-58802631725b"}]: dispatch 2024-04-27T02:01:37.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:37 smithi052 ceph-mon[32586]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59548c0e-dc14-4d71-940c-58802631725b"}]': finished 2024-04-27T02:01:37.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:37 smithi052 ceph-mon[32586]: osdmap e23: 4 total, 3 up, 4 in 2024-04-27T02:01:37.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:37 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:01:37.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:37 smithi052 ceph-mon[32586]: from='client.? 172.21.15.52:0/2517663242' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:01:37.385 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:37 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/1023307063' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59548c0e-dc14-4d71-940c-58802631725b"}]: dispatch 2024-04-27T02:01:37.385 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:37 smithi029 ceph-mon[26769]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59548c0e-dc14-4d71-940c-58802631725b"}]: dispatch 2024-04-27T02:01:37.385 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:37 smithi029 ceph-mon[26769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59548c0e-dc14-4d71-940c-58802631725b"}]': finished 2024-04-27T02:01:37.385 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:37 smithi029 ceph-mon[26769]: osdmap e23: 4 total, 3 up, 4 in 2024-04-27T02:01:37.385 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:37 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:01:37.385 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:37 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/2517663242' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:01:38.450 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:38 smithi029 ceph-mon[26769]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.9 KiB/s rd, 238 KiB/s wr, 16 op/s 2024-04-27T02:01:38.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:38 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:01:38.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:38 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:38.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:38 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:38.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:38 smithi052 ceph-mon[32586]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.9 KiB/s rd, 238 KiB/s wr, 16 op/s 2024-04-27T02:01:38.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:38 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:01:38.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:38 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:38.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:38 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:39.455 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:39 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:01:39.455 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:39 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:01:39.455 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:39 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:39.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:39 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:01:39.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:39 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:01:39.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:39 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:40.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:40 smithi052 ceph-mon[32586]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 175 KiB/s wr, 12 op/s 2024-04-27T02:01:40.450 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:40 smithi029 ceph-mon[26769]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 175 KiB/s wr, 12 op/s 2024-04-27T02:01:43.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:42 smithi029 ceph-mon[26769]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 138 KiB/s wr, 9 op/s 2024-04-27T02:01:43.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:42 smithi052 ceph-mon[32586]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 138 KiB/s wr, 9 op/s 2024-04-27T02:01:44.147 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:43 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-27T02:01:44.147 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:43 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:01:44.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:43 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-27T02:01:44.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:43 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:01:44.995 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:44 smithi052 ceph-mon[32586]: Deploying daemon osd.3 on smithi052 2024-04-27T02:01:44.995 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:44 smithi052 ceph-mon[32586]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-27T02:01:45.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:44 smithi029 ceph-mon[26769]: Deploying daemon osd.3 on smithi052 2024-04-27T02:01:45.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:44 smithi029 ceph-mon[26769]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-27T02:01:47.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:46 smithi052 ceph-mon[32586]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-27T02:01:47.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:46 smithi029 ceph-mon[26769]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-27T02:01:49.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:48 smithi029 ceph-mon[26769]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.3 KiB/s rd, 114 KiB/s wr, 7 op/s 2024-04-27T02:01:49.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:48 smithi052 ceph-mon[32586]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.3 KiB/s rd, 114 KiB/s wr, 7 op/s 2024-04-27T02:01:50.450 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:50 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:50.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:50 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:50.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:50 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:01:50.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:50 smithi029 ceph-mon[26769]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:01:50.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:50 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:50.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:50 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:50.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:50 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:01:50.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:50 smithi052 ceph-mon[32586]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:01:52.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:52.461+0000 7f3ef08c3700 1 -- 172.21.15.52:0/176944261 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3eec19df60 con 0x7f3ed406c680 2024-04-27T02:01:52.443 INFO:teuthology.orchestra.run.smithi052.stdout:Created osd(s) 3 on host 'smithi052' 2024-04-27T02:01:52.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:52.463+0000 7f3ef18c5700 1 -- 172.21.15.52:0/176944261 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3ed406c680 msgr2=0x7f3ed406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:01:52.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:52.463+0000 7f3ef18c5700 1 --2- 172.21.15.52:0/176944261 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3ed406c680 0x7f3ed406eb30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0007be0 tx=0x7f3ee00073d0 comp rx=0 tx=0).stop 2024-04-27T02:01:52.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:52.463+0000 7f3ef18c5700 1 -- 172.21.15.52:0/176944261 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3eec100240 msgr2=0x7f3eec074af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:01:52.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:52.464+0000 7f3ef18c5700 1 --2- 172.21.15.52:0/176944261 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3eec100240 0x7f3eec074af0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3edc00bda0 tx=0x7f3edc00bdd0 comp rx=0 tx=0).stop 2024-04-27T02:01:52.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:52.464+0000 7f3ef18c5700 1 -- 172.21.15.52:0/176944261 shutdown_connections 2024-04-27T02:01:52.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:52.464+0000 7f3ef18c5700 1 --2- 172.21.15.52:0/176944261 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3ed406c680 0x7f3ed406eb30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:01:52.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:52.464+0000 7f3ef18c5700 1 --2- 172.21.15.52:0/176944261 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3eec100240 0x7f3eec074af0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:01:52.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:52.464+0000 7f3ef18c5700 1 --2- 172.21.15.52:0/176944261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3eec101440 0x7f3eec073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:01:52.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:52.464+0000 7f3ef18c5700 1 -- 172.21.15.52:0/176944261 >> 172.21.15.52:0/176944261 conn(0x7f3eec0fb7f0 msgr2=0x7f3eec104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:01:52.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:52.464+0000 7f3ef18c5700 1 -- 172.21.15.52:0/176944261 shutdown_connections 2024-04-27T02:01:52.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:52.464+0000 7f3ef18c5700 1 -- 172.21.15.52:0/176944261 wait complete. 2024-04-27T02:01:53.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:52 smithi052 ceph-mon[32586]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:01:53.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:52 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:53.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:52 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:53.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:52 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:01:53.055 DEBUG:teuthology.orchestra.run.smithi052:osd.3> sudo journalctl -f -n 0 -u ceph-4c587076-0439-11ef-bc93-c7b262605968@osd.3.service 2024-04-27T02:01:53.058 INFO:tasks.cephadm:Deploying osd.4 on smithi052 with /dev/vg_nvme/lv_3... 2024-04-27T02:01:53.058 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-27T02:01:53.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:52 smithi029 ceph-mon[26769]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:01:53.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:52 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:53.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:52 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:53.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:52 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:01:53.755 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi052/config 2024-04-27T02:01:54.113 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:53 smithi052 ceph-mon[32586]: from='osd.3 [v2:172.21.15.52:6800/263370941,v1:172.21.15.52:6801/263370941]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-27T02:01:54.114 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:53 smithi052 ceph-mon[32586]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-27T02:01:54.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:53 smithi029 ceph-mon[26769]: from='osd.3 [v2:172.21.15.52:6800/263370941,v1:172.21.15.52:6801/263370941]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-27T02:01:54.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:53 smithi029 ceph-mon[26769]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-27T02:01:55.169 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:54 smithi052 ceph-mon[32586]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:01:55.169 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:54 smithi052 ceph-mon[32586]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-27T02:01:55.169 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:54 smithi052 ceph-mon[32586]: osdmap e24: 4 total, 3 up, 4 in 2024-04-27T02:01:55.169 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:54 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:01:55.169 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:54 smithi052 ceph-mon[32586]: from='osd.3 [v2:172.21.15.52:6800/263370941,v1:172.21.15.52:6801/263370941]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-27T02:01:55.169 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:54 smithi052 ceph-mon[32586]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-27T02:01:55.170 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 27 02:01:54 smithi052 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-3[38663]: 2024-04-27T02:01:54.843+0000 7fb465542700 -1 osd.3 0 waiting for initial osdmap 2024-04-27T02:01:55.170 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 27 02:01:54 smithi052 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-3[38663]: 2024-04-27T02:01:54.850+0000 7fb45fb34700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T02:01:55.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:54 smithi029 ceph-mon[26769]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:01:55.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:54 smithi029 ceph-mon[26769]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-27T02:01:55.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:54 smithi029 ceph-mon[26769]: osdmap e24: 4 total, 3 up, 4 in 2024-04-27T02:01:55.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:54 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:01:55.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:54 smithi029 ceph-mon[26769]: from='osd.3 [v2:172.21.15.52:6800/263370941,v1:172.21.15.52:6801/263370941]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-27T02:01:55.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:54 smithi029 ceph-mon[26769]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-27T02:01:55.698 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-27T02:01:55.698 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-27T02:01:55.698 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10+0 records in 2024-04-27T02:01:55.699 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records out 2024-04-27T02:01:55.699 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0094059 s, 1.1 GB/s 2024-04-27T02:01:55.699 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping successful for: 2024-04-27T02:01:56.087 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:55 smithi052 ceph-mon[32586]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-04-27T02:01:56.087 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:55 smithi052 ceph-mon[32586]: osdmap e25: 4 total, 3 up, 4 in 2024-04-27T02:01:56.087 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:55 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:01:56.087 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:55 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:01:56.087 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:55 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:56.087 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:55 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:56.087 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:55 smithi052 ceph-mon[32586]: osd.3 [v2:172.21.15.52:6800/263370941,v1:172.21.15.52:6801/263370941] boot 2024-04-27T02:01:56.088 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:55 smithi052 ceph-mon[32586]: osdmap e26: 4 total, 4 up, 4 in 2024-04-27T02:01:56.088 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:55 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:01:56.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:55 smithi029 ceph-mon[26769]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-04-27T02:01:56.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:55 smithi029 ceph-mon[26769]: osdmap e25: 4 total, 3 up, 4 in 2024-04-27T02:01:56.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:55 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:01:56.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:55 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:01:56.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:55 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:56.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:55 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:01:56.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:55 smithi029 ceph-mon[26769]: osd.3 [v2:172.21.15.52:6800/263370941,v1:172.21.15.52:6801/263370941] boot 2024-04-27T02:01:56.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:55 smithi029 ceph-mon[26769]: osdmap e26: 4 total, 4 up, 4 in 2024-04-27T02:01:56.103 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:55 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:01:56.789 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi052:vg_nvme/lv_3 2024-04-27T02:01:57.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:56 smithi029 ceph-mon[26769]: purged_snaps scrub starts 2024-04-27T02:01:57.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:56 smithi029 ceph-mon[26769]: purged_snaps scrub ok 2024-04-27T02:01:57.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:56 smithi029 ceph-mon[26769]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:01:57.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:56 smithi029 ceph-mon[26769]: osdmap e27: 4 total, 4 up, 4 in 2024-04-27T02:01:57.229 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:56 smithi052 ceph-mon[32586]: purged_snaps scrub starts 2024-04-27T02:01:57.230 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:56 smithi052 ceph-mon[32586]: purged_snaps scrub ok 2024-04-27T02:01:57.230 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:56 smithi052 ceph-mon[32586]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:01:57.230 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:56 smithi052 ceph-mon[32586]: osdmap e27: 4 total, 4 up, 4 in 2024-04-27T02:01:57.730 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi052/config 2024-04-27T02:01:59.040 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:58 smithi052 ceph-mon[32586]: pgmap v100: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:01:59.041 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:58 smithi052 ceph-mon[32586]: osdmap e28: 4 total, 4 up, 4 in 2024-04-27T02:01:59.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:58 smithi029 ceph-mon[26769]: pgmap v100: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:01:59.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:58 smithi029 ceph-mon[26769]: osdmap e28: 4 total, 4 up, 4 in 2024-04-27T02:01:59.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.926+0000 7fb45ced3700 1 -- 172.21.15.52:0/144509891 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb458100340 msgr2=0x7fb458100790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:01:59.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.926+0000 7fb45ced3700 1 --2- 172.21.15.52:0/144509891 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb458100340 0x7fb458100790 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb448009910 tx=0x7fb448009c20 comp rx=0 tx=0).stop 2024-04-27T02:01:59.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.926+0000 7fb45ced3700 1 -- 172.21.15.52:0/144509891 shutdown_connections 2024-04-27T02:01:59.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.926+0000 7fb45ced3700 1 --2- 172.21.15.52:0/144509891 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb458100340 0x7fb458100790 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:01:59.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.926+0000 7fb45ced3700 1 --2- 172.21.15.52:0/144509891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4580ff140 0x7fb4580ff550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:01:59.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.926+0000 7fb45ced3700 1 -- 172.21.15.52:0/144509891 >> 172.21.15.52:0/144509891 conn(0x7fb4580fa6d0 msgr2=0x7fb4580fcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:01:59.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.926+0000 7fb45ced3700 1 -- 172.21.15.52:0/144509891 shutdown_connections 2024-04-27T02:01:59.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.926+0000 7fb45ced3700 1 -- 172.21.15.52:0/144509891 wait complete. 2024-04-27T02:01:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.927+0000 7fb45ced3700 1 Processor -- start 2024-04-27T02:01:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.927+0000 7fb45ced3700 1 -- start start 2024-04-27T02:01:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.927+0000 7fb45ced3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4580ff140 0x7fb458195a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:01:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.927+0000 7fb45ced3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb458100340 0x7fb458195fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:01:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.927+0000 7fb45ced3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4581965c0 con 0x7fb458100340 2024-04-27T02:01:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.927+0000 7fb45ced3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb458196700 con 0x7fb4580ff140 2024-04-27T02:01:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.927+0000 7fb45659c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4580ff140 0x7fb458195a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:01:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.928+0000 7fb45659c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4580ff140 0x7fb458195a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58010/0 (socket says 172.21.15.52:58010) 2024-04-27T02:01:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.928+0000 7fb45659c700 1 -- 172.21.15.52:0/169933226 learned_addr learned my addr 172.21.15.52:0/169933226 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T02:01:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.928+0000 7fb455d9b700 1 --2- 172.21.15.52:0/169933226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb458100340 0x7fb458195fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:01:59.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.928+0000 7fb45659c700 1 -- 172.21.15.52:0/169933226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb458100340 msgr2=0x7fb458195fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:01:59.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.928+0000 7fb45659c700 1 --2- 172.21.15.52:0/169933226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb458100340 0x7fb458195fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:01:59.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.928+0000 7fb45659c700 1 -- 172.21.15.52:0/169933226 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4480095f0 con 0x7fb4580ff140 2024-04-27T02:01:59.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.928+0000 7fb45659c700 1 --2- 172.21.15.52:0/169933226 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4580ff140 0x7fb458195a60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb44000d900 tx=0x7fb44000dc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:01:59.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.928+0000 7fb44f7fe700 1 -- 172.21.15.52:0/169933226 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4400041f0 con 0x7fb4580ff140 2024-04-27T02:01:59.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.928+0000 7fb45ced3700 1 -- 172.21.15.52:0/169933226 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb45819b1b0 con 0x7fb4580ff140 2024-04-27T02:01:59.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.928+0000 7fb44f7fe700 1 -- 172.21.15.52:0/169933226 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb440004350 con 0x7fb4580ff140 2024-04-27T02:01:59.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.928+0000 7fb44f7fe700 1 -- 172.21.15.52:0/169933226 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb440003d20 con 0x7fb4580ff140 2024-04-27T02:01:59.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.928+0000 7fb45ced3700 1 -- 172.21.15.52:0/169933226 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb45819b700 con 0x7fb4580ff140 2024-04-27T02:01:59.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.929+0000 7fb45ced3700 1 -- 172.21.15.52:0/169933226 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb45818fb80 con 0x7fb4580ff140 2024-04-27T02:01:59.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.930+0000 7fb44f7fe700 1 -- 172.21.15.52:0/169933226 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fb440009730 con 0x7fb4580ff140 2024-04-27T02:01:59.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.930+0000 7fb44f7fe700 1 --2- 172.21.15.52:0/169933226 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb44406c680 0x7fb44406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:01:59.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.930+0000 7fb44f7fe700 1 -- 172.21.15.52:0/169933226 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3774+0+0 (secure 0 0 0) 0x7fb44008b0e0 con 0x7fb4580ff140 2024-04-27T02:01:59.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.931+0000 7fb455d9b700 1 --2- 172.21.15.52:0/169933226 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb44406c680 0x7fb44406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:01:59.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.931+0000 7fb455d9b700 1 --2- 172.21.15.52:0/169933226 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb44406c680 0x7fb44406eb30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb448006010 tx=0x7fb44800b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:01:59.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:01:59.935+0000 7fb44f7fe700 1 -- 172.21.15.52:0/169933226 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb4400598f0 con 0x7fb4580ff140 2024-04-27T02:02:00.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:59 smithi052 ceph-mon[32586]: osdmap e29: 4 total, 4 up, 4 in 2024-04-27T02:02:00.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:59 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:00.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:59 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:00.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:59 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:02:00.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:59 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:00.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:59 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:02:00.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:59 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:02:00.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:59 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:00.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:59 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:02:00.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:59 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:02:00.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:59 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:02:00.005 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:01:59 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:00.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:00.145+0000 7fb45ced3700 1 -- 172.21.15.52:0/169933226 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fb458061190 con 0x7fb44406c680 2024-04-27T02:02:00.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:59 smithi029 ceph-mon[26769]: osdmap e29: 4 total, 4 up, 4 in 2024-04-27T02:02:00.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:59 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:00.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:59 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:00.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:59 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:02:00.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:59 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:00.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:59 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:02:00.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:59 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:02:00.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:59 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:00.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:59 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:02:00.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:59 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:02:00.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:59 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:02:00.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:01:59 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:01.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:00 smithi052 ceph-mon[32586]: Detected new or changed devices on smithi052 2024-04-27T02:02:01.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:00 smithi052 ceph-mon[32586]: Adjusting osd_memory_target on smithi052 to 14917M 2024-04-27T02:02:01.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:00 smithi052 ceph-mon[32586]: pgmap v103: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:02:01.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:00 smithi052 ceph-mon[32586]: osdmap e30: 4 total, 4 up, 4 in 2024-04-27T02:02:01.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:00 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:02:01.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:00 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:02:01.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:00 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:02:01.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:00 smithi029 ceph-mon[26769]: Detected new or changed devices on smithi052 2024-04-27T02:02:01.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:00 smithi029 ceph-mon[26769]: Adjusting osd_memory_target on smithi052 to 14917M 2024-04-27T02:02:01.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:00 smithi029 ceph-mon[26769]: pgmap v103: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:02:01.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:00 smithi029 ceph-mon[26769]: osdmap e30: 4 total, 4 up, 4 in 2024-04-27T02:02:01.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:00 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:02:01.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:00 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:02:01.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:00 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:02:02.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:01 smithi052 ceph-mon[32586]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:02:02.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:01 smithi029 ceph-mon[26769]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:02:03.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:02 smithi029 ceph-mon[26769]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:02:03.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:02 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/3774530747' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7a75f26-ffa7-447e-8205-478e69d303b6"}]: dispatch 2024-04-27T02:02:03.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:02 smithi029 ceph-mon[26769]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7a75f26-ffa7-447e-8205-478e69d303b6"}]: dispatch 2024-04-27T02:02:03.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:02 smithi029 ceph-mon[26769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7a75f26-ffa7-447e-8205-478e69d303b6"}]': finished 2024-04-27T02:02:03.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:02 smithi029 ceph-mon[26769]: osdmap e31: 5 total, 4 up, 5 in 2024-04-27T02:02:03.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:02 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:02:03.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:02 smithi052 ceph-mon[32586]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:02:03.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:02 smithi052 ceph-mon[32586]: from='client.? 172.21.15.52:0/3774530747' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7a75f26-ffa7-447e-8205-478e69d303b6"}]: dispatch 2024-04-27T02:02:03.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:02 smithi052 ceph-mon[32586]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7a75f26-ffa7-447e-8205-478e69d303b6"}]: dispatch 2024-04-27T02:02:03.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:02 smithi052 ceph-mon[32586]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7a75f26-ffa7-447e-8205-478e69d303b6"}]': finished 2024-04-27T02:02:03.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:02 smithi052 ceph-mon[32586]: osdmap e31: 5 total, 4 up, 5 in 2024-04-27T02:02:03.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:02 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:02:04.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:03 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/1540099393' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:02:04.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:03 smithi052 ceph-mon[32586]: from='client.? 172.21.15.52:0/1540099393' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:02:05.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:04 smithi029 ceph-mon[26769]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-27T02:02:05.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:04 smithi052 ceph-mon[32586]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-27T02:02:06.928 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:06 smithi052 ceph-mon[32586]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-27T02:02:07.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:06 smithi029 ceph-mon[26769]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-27T02:02:08.127 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:07 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:02:08.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:07 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:02:09.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:08 smithi052 ceph-mon[32586]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-27T02:02:09.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:08 smithi029 ceph-mon[26769]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-27T02:02:10.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:09 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-27T02:02:10.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:09 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:02:10.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:09 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-27T02:02:10.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:09 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:02:11.145 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:10 smithi052 ceph-mon[32586]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-27T02:02:11.145 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:10 smithi052 ceph-mon[32586]: Deploying daemon osd.4 on smithi052 2024-04-27T02:02:11.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:10 smithi029 ceph-mon[26769]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-27T02:02:11.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:10 smithi029 ceph-mon[26769]: Deploying daemon osd.4 on smithi052 2024-04-27T02:02:12.865 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:12 smithi052 ceph-mon[32586]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-27T02:02:13.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:12 smithi029 ceph-mon[26769]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-27T02:02:15.125 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:14 smithi052 ceph-mon[32586]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:02:15.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:14 smithi029 ceph-mon[26769]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:02:16.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:16 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:16.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:16 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:16.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:16 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:02:16.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:16 smithi029 ceph-mon[26769]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:02:16.481 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:16 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:16.481 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:16 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:16.482 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:16 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:02:16.482 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:16 smithi052 ceph-mon[32586]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:02:19.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:18 smithi029 ceph-mon[26769]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:02:19.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:18 smithi052 ceph-mon[32586]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:02:19.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:19.413+0000 7fb44f7fe700 1 -- 172.21.15.52:0/169933226 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb458061190 con 0x7fb44406c680 2024-04-27T02:02:19.394 INFO:teuthology.orchestra.run.smithi052.stdout:Created osd(s) 4 on host 'smithi052' 2024-04-27T02:02:19.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:19.415+0000 7fb45ced3700 1 -- 172.21.15.52:0/169933226 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb44406c680 msgr2=0x7fb44406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:02:19.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:19.415+0000 7fb45ced3700 1 --2- 172.21.15.52:0/169933226 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb44406c680 0x7fb44406eb30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb448006010 tx=0x7fb44800b540 comp rx=0 tx=0).stop 2024-04-27T02:02:19.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:19.415+0000 7fb45ced3700 1 -- 172.21.15.52:0/169933226 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4580ff140 msgr2=0x7fb458195a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:02:19.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:19.415+0000 7fb45ced3700 1 --2- 172.21.15.52:0/169933226 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4580ff140 0x7fb458195a60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb44000d900 tx=0x7fb44000dc10 comp rx=0 tx=0).stop 2024-04-27T02:02:19.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:19.415+0000 7fb45ced3700 1 -- 172.21.15.52:0/169933226 shutdown_connections 2024-04-27T02:02:19.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:19.415+0000 7fb45ced3700 1 --2- 172.21.15.52:0/169933226 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb44406c680 0x7fb44406eb30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:19.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:19.415+0000 7fb45ced3700 1 --2- 172.21.15.52:0/169933226 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4580ff140 0x7fb458195a60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:19.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:19.415+0000 7fb45ced3700 1 --2- 172.21.15.52:0/169933226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb458100340 0x7fb458195fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:19.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:19.415+0000 7fb45ced3700 1 -- 172.21.15.52:0/169933226 >> 172.21.15.52:0/169933226 conn(0x7fb4580fa6d0 msgr2=0x7fb4580fcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:02:19.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:19.416+0000 7fb45ced3700 1 -- 172.21.15.52:0/169933226 shutdown_connections 2024-04-27T02:02:19.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:19.416+0000 7fb45ced3700 1 -- 172.21.15.52:0/169933226 wait complete. 2024-04-27T02:02:20.091 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:19 smithi052 ceph-mon[32586]: from='osd.4 [v2:172.21.15.52:6808/2191623063,v1:172.21.15.52:6809/2191623063]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T02:02:20.091 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:19 smithi052 ceph-mon[32586]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T02:02:20.091 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:19 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:20.091 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:19 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:20.091 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:19 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:20.091 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:19 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:20.112 DEBUG:teuthology.orchestra.run.smithi052:osd.4> sudo journalctl -f -n 0 -u ceph-4c587076-0439-11ef-bc93-c7b262605968@osd.4.service 2024-04-27T02:02:20.114 INFO:tasks.cephadm:Deploying osd.5 on smithi052 with /dev/vg_nvme/lv_2... 2024-04-27T02:02:20.115 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-27T02:02:20.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:19 smithi029 ceph-mon[26769]: from='osd.4 [v2:172.21.15.52:6808/2191623063,v1:172.21.15.52:6809/2191623063]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T02:02:20.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:19 smithi029 ceph-mon[26769]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T02:02:20.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:19 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:20.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:19 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:20.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:19 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:20.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:19 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:21.186 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:20 smithi052 ceph-mon[32586]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:02:21.186 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:20 smithi052 ceph-mon[32586]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-27T02:02:21.187 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:20 smithi052 ceph-mon[32586]: osdmap e32: 5 total, 4 up, 5 in 2024-04-27T02:02:21.187 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:20 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:02:21.187 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:20 smithi052 ceph-mon[32586]: from='osd.4 [v2:172.21.15.52:6808/2191623063,v1:172.21.15.52:6809/2191623063]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-27T02:02:21.187 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:20 smithi052 ceph-mon[32586]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-27T02:02:21.187 INFO:journalctl@ceph.osd.4.smithi052.stdout:Apr 27 02:02:20 smithi052 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-4[43353]: 2024-04-27T02:02:20.874+0000 7f885f929700 -1 osd.4 0 waiting for initial osdmap 2024-04-27T02:02:21.187 INFO:journalctl@ceph.osd.4.smithi052.stdout:Apr 27 02:02:20 smithi052 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-4[43353]: 2024-04-27T02:02:20.882+0000 7f8859f1b700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T02:02:21.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:20 smithi029 ceph-mon[26769]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:02:21.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:20 smithi029 ceph-mon[26769]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-27T02:02:21.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:20 smithi029 ceph-mon[26769]: osdmap e32: 5 total, 4 up, 5 in 2024-04-27T02:02:21.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:20 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:02:21.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:20 smithi029 ceph-mon[26769]: from='osd.4 [v2:172.21.15.52:6808/2191623063,v1:172.21.15.52:6809/2191623063]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-27T02:02:21.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:20 smithi029 ceph-mon[26769]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-27T02:02:21.724 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi052/config 2024-04-27T02:02:22.120 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:21 smithi052 ceph-mon[32586]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-04-27T02:02:22.120 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:21 smithi052 ceph-mon[32586]: osdmap e33: 5 total, 4 up, 5 in 2024-04-27T02:02:22.120 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:21 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:02:22.120 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:21 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:02:22.121 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:21 smithi052 ceph-mon[32586]: osd.4 [v2:172.21.15.52:6808/2191623063,v1:172.21.15.52:6809/2191623063] boot 2024-04-27T02:02:22.121 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:21 smithi052 ceph-mon[32586]: osdmap e34: 5 total, 5 up, 5 in 2024-04-27T02:02:22.121 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:21 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:02:22.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:21 smithi029 ceph-mon[26769]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-04-27T02:02:22.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:21 smithi029 ceph-mon[26769]: osdmap e33: 5 total, 4 up, 5 in 2024-04-27T02:02:22.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:21 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:02:22.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:21 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:02:22.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:21 smithi029 ceph-mon[26769]: osd.4 [v2:172.21.15.52:6808/2191623063,v1:172.21.15.52:6809/2191623063] boot 2024-04-27T02:02:22.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:21 smithi029 ceph-mon[26769]: osdmap e34: 5 total, 5 up, 5 in 2024-04-27T02:02:22.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:21 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:02:23.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:22 smithi029 ceph-mon[26769]: purged_snaps scrub starts 2024-04-27T02:02:23.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:22 smithi029 ceph-mon[26769]: purged_snaps scrub ok 2024-04-27T02:02:23.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:22 smithi029 ceph-mon[26769]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:02:23.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:22 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:02:23.208 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:22 smithi052 ceph-mon[32586]: purged_snaps scrub starts 2024-04-27T02:02:23.208 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:22 smithi052 ceph-mon[32586]: purged_snaps scrub ok 2024-04-27T02:02:23.208 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:22 smithi052 ceph-mon[32586]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:02:23.208 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:22 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:02:23.936 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-27T02:02:23.936 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-27T02:02:23.936 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10+0 records in 2024-04-27T02:02:23.936 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records out 2024-04-27T02:02:23.936 INFO:teuthology.orchestra.run.smithi052.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00869136 s, 1.2 GB/s 2024-04-27T02:02:23.936 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping successful for: 2024-04-27T02:02:24.064 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:23 smithi052 ceph-mon[32586]: osdmap e35: 5 total, 5 up, 5 in 2024-04-27T02:02:24.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:23 smithi029 ceph-mon[26769]: osdmap e35: 5 total, 5 up, 5 in 2024-04-27T02:02:24.657 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi052:vg_nvme/lv_2 2024-04-27T02:02:24.864 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi052/config 2024-04-27T02:02:24.986 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:24 smithi052 ceph-mon[32586]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:24.986 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:24 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:24.986 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:24 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:24.986 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:24 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:02:24.986 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:24 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:02:24.986 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:24 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:24.986 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:24 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:02:24.986 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:24 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:02:24.986 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:24 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:24.986 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:24 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:02:24.986 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:24 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:02:24.986 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:24 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:02:24.987 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:24 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:24 smithi029 ceph-mon[26769]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:24 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:24 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:24 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:02:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:24 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:02:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:24 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:24 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:02:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:24 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:02:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:24 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:24 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:02:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:24 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:02:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:24 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:02:25.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:24 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:26.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:25 smithi052 ceph-mon[32586]: Detected new or changed devices on smithi052 2024-04-27T02:02:26.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:25 smithi052 ceph-mon[32586]: Adjusting osd_memory_target on smithi052 to 7458M 2024-04-27T02:02:26.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:25 smithi029 ceph-mon[26769]: Detected new or changed devices on smithi052 2024-04-27T02:02:26.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:25 smithi029 ceph-mon[26769]: Adjusting osd_memory_target on smithi052 to 7458M 2024-04-27T02:02:26.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.614+0000 7f2b2ad54700 1 -- 172.21.15.52:0/3865467446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2b24101440 msgr2=0x7f2b24101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:02:26.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.614+0000 7f2b2ad54700 1 --2- 172.21.15.52:0/3865467446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2b24101440 0x7f2b24101890 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2b10009910 tx=0x7f2b10009c20 comp rx=0 tx=0).stop 2024-04-27T02:02:26.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.614+0000 7f2b2ad54700 1 -- 172.21.15.52:0/3865467446 shutdown_connections 2024-04-27T02:02:26.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.614+0000 7f2b2ad54700 1 --2- 172.21.15.52:0/3865467446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2b24101440 0x7f2b24101890 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:26.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.614+0000 7f2b2ad54700 1 --2- 172.21.15.52:0/3865467446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b24100240 0x7f2b24100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:26.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.614+0000 7f2b2ad54700 1 -- 172.21.15.52:0/3865467446 >> 172.21.15.52:0/3865467446 conn(0x7f2b240fb7f0 msgr2=0x7f2b240fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:02:26.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.614+0000 7f2b2ad54700 1 -- 172.21.15.52:0/3865467446 shutdown_connections 2024-04-27T02:02:26.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.614+0000 7f2b2ad54700 1 -- 172.21.15.52:0/3865467446 wait complete. 2024-04-27T02:02:26.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.615+0000 7f2b2ad54700 1 Processor -- start 2024-04-27T02:02:26.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.615+0000 7f2b2ad54700 1 -- start start 2024-04-27T02:02:26.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.615+0000 7f2b2ad54700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2b24100240 0x7f2b24195b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:02:26.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.615+0000 7f2b2ad54700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b24101440 0x7f2b241960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:02:26.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.615+0000 7f2b2ad54700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b241966e0 con 0x7f2b24101440 2024-04-27T02:02:26.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.615+0000 7f2b2ad54700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b24196820 con 0x7f2b24100240 2024-04-27T02:02:26.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.615+0000 7f2b28af0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2b24100240 0x7f2b24195b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:02:26.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.615+0000 7f2b28af0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2b24100240 0x7f2b24195b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48672/0 (socket says 172.21.15.52:48672) 2024-04-27T02:02:26.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.615+0000 7f2b28af0700 1 -- 172.21.15.52:0/1364000816 learned_addr learned my addr 172.21.15.52:0/1364000816 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T02:02:26.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.615+0000 7f2b28af0700 1 -- 172.21.15.52:0/1364000816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b24101440 msgr2=0x7f2b241960c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:02:26.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.615+0000 7f2b28af0700 1 --2- 172.21.15.52:0/1364000816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b24101440 0x7f2b241960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:26.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.616+0000 7f2b28af0700 1 -- 172.21.15.52:0/1364000816 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b100095f0 con 0x7f2b24100240 2024-04-27T02:02:26.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.616+0000 7f2b28af0700 1 --2- 172.21.15.52:0/1364000816 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2b24100240 0x7f2b24195b80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2b1800b480 tx=0x7f2b1800b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:02:26.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.616+0000 7f2b21ffb700 1 -- 172.21.15.52:0/1364000816 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b180118c0 con 0x7f2b24100240 2024-04-27T02:02:26.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.616+0000 7f2b2ad54700 1 -- 172.21.15.52:0/1364000816 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b2419b2d0 con 0x7f2b24100240 2024-04-27T02:02:26.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.616+0000 7f2b21ffb700 1 -- 172.21.15.52:0/1364000816 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2b18010380 con 0x7f2b24100240 2024-04-27T02:02:26.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.616+0000 7f2b21ffb700 1 -- 172.21.15.52:0/1364000816 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b1800f4f0 con 0x7f2b24100240 2024-04-27T02:02:26.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.616+0000 7f2b2ad54700 1 -- 172.21.15.52:0/1364000816 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b2419b820 con 0x7f2b24100240 2024-04-27T02:02:26.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.617+0000 7f2b2ad54700 1 -- 172.21.15.52:0/1364000816 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b24108e30 con 0x7f2b24100240 2024-04-27T02:02:26.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.617+0000 7f2b21ffb700 1 -- 172.21.15.52:0/1364000816 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f2b180104f0 con 0x7f2b24100240 2024-04-27T02:02:26.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.618+0000 7f2b21ffb700 1 --2- 172.21.15.52:0/1364000816 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f2b1406c680 0x7f2b1406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:02:26.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.618+0000 7f2b21ffb700 1 -- 172.21.15.52:0/1364000816 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4206+0+0 (secure 0 0 0) 0x7f2b1808afe0 con 0x7f2b24100240 2024-04-27T02:02:26.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.618+0000 7f2b23fff700 1 --2- 172.21.15.52:0/1364000816 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f2b1406c680 0x7f2b1406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:02:26.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.619+0000 7f2b23fff700 1 --2- 172.21.15.52:0/1364000816 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f2b1406c680 0x7f2b1406eb30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000b5c0 tx=0x7f2b10005960 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:02:26.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.623+0000 7f2b21ffb700 1 -- 172.21.15.52:0/1364000816 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2b18059640 con 0x7f2b24100240 2024-04-27T02:02:26.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:26.831+0000 7f2b2ad54700 1 -- 172.21.15.52:0/1364000816 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b24105d90 con 0x7f2b1406c680 2024-04-27T02:02:27.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:26 smithi029 ceph-mon[26769]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:27.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:26 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:02:27.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:26 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:02:27.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:26 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:02:27.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:26 smithi052 ceph-mon[32586]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:27.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:26 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:02:27.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:26 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:02:27.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:26 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:02:27.984 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:27 smithi052 ceph-mon[32586]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:02:28.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:27 smithi029 ceph-mon[26769]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:02:29.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:28 smithi029 ceph-mon[26769]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:29.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:28 smithi052 ceph-mon[32586]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:30.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:29 smithi052 ceph-mon[32586]: from='client.? 172.21.15.52:0/255704307' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cad5f7f-079e-476a-aab5-d5328eda9c8e"}]: dispatch 2024-04-27T02:02:30.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:29 smithi052 ceph-mon[32586]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cad5f7f-079e-476a-aab5-d5328eda9c8e"}]: dispatch 2024-04-27T02:02:30.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:29 smithi052 ceph-mon[32586]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5cad5f7f-079e-476a-aab5-d5328eda9c8e"}]': finished 2024-04-27T02:02:30.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:29 smithi052 ceph-mon[32586]: osdmap e36: 6 total, 5 up, 6 in 2024-04-27T02:02:30.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:29 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:02:30.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:29 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/255704307' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cad5f7f-079e-476a-aab5-d5328eda9c8e"}]: dispatch 2024-04-27T02:02:30.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:29 smithi029 ceph-mon[26769]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cad5f7f-079e-476a-aab5-d5328eda9c8e"}]: dispatch 2024-04-27T02:02:30.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:29 smithi029 ceph-mon[26769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5cad5f7f-079e-476a-aab5-d5328eda9c8e"}]': finished 2024-04-27T02:02:30.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:29 smithi029 ceph-mon[26769]: osdmap e36: 6 total, 5 up, 6 in 2024-04-27T02:02:30.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:29 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:02:31.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:30 smithi029 ceph-mon[26769]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:31.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:30 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/90830874' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:02:31.241 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:30 smithi052 ceph-mon[32586]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:31.241 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:30 smithi052 ceph-mon[32586]: from='client.? 172.21.15.52:0/90830874' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:02:33.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:32 smithi029 ceph-mon[26769]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:33.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:32 smithi052 ceph-mon[32586]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:35.182 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:34 smithi052 ceph-mon[32586]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:35.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:34 smithi029 ceph-mon[26769]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:37.156 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:36 smithi052 ceph-mon[32586]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:37.156 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:36 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-27T02:02:37.157 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:36 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:02:37.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:36 smithi029 ceph-mon[26769]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:37.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:36 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-27T02:02:37.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:36 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:02:38.112 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:37 smithi052 ceph-mon[32586]: Deploying daemon osd.5 on smithi052 2024-04-27T02:02:38.112 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:37 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:02:38.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:37 smithi029 ceph-mon[26769]: Deploying daemon osd.5 on smithi052 2024-04-27T02:02:38.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:37 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:02:39.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:38 smithi052 ceph-mon[32586]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:39.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:38 smithi029 ceph-mon[26769]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:41.162 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:40 smithi052 ceph-mon[32586]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:41.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:40 smithi029 ceph-mon[26769]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:43.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:42 smithi029 ceph-mon[26769]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:43.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:42 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:43.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:42 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:43.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:42 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:02:43.227 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:42 smithi052 ceph-mon[32586]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:43.227 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:42 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:43.227 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:42 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:43.227 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:42 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:02:45.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:44 smithi052 ceph-mon[32586]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:45.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:44 smithi029 ceph-mon[26769]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:45.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:45.504+0000 7f2b21ffb700 1 -- 172.21.15.52:0/1364000816 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2b24105d90 con 0x7f2b1406c680 2024-04-27T02:02:45.485 INFO:teuthology.orchestra.run.smithi052.stdout:Created osd(s) 5 on host 'smithi052' 2024-04-27T02:02:45.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:45.507+0000 7f2b2ad54700 1 -- 172.21.15.52:0/1364000816 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f2b1406c680 msgr2=0x7f2b1406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:02:45.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:45.507+0000 7f2b2ad54700 1 --2- 172.21.15.52:0/1364000816 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f2b1406c680 0x7f2b1406eb30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000b5c0 tx=0x7f2b10005960 comp rx=0 tx=0).stop 2024-04-27T02:02:45.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:45.507+0000 7f2b2ad54700 1 -- 172.21.15.52:0/1364000816 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2b24100240 msgr2=0x7f2b24195b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:02:45.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:45.507+0000 7f2b2ad54700 1 --2- 172.21.15.52:0/1364000816 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2b24100240 0x7f2b24195b80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2b1800b480 tx=0x7f2b1800b840 comp rx=0 tx=0).stop 2024-04-27T02:02:45.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:45.507+0000 7f2b2ad54700 1 -- 172.21.15.52:0/1364000816 shutdown_connections 2024-04-27T02:02:45.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:45.507+0000 7f2b2ad54700 1 --2- 172.21.15.52:0/1364000816 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f2b1406c680 0x7f2b1406eb30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:45.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:45.507+0000 7f2b2ad54700 1 --2- 172.21.15.52:0/1364000816 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2b24100240 0x7f2b24195b80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:45.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:45.507+0000 7f2b2ad54700 1 --2- 172.21.15.52:0/1364000816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b24101440 0x7f2b241960c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:45.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:45.507+0000 7f2b2ad54700 1 -- 172.21.15.52:0/1364000816 >> 172.21.15.52:0/1364000816 conn(0x7f2b240fb7f0 msgr2=0x7f2b24104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:02:45.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:45.507+0000 7f2b2ad54700 1 -- 172.21.15.52:0/1364000816 shutdown_connections 2024-04-27T02:02:45.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:02:45.507+0000 7f2b2ad54700 1 -- 172.21.15.52:0/1364000816 wait complete. 2024-04-27T02:02:46.160 DEBUG:teuthology.orchestra.run.smithi052:osd.5> sudo journalctl -f -n 0 -u ceph-4c587076-0439-11ef-bc93-c7b262605968@osd.5.service 2024-04-27T02:02:46.162 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-04-27T02:02:46.163 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-27T02:02:46.390 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:02:46.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:46 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:46.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:46 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:46.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:46 smithi052 ceph-mon[32586]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:46.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:46 smithi052 ceph-mon[32586]: from='osd.5 [v2:172.21.15.52:6816/2667246548,v1:172.21.15.52:6817/2667246548]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T02:02:46.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:46 smithi052 ceph-mon[32586]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T02:02:46.758 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:46 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:46.758 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:46 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:46.758 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:46 smithi029 ceph-mon[26769]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:46.758 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:46 smithi029 ceph-mon[26769]: from='osd.5 [v2:172.21.15.52:6816/2667246548,v1:172.21.15.52:6817/2667246548]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T02:02:46.758 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:46 smithi029 ceph-mon[26769]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T02:02:48.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:47 smithi029 ceph-mon[26769]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-27T02:02:48.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:47 smithi029 ceph-mon[26769]: osdmap e37: 6 total, 5 up, 6 in 2024-04-27T02:02:48.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:47 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:02:48.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:47 smithi029 ceph-mon[26769]: from='osd.5 [v2:172.21.15.52:6816/2667246548,v1:172.21.15.52:6817/2667246548]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-27T02:02:48.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:47 smithi029 ceph-mon[26769]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-27T02:02:48.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:47 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:48.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:47 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:48.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.153+0000 7f8b6d56f700 1 -- 172.21.15.29:0/1744746383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b681003d0 msgr2=0x7f8b68100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:02:48.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.153+0000 7f8b6d56f700 1 --2- 172.21.15.29:0/1744746383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b681003d0 0x7f8b68100840 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8b50009910 tx=0x7f8b50009c20 comp rx=0 tx=0).stop 2024-04-27T02:02:48.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.153+0000 7f8b6d56f700 1 -- 172.21.15.29:0/1744746383 shutdown_connections 2024-04-27T02:02:48.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.153+0000 7f8b6d56f700 1 --2- 172.21.15.29:0/1744746383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b681003d0 0x7f8b68100840 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:48.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.153+0000 7f8b6d56f700 1 --2- 172.21.15.29:0/1744746383 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8b680ff130 0x7f8b680ff540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:48.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.153+0000 7f8b6d56f700 1 -- 172.21.15.29:0/1744746383 >> 172.21.15.29:0/1744746383 conn(0x7f8b680fa780 msgr2=0x7f8b680fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:02:48.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.153+0000 7f8b6d56f700 1 -- 172.21.15.29:0/1744746383 shutdown_connections 2024-04-27T02:02:48.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.153+0000 7f8b6d56f700 1 -- 172.21.15.29:0/1744746383 wait complete. 2024-04-27T02:02:48.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.153+0000 7f8b6d56f700 1 Processor -- start 2024-04-27T02:02:48.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.154+0000 7f8b6d56f700 1 -- start start 2024-04-27T02:02:48.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.154+0000 7f8b6d56f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b680ff130 0x7f8b68069100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:02:48.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.154+0000 7f8b6d56f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8b681003d0 0x7f8b6819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:02:48.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.154+0000 7f8b6d56f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6819ad80 con 0x7f8b680ff130 2024-04-27T02:02:48.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.154+0000 7f8b6d56f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6819aec0 con 0x7f8b681003d0 2024-04-27T02:02:48.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.154+0000 7f8b66ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b680ff130 0x7f8b68069100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:02:48.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.154+0000 7f8b66ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b680ff130 0x7f8b68069100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49994/0 (socket says 172.21.15.29:49994) 2024-04-27T02:02:48.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.154+0000 7f8b667fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8b681003d0 0x7f8b6819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:02:48.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.154+0000 7f8b66ffd700 1 -- 172.21.15.29:0/556587187 learned_addr learned my addr 172.21.15.29:0/556587187 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:02:48.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.155+0000 7f8b66ffd700 1 -- 172.21.15.29:0/556587187 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8b681003d0 msgr2=0x7f8b6819a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:02:48.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.155+0000 7f8b66ffd700 1 --2- 172.21.15.29:0/556587187 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8b681003d0 0x7f8b6819a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:48.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.155+0000 7f8b66ffd700 1 -- 172.21.15.29:0/556587187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b500095f0 con 0x7f8b680ff130 2024-04-27T02:02:48.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.155+0000 7f8b667fc700 1 --2- 172.21.15.29:0/556587187 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8b681003d0 0x7f8b6819a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T02:02:48.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.155+0000 7f8b66ffd700 1 --2- 172.21.15.29:0/556587187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b680ff130 0x7f8b68069100 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8b5800d8d0 tx=0x7f8b5800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:02:48.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.155+0000 7f8b5ffff700 1 -- 172.21.15.29:0/556587187 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b580098a0 con 0x7f8b680ff130 2024-04-27T02:02:48.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.155+0000 7f8b5ffff700 1 -- 172.21.15.29:0/556587187 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8b58010460 con 0x7f8b680ff130 2024-04-27T02:02:48.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.155+0000 7f8b5ffff700 1 -- 172.21.15.29:0/556587187 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b5800f5f0 con 0x7f8b680ff130 2024-04-27T02:02:48.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.155+0000 7f8b6d56f700 1 -- 172.21.15.29:0/556587187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b6819b1a0 con 0x7f8b680ff130 2024-04-27T02:02:48.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.155+0000 7f8b6d56f700 1 -- 172.21.15.29:0/556587187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b6819b6f0 con 0x7f8b680ff130 2024-04-27T02:02:48.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.156+0000 7f8b6d56f700 1 -- 172.21.15.29:0/556587187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b68107c10 con 0x7f8b680ff130 2024-04-27T02:02:48.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.159+0000 7f8b5ffff700 1 -- 172.21.15.29:0/556587187 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f8b58009a00 con 0x7f8b680ff130 2024-04-27T02:02:48.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.160+0000 7f8b5ffff700 1 --2- 172.21.15.29:0/556587187 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f8b5406c680 0x7f8b5406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:02:48.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.160+0000 7f8b5ffff700 1 -- 172.21.15.29:0/556587187 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4358+0+0 (secure 0 0 0) 0x7f8b5808b300 con 0x7f8b680ff130 2024-04-27T02:02:48.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.160+0000 7f8b667fc700 1 --2- 172.21.15.29:0/556587187 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f8b5406c680 0x7f8b5406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:02:48.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.160+0000 7f8b667fc700 1 --2- 172.21.15.29:0/556587187 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f8b5406c680 0x7f8b5406eb30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8b50006010 tx=0x7f8b5000b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:02:48.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.164+0000 7f8b5ffff700 1 -- 172.21.15.29:0/556587187 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8b580599f0 con 0x7f8b680ff130 2024-04-27T02:02:48.177 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:47 smithi052 ceph-mon[32586]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-27T02:02:48.177 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:47 smithi052 ceph-mon[32586]: osdmap e37: 6 total, 5 up, 6 in 2024-04-27T02:02:48.177 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:47 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:02:48.177 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:47 smithi052 ceph-mon[32586]: from='osd.5 [v2:172.21.15.52:6816/2667246548,v1:172.21.15.52:6817/2667246548]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-27T02:02:48.177 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:47 smithi052 ceph-mon[32586]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-27T02:02:48.177 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:47 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:48.178 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:47 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:48.178 INFO:journalctl@ceph.osd.5.smithi052.stdout:Apr 27 02:02:47 smithi052 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-5[48061]: 2024-04-27T02:02:47.846+0000 7f3974c28700 -1 osd.5 0 waiting for initial osdmap 2024-04-27T02:02:48.178 INFO:journalctl@ceph.osd.5.smithi052.stdout:Apr 27 02:02:47 smithi052 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-5[48061]: 2024-04-27T02:02:47.855+0000 7f396da17700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T02:02:48.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.365+0000 7f8b6d56f700 1 -- 172.21.15.29:0/556587187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f8b68107e00 con 0x7f8b680ff130 2024-04-27T02:02:48.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.366+0000 7f8b5ffff700 1 -- 172.21.15.29:0/556587187 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v38) v1 ==== 74+0+130 (secure 0 0 0) 0x7f8b580200a0 con 0x7f8b680ff130 2024-04-27T02:02:48.347 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:02:48.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.368+0000 7f8b6d56f700 1 -- 172.21.15.29:0/556587187 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f8b5406c680 msgr2=0x7f8b5406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:02:48.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.368+0000 7f8b6d56f700 1 --2- 172.21.15.29:0/556587187 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f8b5406c680 0x7f8b5406eb30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8b50006010 tx=0x7f8b5000b540 comp rx=0 tx=0).stop 2024-04-27T02:02:48.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.368+0000 7f8b6d56f700 1 -- 172.21.15.29:0/556587187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b680ff130 msgr2=0x7f8b68069100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:02:48.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.368+0000 7f8b6d56f700 1 --2- 172.21.15.29:0/556587187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b680ff130 0x7f8b68069100 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8b5800d8d0 tx=0x7f8b5800dbe0 comp rx=0 tx=0).stop 2024-04-27T02:02:48.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.368+0000 7f8b6d56f700 1 -- 172.21.15.29:0/556587187 shutdown_connections 2024-04-27T02:02:48.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.368+0000 7f8b6d56f700 1 --2- 172.21.15.29:0/556587187 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f8b5406c680 0x7f8b5406eb30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:48.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.368+0000 7f8b6d56f700 1 --2- 172.21.15.29:0/556587187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b680ff130 0x7f8b68069100 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:48.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.368+0000 7f8b6d56f700 1 --2- 172.21.15.29:0/556587187 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8b681003d0 0x7f8b6819a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:48.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.368+0000 7f8b6d56f700 1 -- 172.21.15.29:0/556587187 >> 172.21.15.29:0/556587187 conn(0x7f8b680fa780 msgr2=0x7f8b68103600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:02:48.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.368+0000 7f8b6d56f700 1 -- 172.21.15.29:0/556587187 shutdown_connections 2024-04-27T02:02:48.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:48.368+0000 7f8b6d56f700 1 -- 172.21.15.29:0/556587187 wait complete. 2024-04-27T02:02:49.134 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:48 smithi052 ceph-mon[32586]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:49.134 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:48 smithi052 ceph-mon[32586]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-04-27T02:02:49.134 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:48 smithi052 ceph-mon[32586]: osdmap e38: 6 total, 5 up, 6 in 2024-04-27T02:02:49.134 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:48 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:02:49.134 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:48 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:02:49.134 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:48 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/556587187' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-27T02:02:49.136 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:48 smithi029 ceph-mon[26769]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:49.136 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:48 smithi029 ceph-mon[26769]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-04-27T02:02:49.136 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:48 smithi029 ceph-mon[26769]: osdmap e38: 6 total, 5 up, 6 in 2024-04-27T02:02:49.136 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:48 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:02:49.136 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:48 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:02:49.136 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:48 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/556587187' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-27T02:02:49.159 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":38,"num_osds":6,"num_up_osds":5,"osd_up_since":1714183341,"num_in_osds":6,"osd_in_since":1714183349,"num_remapped_pgs":0} 2024-04-27T02:02:50.156 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:49 smithi052 ceph-mon[32586]: purged_snaps scrub starts 2024-04-27T02:02:50.156 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:49 smithi052 ceph-mon[32586]: purged_snaps scrub ok 2024-04-27T02:02:50.156 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:49 smithi052 ceph-mon[32586]: osd.5 [v2:172.21.15.52:6816/2667246548,v1:172.21.15.52:6817/2667246548] boot 2024-04-27T02:02:50.156 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:49 smithi052 ceph-mon[32586]: osdmap e39: 6 total, 6 up, 6 in 2024-04-27T02:02:50.156 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:49 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:02:50.162 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-27T02:02:50.190 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:49 smithi029 ceph-mon[26769]: purged_snaps scrub starts 2024-04-27T02:02:50.190 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:49 smithi029 ceph-mon[26769]: purged_snaps scrub ok 2024-04-27T02:02:50.191 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:49 smithi029 ceph-mon[26769]: osd.5 [v2:172.21.15.52:6816/2667246548,v1:172.21.15.52:6817/2667246548] boot 2024-04-27T02:02:50.191 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:49 smithi029 ceph-mon[26769]: osdmap e39: 6 total, 6 up, 6 in 2024-04-27T02:02:50.191 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:49 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:02:50.394 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:02:51.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:50 smithi029 ceph-mon[26769]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:51.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:50 smithi029 ceph-mon[26769]: osdmap e40: 6 total, 6 up, 6 in 2024-04-27T02:02:51.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:50 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:51.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:50 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:51.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:50 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:02:51.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:50 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:02:51.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:50 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:02:51.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:50 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:51.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:50 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:02:51.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:50 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:02:51.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:50 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:51.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:50 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:02:51.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:50 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:02:51.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:50 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:02:51.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:50 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:51.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:50 smithi052 ceph-mon[32586]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:02:51.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:50 smithi052 ceph-mon[32586]: osdmap e40: 6 total, 6 up, 6 in 2024-04-27T02:02:51.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:50 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:51.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:50 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:51.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:50 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:02:51.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:50 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:02:51.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:50 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:02:51.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:50 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:51.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:50 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:02:51.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:50 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:02:51.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:50 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:51.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:50 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:02:51.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:50 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:02:51.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:50 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:02:51.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:50 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:02:52.090 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:51 smithi029 ceph-mon[26769]: Detected new or changed devices on smithi052 2024-04-27T02:02:52.090 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:51 smithi029 ceph-mon[26769]: Adjusting osd_memory_target on smithi052 to 4972M 2024-04-27T02:02:52.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.214+0000 7f9349bdf700 1 -- 172.21.15.29:0/957973868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9344103370 msgr2=0x7f9344105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:02:52.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.214+0000 7f9349bdf700 1 --2- 172.21.15.29:0/957973868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9344103370 0x7f9344105790 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9338009910 tx=0x7f9338009c20 comp rx=0 tx=0).stop 2024-04-27T02:02:52.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.214+0000 7f9349bdf700 1 -- 172.21.15.29:0/957973868 shutdown_connections 2024-04-27T02:02:52.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.214+0000 7f9349bdf700 1 --2- 172.21.15.29:0/957973868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9344103370 0x7f9344105790 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:52.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.214+0000 7f9349bdf700 1 --2- 172.21.15.29:0/957973868 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9344069210 0x7f9344102e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:52.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.214+0000 7f9349bdf700 1 -- 172.21.15.29:0/957973868 >> 172.21.15.29:0/957973868 conn(0x7f93440fa780 msgr2=0x7f93440fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:02:52.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.214+0000 7f9349bdf700 1 -- 172.21.15.29:0/957973868 shutdown_connections 2024-04-27T02:02:52.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.214+0000 7f9349bdf700 1 -- 172.21.15.29:0/957973868 wait complete. 2024-04-27T02:02:52.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.215+0000 7f9349bdf700 1 Processor -- start 2024-04-27T02:02:52.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.215+0000 7f9349bdf700 1 -- start start 2024-04-27T02:02:52.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.215+0000 7f9349bdf700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9344069210 0x7f9344195a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:02:52.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.215+0000 7f9349bdf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9344103370 0x7f9344195fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:02:52.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.215+0000 7f9349bdf700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93441965d0 con 0x7f9344103370 2024-04-27T02:02:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.215+0000 7f9349bdf700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9344196710 con 0x7f9344069210 2024-04-27T02:02:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.215+0000 7f9342ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9344103370 0x7f9344195fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:02:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.215+0000 7f9342ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9344103370 0x7f9344195fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50004/0 (socket says 172.21.15.29:50004) 2024-04-27T02:02:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.215+0000 7f9342ffd700 1 -- 172.21.15.29:0/1922695757 learned_addr learned my addr 172.21.15.29:0/1922695757 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:02:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.215+0000 7f93437fe700 1 --2- 172.21.15.29:0/1922695757 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9344069210 0x7f9344195a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:02:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.216+0000 7f9342ffd700 1 -- 172.21.15.29:0/1922695757 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9344069210 msgr2=0x7f9344195a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:02:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.216+0000 7f9342ffd700 1 --2- 172.21.15.29:0/1922695757 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9344069210 0x7f9344195a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.216+0000 7f9342ffd700 1 -- 172.21.15.29:0/1922695757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93380095f0 con 0x7f9344103370 2024-04-27T02:02:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.216+0000 7f9342ffd700 1 --2- 172.21.15.29:0/1922695757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9344103370 0x7f9344195fb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9338000c00 tx=0x7f933800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:02:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.216+0000 7f9340ff9700 1 -- 172.21.15.29:0/1922695757 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f933801d070 con 0x7f9344103370 2024-04-27T02:02:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.216+0000 7f9340ff9700 1 -- 172.21.15.29:0/1922695757 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9338005dc0 con 0x7f9344103370 2024-04-27T02:02:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.216+0000 7f9349bdf700 1 -- 172.21.15.29:0/1922695757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f934419b160 con 0x7f9344103370 2024-04-27T02:02:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.216+0000 7f9340ff9700 1 -- 172.21.15.29:0/1922695757 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93380226a0 con 0x7f9344103370 2024-04-27T02:02:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.216+0000 7f9349bdf700 1 -- 172.21.15.29:0/1922695757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f934419b650 con 0x7f9344103370 2024-04-27T02:02:52.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.217+0000 7f9349bdf700 1 -- 172.21.15.29:0/1922695757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f934418fbd0 con 0x7f9344103370 2024-04-27T02:02:52.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.217+0000 7f9340ff9700 1 -- 172.21.15.29:0/1922695757 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f93380038f0 con 0x7f9344103370 2024-04-27T02:02:52.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.218+0000 7f9340ff9700 1 --2- 172.21.15.29:0/1922695757 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f932c06c680 0x7f932c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:02:52.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.218+0000 7f93437fe700 1 --2- 172.21.15.29:0/1922695757 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f932c06c680 0x7f932c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:02:52.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.218+0000 7f9340ff9700 1 -- 172.21.15.29:0/1922695757 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f933808c910 con 0x7f9344103370 2024-04-27T02:02:52.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.218+0000 7f93437fe700 1 --2- 172.21.15.29:0/1922695757 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f932c06c680 0x7f932c06eb30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9344068f40 tx=0x7f9334009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:02:52.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.223+0000 7f9340ff9700 1 -- 172.21.15.29:0/1922695757 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f933805ae40 con 0x7f9344103370 2024-04-27T02:02:52.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:51 smithi052 ceph-mon[32586]: Detected new or changed devices on smithi052 2024-04-27T02:02:52.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:51 smithi052 ceph-mon[32586]: Adjusting osd_memory_target on smithi052 to 4972M 2024-04-27T02:02:52.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.421+0000 7f9349bdf700 1 -- 172.21.15.29:0/1922695757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f934402cc30 con 0x7f9344103370 2024-04-27T02:02:52.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.422+0000 7f9340ff9700 1 -- 172.21.15.29:0/1922695757 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f9338003680 con 0x7f9344103370 2024-04-27T02:02:52.403 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:02:52.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.424+0000 7f9349bdf700 1 -- 172.21.15.29:0/1922695757 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f932c06c680 msgr2=0x7f932c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:02:52.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.424+0000 7f9349bdf700 1 --2- 172.21.15.29:0/1922695757 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f932c06c680 0x7f932c06eb30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9344068f40 tx=0x7f9334009260 comp rx=0 tx=0).stop 2024-04-27T02:02:52.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.424+0000 7f9349bdf700 1 -- 172.21.15.29:0/1922695757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9344103370 msgr2=0x7f9344195fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:02:52.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.424+0000 7f9349bdf700 1 --2- 172.21.15.29:0/1922695757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9344103370 0x7f9344195fb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9338000c00 tx=0x7f933800f690 comp rx=0 tx=0).stop 2024-04-27T02:02:52.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.425+0000 7f9349bdf700 1 -- 172.21.15.29:0/1922695757 shutdown_connections 2024-04-27T02:02:52.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.425+0000 7f9349bdf700 1 --2- 172.21.15.29:0/1922695757 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f932c06c680 0x7f932c06eb30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:52.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.425+0000 7f9349bdf700 1 --2- 172.21.15.29:0/1922695757 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9344069210 0x7f9344195a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:52.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.425+0000 7f9349bdf700 1 --2- 172.21.15.29:0/1922695757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9344103370 0x7f9344195fb0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:52.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.425+0000 7f9349bdf700 1 -- 172.21.15.29:0/1922695757 >> 172.21.15.29:0/1922695757 conn(0x7f93440fa780 msgr2=0x7f93440fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:02:52.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.425+0000 7f9349bdf700 1 -- 172.21.15.29:0/1922695757 shutdown_connections 2024-04-27T02:02:52.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:52.425+0000 7f9349bdf700 1 -- 172.21.15.29:0/1922695757 wait complete. 2024-04-27T02:02:53.171 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:52 smithi029 ceph-mon[26769]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:02:53.172 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:52 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/1922695757' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-27T02:02:53.172 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:52 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:02:53.200 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1714183368,"num_in_osds":6,"osd_in_since":1714183349,"num_remapped_pgs":0} 2024-04-27T02:02:53.200 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-27T02:02:53.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:52 smithi052 ceph-mon[32586]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:02:53.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:52 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/1922695757' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-27T02:02:53.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:52 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:02:53.464 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:02:55.103 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:54 smithi029 ceph-mon[26769]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:02:55.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.127+0000 7f11c02a3700 1 -- 172.21.15.29:0/880407364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11b80ff3b0 msgr2=0x7f11b80ff780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:02:55.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.127+0000 7f11c02a3700 1 --2- 172.21.15.29:0/880407364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11b80ff3b0 0x7f11b80ff780 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f11b4009910 tx=0x7f11b4009c20 comp rx=0 tx=0).stop 2024-04-27T02:02:55.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.128+0000 7f11c02a3700 1 -- 172.21.15.29:0/880407364 shutdown_connections 2024-04-27T02:02:55.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.128+0000 7f11c02a3700 1 --2- 172.21.15.29:0/880407364 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f11b80ffd50 0x7f11b8103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:55.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.128+0000 7f11c02a3700 1 --2- 172.21.15.29:0/880407364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11b80ff3b0 0x7f11b80ff780 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:55.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.128+0000 7f11c02a3700 1 -- 172.21.15.29:0/880407364 >> 172.21.15.29:0/880407364 conn(0x7f11b80fac90 msgr2=0x7f11b80fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:02:55.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.128+0000 7f11c02a3700 1 -- 172.21.15.29:0/880407364 shutdown_connections 2024-04-27T02:02:55.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.128+0000 7f11c02a3700 1 -- 172.21.15.29:0/880407364 wait complete. 2024-04-27T02:02:55.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.128+0000 7f11c02a3700 1 Processor -- start 2024-04-27T02:02:55.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.128+0000 7f11c02a3700 1 -- start start 2024-04-27T02:02:55.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.128+0000 7f11c02a3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f11b80ff3b0 0x7f11b8195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:02:55.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.128+0000 7f11c02a3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11b80ffd50 0x7f11b8196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:02:55.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.128+0000 7f11c02a3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b8196b50 con 0x7f11b80ffd50 2024-04-27T02:02:55.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.128+0000 7f11c02a3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b819a8e0 con 0x7f11b80ff3b0 2024-04-27T02:02:55.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.129+0000 7f11bd83e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11b80ffd50 0x7f11b8196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:02:55.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.129+0000 7f11bd83e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11b80ffd50 0x7f11b8196470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50024/0 (socket says 172.21.15.29:50024) 2024-04-27T02:02:55.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.129+0000 7f11bd83e700 1 -- 172.21.15.29:0/916737704 learned_addr learned my addr 172.21.15.29:0/916737704 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:02:55.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.129+0000 7f11be03f700 1 --2- 172.21.15.29:0/916737704 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f11b80ff3b0 0x7f11b8195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:02:55.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.129+0000 7f11bd83e700 1 -- 172.21.15.29:0/916737704 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f11b80ff3b0 msgr2=0x7f11b8195f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:02:55.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.129+0000 7f11bd83e700 1 --2- 172.21.15.29:0/916737704 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f11b80ff3b0 0x7f11b8195f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:55.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.129+0000 7f11bd83e700 1 -- 172.21.15.29:0/916737704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b40095f0 con 0x7f11b80ffd50 2024-04-27T02:02:55.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.129+0000 7f11bd83e700 1 --2- 172.21.15.29:0/916737704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11b80ffd50 0x7f11b8196470 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f11a800c8f0 tx=0x7f11a800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:02:55.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.129+0000 7f11af7fe700 1 -- 172.21.15.29:0/916737704 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11a8007a50 con 0x7f11b80ffd50 2024-04-27T02:02:55.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.129+0000 7f11af7fe700 1 -- 172.21.15.29:0/916737704 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f11a8007bb0 con 0x7f11b80ffd50 2024-04-27T02:02:55.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.129+0000 7f11c02a3700 1 -- 172.21.15.29:0/916737704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b819abc0 con 0x7f11b80ffd50 2024-04-27T02:02:55.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.129+0000 7f11af7fe700 1 -- 172.21.15.29:0/916737704 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11a8018630 con 0x7f11b80ffd50 2024-04-27T02:02:55.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.129+0000 7f11c02a3700 1 -- 172.21.15.29:0/916737704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b819b110 con 0x7f11b80ffd50 2024-04-27T02:02:55.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.130+0000 7f11c02a3700 1 -- 172.21.15.29:0/916737704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b81076c0 con 0x7f11b80ffd50 2024-04-27T02:02:55.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.131+0000 7f11af7fe700 1 -- 172.21.15.29:0/916737704 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f11a8007d20 con 0x7f11b80ffd50 2024-04-27T02:02:55.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.131+0000 7f11af7fe700 1 --2- 172.21.15.29:0/916737704 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f11a406c680 0x7f11a406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:02:55.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.131+0000 7f11af7fe700 1 -- 172.21.15.29:0/916737704 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f11a808b500 con 0x7f11b80ffd50 2024-04-27T02:02:55.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.131+0000 7f11be03f700 1 --2- 172.21.15.29:0/916737704 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f11a406c680 0x7f11a406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:02:55.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.132+0000 7f11be03f700 1 --2- 172.21.15.29:0/916737704 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f11a406c680 0x7f11a406eb30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f11b400b5c0 tx=0x7f11b401a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:02:55.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.136+0000 7f11af7fe700 1 -- 172.21.15.29:0/916737704 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f11a8055f20 con 0x7f11b80ffd50 2024-04-27T02:02:55.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:54 smithi052 ceph-mon[32586]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:02:55.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.334+0000 7f11c02a3700 1 -- 172.21.15.29:0/916737704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11b819b390 con 0x7f11b80ffd50 2024-04-27T02:02:55.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.335+0000 7f11af7fe700 1 -- 172.21.15.29:0/916737704 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11299 (secure 0 0 0) 0x7f11a8059540 con 0x7f11b80ffd50 2024-04-27T02:02:55.316 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:02:55.316 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":40,"fsid":"4c587076-0439-11ef-bc93-c7b262605968","created":"2024-04-27T01:56:52.758582+0000","modified":"2024-04-27T02:02:49.850602+0000","last_up_change":"2024-04-27T02:02:48.845017+0000","last_in_change":"2024-04-27T02:02:29.359236+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-27T02:01:30.306687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3686f584-4bb4-4d6a-8daa-93b9184e4bc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":1138569897},{"type":"v1","addr":"172.21.15.29:6803","nonce":1138569897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":1138569897},{"type":"v1","addr":"172.21.15.29:6805","nonce":1138569897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1138569897},{"type":"v1","addr":"172.21.15.29:6809","nonce":1138569897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":1138569897},{"type":"v1","addr":"172.21.15.29:6807","nonce":1138569897}]},"public_addr":"172.21.15.29:6803/1138569897","cluster_addr":"172.21.15.29:6805/1138569897","heartbeat_back_addr":"172.21.15.29:6809/1138569897","heartbeat_front_addr":"172.21.15.29:6807/1138569897","state":["exists","up"]},{"osd":1,"uuid":"ad9bc7ef-74a2-46c2-a201-a2820c9c8923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3705295680},{"type":"v1","addr":"172.21.15.29:6811","nonce":3705295680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3705295680},{"type":"v1","addr":"172.21.15.29:6813","nonce":3705295680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3705295680},{"type":"v1","addr":"172.21.15.29:6817","nonce":3705295680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3705295680},{"type":"v1","addr":"172.21.15.29:6815","nonce":3705295680}]},"public_addr":"172.21.15.29:6811/3705295680","cluster_addr":"172.21.15.29:6813/3705295680","heartbeat_back_addr":"172.21.15.29:6817/3705295680","heartbeat_front_addr":"172.21.15.29:6815/3705295680","state":["exists","up"]},{"osd":2,"uuid":"315cd205-cf98-4743-8b62-9a158d99a4e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3013669823},{"type":"v1","addr":"172.21.15.29:6819","nonce":3013669823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":3013669823},{"type":"v1","addr":"172.21.15.29:6821","nonce":3013669823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3013669823},{"type":"v1","addr":"172.21.15.29:6825","nonce":3013669823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":3013669823},{"type":"v1","addr":"172.21.15.29:6823","nonce":3013669823}]},"public_addr":"172.21.15.29:6819/3013669823","cluster_addr":"172.21.15.29:6821/3013669823","heartbeat_back_addr":"172.21.15.29:6825/3013669823","heartbeat_front_addr":"172.21.15.29:6823/3013669823","state":["exists","up"]},{"osd":3,"uuid":"59548c0e-dc14-4d71-940c-58802631725b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":263370941},{"type":"v1","addr":"172.21.15.52:6801","nonce":263370941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":263370941},{"type":"v1","addr":"172.21.15.52:6803","nonce":263370941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":263370941},{"type":"v1","addr":"172.21.15.52:6807","nonce":263370941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":263370941},{"type":"v1","addr":"172.21.15.52:6805","nonce":263370941}]},"public_addr":"172.21.15.52:6801/263370941","cluster_addr":"172.21.15.52:6803/263370941","heartbeat_back_addr":"172.21.15.52:6807/263370941","heartbeat_front_addr":"172.21.15.52:6805/263370941","state":["exists","up"]},{"osd":4,"uuid":"c7a75f26-ffa7-447e-8205-478e69d303b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":2191623063},{"type":"v1","addr":"172.21.15.52:6809","nonce":2191623063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":2191623063},{"type":"v1","addr":"172.21.15.52:6811","nonce":2191623063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":2191623063},{"type":"v1","addr":"172.21.15.52:6815","nonce":2191623063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":2191623063},{"type":"v1","addr":"172.21.15.52:6813","nonce":2191623063}]},"public_addr":"172.21.15.52:6809/2191623063","cluster_addr":"172.21.15.52:6811/2191623063","heartbeat_back_addr":"172.21.15.52:6815/2191623063","heartbeat_front_addr":"172.21.15.52:6813/2191623063","state":["exists","up"]},{"osd":5,"uuid":"5cad5f7f-079e-476a-aab5-d5328eda9c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":2667246548},{"type":"v1","addr":"172.21.15.52:6817","nonce":2667246548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":2667246548},{"type":"v1","addr":"172.21.15.52:6819","nonce":2667246548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":2667246548},{"type":"v1","addr":"172.21.15.52:6823","nonce":2667246548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":2667246548},{"type":"v1","addr":"172.21.15.52:6821","nonce":2667246548}]},"public_addr":"172.21.15.52:6817/2667246548","cluster_addr":"172.21.15.52:6819/2667246548","heartbeat_back_addr":"172.21.15.52:6823/2667246548","heartbeat_front_addr":"172.21.15.52:6821/2667246548","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:00:30.289475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:00:59.350372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:01:27.504097+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:01:54.076675+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:02:20.069253+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:02:46.979018+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6801/2469926221":"2024-04-28T01:59:07.772795+0000","172.21.15.29:6801/3873642219":"2024-04-28T01:57:57.788454+0000","172.21.15.29:6800/3873642219":"2024-04-28T01:57:57.788454+0000","172.21.15.29:0/202142181":"2024-04-28T01:57:57.788454+0000","172.21.15.29:0/3675815378":"2024-04-28T01:57:57.788454+0000","172.21.15.29:0/1899826075":"2024-04-28T01:59:07.772795+0000","172.21.15.29:0/297079675":"2024-04-28T01:59:07.772795+0000","172.21.15.29:6800/3122572856":"2024-04-28T01:57:23.638376+0000","172.21.15.29:6800/2469926221":"2024-04-28T01:59:07.772795+0000","172.21.15.29:0/3724837509":"2024-04-28T01:57:23.638376+0000","172.21.15.29:0/909831728":"2024-04-28T01:57:23.638376+0000","172.21.15.29:0/3346693028":"2024-04-28T01:59:07.772795+0000","172.21.15.29:6801/3122572856":"2024-04-28T01:57:23.638376+0000","172.21.15.29:0/1232596821":"2024-04-28T01:57:23.638376+0000","172.21.15.29:0/611503923":"2024-04-28T01:57:57.788454+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T02:02:55.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.337+0000 7f11c02a3700 1 -- 172.21.15.29:0/916737704 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f11a406c680 msgr2=0x7f11a406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:02:55.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.337+0000 7f11c02a3700 1 --2- 172.21.15.29:0/916737704 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f11a406c680 0x7f11a406eb30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f11b400b5c0 tx=0x7f11b401a040 comp rx=0 tx=0).stop 2024-04-27T02:02:55.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.337+0000 7f11c02a3700 1 -- 172.21.15.29:0/916737704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11b80ffd50 msgr2=0x7f11b8196470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:02:55.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.337+0000 7f11c02a3700 1 --2- 172.21.15.29:0/916737704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11b80ffd50 0x7f11b8196470 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f11a800c8f0 tx=0x7f11a800ccb0 comp rx=0 tx=0).stop 2024-04-27T02:02:55.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.337+0000 7f11c02a3700 1 -- 172.21.15.29:0/916737704 shutdown_connections 2024-04-27T02:02:55.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.337+0000 7f11c02a3700 1 --2- 172.21.15.29:0/916737704 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f11a406c680 0x7f11a406eb30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:55.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.337+0000 7f11c02a3700 1 --2- 172.21.15.29:0/916737704 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f11b80ff3b0 0x7f11b8195f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:55.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.337+0000 7f11c02a3700 1 --2- 172.21.15.29:0/916737704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11b80ffd50 0x7f11b8196470 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:55.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.337+0000 7f11c02a3700 1 -- 172.21.15.29:0/916737704 >> 172.21.15.29:0/916737704 conn(0x7f11b80fac90 msgr2=0x7f11b80fc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:02:55.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.338+0000 7f11c02a3700 1 -- 172.21.15.29:0/916737704 shutdown_connections 2024-04-27T02:02:55.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:55.338+0000 7f11c02a3700 1 -- 172.21.15.29:0/916737704 wait complete. 2024-04-27T02:02:55.993 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-27T02:01:30.306687+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-27T02:02:55.993 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-27T02:02:56.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:55 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/916737704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T02:02:56.207 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:02:56.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:55 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/916737704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T02:02:57.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:56 smithi029 ceph-mon[26769]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:02:57.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:56 smithi052 ceph-mon[32586]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:02:57.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.913+0000 7fa9949df700 1 -- 172.21.15.29:0/2312690415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa990100460 msgr2=0x7fa990100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:02:57.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.913+0000 7fa9949df700 1 --2- 172.21.15.29:0/2312690415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa990100460 0x7fa990100830 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa980009940 tx=0x7fa980009c50 comp rx=0 tx=0).stop 2024-04-27T02:02:57.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.913+0000 7fa9949df700 1 -- 172.21.15.29:0/2312690415 shutdown_connections 2024-04-27T02:02:57.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.913+0000 7fa9949df700 1 --2- 172.21.15.29:0/2312690415 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa990100d70 0x7fa9901094d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:57.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.913+0000 7fa9949df700 1 --2- 172.21.15.29:0/2312690415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa990100460 0x7fa990100830 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:57.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.913+0000 7fa9949df700 1 -- 172.21.15.29:0/2312690415 >> 172.21.15.29:0/2312690415 conn(0x7fa9900748b0 msgr2=0x7fa990074cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:02:57.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.913+0000 7fa9949df700 1 -- 172.21.15.29:0/2312690415 shutdown_connections 2024-04-27T02:02:57.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.913+0000 7fa9949df700 1 -- 172.21.15.29:0/2312690415 wait complete. 2024-04-27T02:02:57.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.914+0000 7fa9949df700 1 Processor -- start 2024-04-27T02:02:57.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.914+0000 7fa9949df700 1 -- start start 2024-04-27T02:02:57.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.914+0000 7fa9949df700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa990100460 0x7fa990197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:02:57.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.914+0000 7fa9949df700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa990100d70 0x7fa9901984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:02:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.914+0000 7fa9949df700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa990198b90 con 0x7fa990100d70 2024-04-27T02:02:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.914+0000 7fa9949df700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99019c920 con 0x7fa990100460 2024-04-27T02:02:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.914+0000 7fa98dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa990100d70 0x7fa9901984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:02:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.914+0000 7fa98dd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa990100d70 0x7fa9901984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46158/0 (socket says 172.21.15.29:46158) 2024-04-27T02:02:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.914+0000 7fa98dd9b700 1 -- 172.21.15.29:0/1925485682 learned_addr learned my addr 172.21.15.29:0/1925485682 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:02:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.914+0000 7fa98e59c700 1 --2- 172.21.15.29:0/1925485682 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa990100460 0x7fa990197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:02:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.915+0000 7fa98dd9b700 1 -- 172.21.15.29:0/1925485682 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa990100460 msgr2=0x7fa990197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:02:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.915+0000 7fa98dd9b700 1 --2- 172.21.15.29:0/1925485682 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa990100460 0x7fa990197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.915+0000 7fa98dd9b700 1 -- 172.21.15.29:0/1925485682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9800095f0 con 0x7fa990100d70 2024-04-27T02:02:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.915+0000 7fa98e59c700 1 --2- 172.21.15.29:0/1925485682 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa990100460 0x7fa990197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T02:02:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.915+0000 7fa98dd9b700 1 --2- 172.21.15.29:0/1925485682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa990100d70 0x7fa9901984b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa98400d8d0 tx=0x7fa98400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:02:57.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.915+0000 7fa97f7fe700 1 -- 172.21.15.29:0/1925485682 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa984009940 con 0x7fa990100d70 2024-04-27T02:02:57.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.915+0000 7fa9949df700 1 -- 172.21.15.29:0/1925485682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99019cc00 con 0x7fa990100d70 2024-04-27T02:02:57.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.915+0000 7fa97f7fe700 1 -- 172.21.15.29:0/1925485682 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa984010460 con 0x7fa990100d70 2024-04-27T02:02:57.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.915+0000 7fa97f7fe700 1 -- 172.21.15.29:0/1925485682 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa98400f5f0 con 0x7fa990100d70 2024-04-27T02:02:57.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.915+0000 7fa9949df700 1 -- 172.21.15.29:0/1925485682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99019d150 con 0x7fa990100d70 2024-04-27T02:02:57.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.916+0000 7fa9949df700 1 -- 172.21.15.29:0/1925485682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa99004f2a0 con 0x7fa990100d70 2024-04-27T02:02:57.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.916+0000 7fa97f7fe700 1 -- 172.21.15.29:0/1925485682 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fa9840105d0 con 0x7fa990100d70 2024-04-27T02:02:57.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.917+0000 7fa97f7fe700 1 --2- 172.21.15.29:0/1925485682 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa97806c680 0x7fa97806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:02:57.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.917+0000 7fa97f7fe700 1 -- 172.21.15.29:0/1925485682 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa98408b540 con 0x7fa990100d70 2024-04-27T02:02:57.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.917+0000 7fa98e59c700 1 --2- 172.21.15.29:0/1925485682 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa97806c680 0x7fa97806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:02:57.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.917+0000 7fa98e59c700 1 --2- 172.21.15.29:0/1925485682 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa97806c680 0x7fa97806eb30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa980009910 tx=0x7fa980005fd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:02:57.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:57.922+0000 7fa97f7fe700 1 -- 172.21.15.29:0/1925485682 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa984055f60 con 0x7fa990100d70 2024-04-27T02:02:58.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:58.121+0000 7fa9949df700 1 -- 172.21.15.29:0/1925485682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa99019d5c0 con 0x7fa990100d70 2024-04-27T02:02:58.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:58.121+0000 7fa97f7fe700 1 -- 172.21.15.29:0/1925485682 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa984059580 con 0x7fa990100d70 2024-04-27T02:02:58.102 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-04-27T02:02:58.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:58.123+0000 7fa9949df700 1 -- 172.21.15.29:0/1925485682 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa97806c680 msgr2=0x7fa97806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:02:58.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:58.123+0000 7fa9949df700 1 --2- 172.21.15.29:0/1925485682 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa97806c680 0x7fa97806eb30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa980009910 tx=0x7fa980005fd0 comp rx=0 tx=0).stop 2024-04-27T02:02:58.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:58.123+0000 7fa9949df700 1 -- 172.21.15.29:0/1925485682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa990100d70 msgr2=0x7fa9901984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:02:58.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:58.123+0000 7fa9949df700 1 --2- 172.21.15.29:0/1925485682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa990100d70 0x7fa9901984b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa98400d8d0 tx=0x7fa98400dc90 comp rx=0 tx=0).stop 2024-04-27T02:02:58.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:58.123+0000 7fa9949df700 1 -- 172.21.15.29:0/1925485682 shutdown_connections 2024-04-27T02:02:58.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:58.123+0000 7fa9949df700 1 --2- 172.21.15.29:0/1925485682 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa97806c680 0x7fa97806eb30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:58.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:58.123+0000 7fa9949df700 1 --2- 172.21.15.29:0/1925485682 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa990100460 0x7fa990197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:58.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:58.123+0000 7fa9949df700 1 --2- 172.21.15.29:0/1925485682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa990100d70 0x7fa9901984b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:02:58.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:58.123+0000 7fa9949df700 1 -- 172.21.15.29:0/1925485682 >> 172.21.15.29:0/1925485682 conn(0x7fa9900748b0 msgr2=0x7fa990107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:02:58.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:58.123+0000 7fa9949df700 1 -- 172.21.15.29:0/1925485682 shutdown_connections 2024-04-27T02:02:58.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:02:58.123+0000 7fa9949df700 1 -- 172.21.15.29:0/1925485682 wait complete. 2024-04-27T02:02:58.760 INFO:tasks.cephadm:Setting up client nodes... 2024-04-27T02:02:58.760 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-27T02:02:58.976 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:02:59.094 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:58 smithi029 ceph-mon[26769]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:02:59.095 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:02:58 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/1925485682' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T02:02:59.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:58 smithi052 ceph-mon[32586]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:02:59.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:02:58 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/1925485682' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T02:03:00.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.711+0000 7f6537be8700 1 -- 172.21.15.29:0/2534474221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65300ff150 msgr2=0x7f65300ff560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:00.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.711+0000 7f6537be8700 1 --2- 172.21.15.29:0/2534474221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65300ff150 0x7f65300ff560 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f652c009910 tx=0x7f652c009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:00.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.712+0000 7f6537be8700 1 -- 172.21.15.29:0/2534474221 shutdown_connections 2024-04-27T02:03:00.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.712+0000 7f6537be8700 1 --2- 172.21.15.29:0/2534474221 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f65301003f0 0x7f6530100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:00.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.712+0000 7f6537be8700 1 --2- 172.21.15.29:0/2534474221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65300ff150 0x7f65300ff560 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:00.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.712+0000 7f6537be8700 1 -- 172.21.15.29:0/2534474221 >> 172.21.15.29:0/2534474221 conn(0x7f65300fa7a0 msgr2=0x7f65300fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:00.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.712+0000 7f6537be8700 1 -- 172.21.15.29:0/2534474221 shutdown_connections 2024-04-27T02:03:00.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.712+0000 7f6537be8700 1 -- 172.21.15.29:0/2534474221 wait complete. 2024-04-27T02:03:00.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.712+0000 7f6537be8700 1 Processor -- start 2024-04-27T02:03:00.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.712+0000 7f6537be8700 1 -- start start 2024-04-27T02:03:00.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.713+0000 7f6537be8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65300ff150 0x7f6530103180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:00.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.713+0000 7f6537be8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f65301003f0 0x7f65301017d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:00.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.713+0000 7f6537be8700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6530101d10 con 0x7f65300ff150 2024-04-27T02:03:00.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.713+0000 7f6537be8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6530101e50 con 0x7f65301003f0 2024-04-27T02:03:00.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.713+0000 7f6535984700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65300ff150 0x7f6530103180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:00.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.713+0000 7f6535984700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65300ff150 0x7f6530103180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46174/0 (socket says 172.21.15.29:46174) 2024-04-27T02:03:00.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.713+0000 7f6535984700 1 -- 172.21.15.29:0/42157286 learned_addr learned my addr 172.21.15.29:0/42157286 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:03:00.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.713+0000 7f6535183700 1 --2- 172.21.15.29:0/42157286 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f65301003f0 0x7f65301017d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:00.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.713+0000 7f6535984700 1 -- 172.21.15.29:0/42157286 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f65301003f0 msgr2=0x7f65301017d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:00.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.713+0000 7f6535984700 1 --2- 172.21.15.29:0/42157286 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f65301003f0 0x7f65301017d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:00.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.713+0000 7f6535984700 1 -- 172.21.15.29:0/42157286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f652c0095f0 con 0x7f65300ff150 2024-04-27T02:03:00.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.713+0000 7f6535984700 1 --2- 172.21.15.29:0/42157286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65300ff150 0x7f6530103180 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f652c000c00 tx=0x7f652c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:00.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.713+0000 7f6526ffd700 1 -- 172.21.15.29:0/42157286 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f652c01d070 con 0x7f65300ff150 2024-04-27T02:03:00.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.713+0000 7f6526ffd700 1 -- 172.21.15.29:0/42157286 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f652c005dc0 con 0x7f65300ff150 2024-04-27T02:03:00.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.713+0000 7f6537be8700 1 -- 172.21.15.29:0/42157286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65301020d0 con 0x7f65300ff150 2024-04-27T02:03:00.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.714+0000 7f6526ffd700 1 -- 172.21.15.29:0/42157286 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f652c0226a0 con 0x7f65300ff150 2024-04-27T02:03:00.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.714+0000 7f6537be8700 1 -- 172.21.15.29:0/42157286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65301025c0 con 0x7f65300ff150 2024-04-27T02:03:00.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.714+0000 7f6537be8700 1 -- 172.21.15.29:0/42157286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6530068ab0 con 0x7f65300ff150 2024-04-27T02:03:00.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.715+0000 7f6526ffd700 1 -- 172.21.15.29:0/42157286 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f652c0038f0 con 0x7f65300ff150 2024-04-27T02:03:00.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.715+0000 7f6526ffd700 1 --2- 172.21.15.29:0/42157286 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f651c06c680 0x7f651c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:00.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.716+0000 7f6526ffd700 1 -- 172.21.15.29:0/42157286 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f652c08c9a0 con 0x7f65300ff150 2024-04-27T02:03:00.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.716+0000 7f6535183700 1 --2- 172.21.15.29:0/42157286 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f651c06c680 0x7f651c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:00.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.716+0000 7f6535183700 1 --2- 172.21.15.29:0/42157286 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f651c06c680 0x7f651c06eb30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6520009c60 tx=0x7f6520009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:00.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:00.720+0000 7f6526ffd700 1 -- 172.21.15.29:0/42157286 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f652c05ae50 con 0x7f65300ff150 2024-04-27T02:03:00.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:00 smithi029 ceph-mon[26769]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:00.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:01.003+0000 7f6537be8700 1 -- 172.21.15.29:0/42157286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f653019e250 con 0x7f65300ff150 2024-04-27T02:03:00.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:01.007+0000 7f6526ffd700 1 -- 172.21.15.29:0/42157286 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f652c003680 con 0x7f65300ff150 2024-04-27T02:03:00.988 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2024-04-27T02:03:00.988 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDVXCxmO6xFABAAbDYM4SKxthmG5jk9AFOEOA== 2024-04-27T02:03:00.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:01.009+0000 7f6537be8700 1 -- 172.21.15.29:0/42157286 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f651c06c680 msgr2=0x7f651c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:00.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:01.009+0000 7f6537be8700 1 --2- 172.21.15.29:0/42157286 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f651c06c680 0x7f651c06eb30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6520009c60 tx=0x7f6520009260 comp rx=0 tx=0).stop 2024-04-27T02:03:00.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:01.009+0000 7f6537be8700 1 -- 172.21.15.29:0/42157286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65300ff150 msgr2=0x7f6530103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:00.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:01.009+0000 7f6537be8700 1 --2- 172.21.15.29:0/42157286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65300ff150 0x7f6530103180 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f652c000c00 tx=0x7f652c00f690 comp rx=0 tx=0).stop 2024-04-27T02:03:00.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:01.009+0000 7f6537be8700 1 -- 172.21.15.29:0/42157286 shutdown_connections 2024-04-27T02:03:00.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:01.009+0000 7f6537be8700 1 --2- 172.21.15.29:0/42157286 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f651c06c680 0x7f651c06eb30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:00.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:01.009+0000 7f6537be8700 1 --2- 172.21.15.29:0/42157286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65300ff150 0x7f6530103180 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:00.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:01.009+0000 7f6537be8700 1 --2- 172.21.15.29:0/42157286 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f65301003f0 0x7f65301017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:00.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:01.009+0000 7f6537be8700 1 -- 172.21.15.29:0/42157286 >> 172.21.15.29:0/42157286 conn(0x7f65300fa7a0 msgr2=0x7f6530068dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:00.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:01.009+0000 7f6537be8700 1 -- 172.21.15.29:0/42157286 shutdown_connections 2024-04-27T02:03:00.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:01.009+0000 7f6537be8700 1 -- 172.21.15.29:0/42157286 wait complete. 2024-04-27T02:03:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:00 smithi052 ceph-mon[32586]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:01.747 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-27T02:03:01.747 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-27T02:03:01.747 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-27T02:03:01.788 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-27T02:03:02.013 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi052/config 2024-04-27T02:03:02.135 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:01 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/42157286' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T02:03:02.135 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:01 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/42157286' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-27T02:03:02.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:01 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/42157286' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T02:03:02.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:01 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/42157286' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-27T02:03:03.114 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:02 smithi052 ceph-mon[32586]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:03.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:02 smithi029 ceph-mon[26769]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:03.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.662+0000 7f92e8ae0700 1 -- 172.21.15.52:0/3559947493 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92e41015c0 msgr2=0x7f92e4101a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:03.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.662+0000 7f92e8ae0700 1 --2- 172.21.15.52:0/3559947493 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92e41015c0 0x7f92e4101a10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f92d8009910 tx=0x7f92d8009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:03.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.662+0000 7f92e8ae0700 1 -- 172.21.15.52:0/3559947493 shutdown_connections 2024-04-27T02:03:03.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.662+0000 7f92e8ae0700 1 --2- 172.21.15.52:0/3559947493 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92e41015c0 0x7f92e4101a10 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:03.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.662+0000 7f92e8ae0700 1 --2- 172.21.15.52:0/3559947493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92e40fe540 0x7f92e40fe950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:03.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.662+0000 7f92e8ae0700 1 -- 172.21.15.52:0/3559947493 >> 172.21.15.52:0/3559947493 conn(0x7f92e40f9e70 msgr2=0x7f92e40fc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:03.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.662+0000 7f92e8ae0700 1 -- 172.21.15.52:0/3559947493 shutdown_connections 2024-04-27T02:03:03.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.662+0000 7f92e8ae0700 1 -- 172.21.15.52:0/3559947493 wait complete. 2024-04-27T02:03:03.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.662+0000 7f92e8ae0700 1 Processor -- start 2024-04-27T02:03:03.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.662+0000 7f92e8ae0700 1 -- start start 2024-04-27T02:03:03.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.663+0000 7f92e8ae0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92e40fe540 0x7f92e4195b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:03.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.663+0000 7f92e8ae0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92e41015c0 0x7f92e4196050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:03.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.663+0000 7f92e8ae0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e4196670 con 0x7f92e40fe540 2024-04-27T02:03:03.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.663+0000 7f92e8ae0700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e41967b0 con 0x7f92e41015c0 2024-04-27T02:03:03.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.663+0000 7f92e1d9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92e41015c0 0x7f92e4196050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:03.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.663+0000 7f92e1d9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92e41015c0 0x7f92e4196050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54248/0 (socket says 172.21.15.52:54248) 2024-04-27T02:03:03.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.663+0000 7f92e1d9b700 1 -- 172.21.15.52:0/908910667 learned_addr learned my addr 172.21.15.52:0/908910667 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-27T02:03:03.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.663+0000 7f92e259c700 1 --2- 172.21.15.52:0/908910667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92e40fe540 0x7f92e4195b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:03.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.663+0000 7f92e1d9b700 1 -- 172.21.15.52:0/908910667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92e40fe540 msgr2=0x7f92e4195b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:03.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.663+0000 7f92e1d9b700 1 --2- 172.21.15.52:0/908910667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92e40fe540 0x7f92e4195b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:03.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.663+0000 7f92e1d9b700 1 -- 172.21.15.52:0/908910667 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d80095f0 con 0x7f92e41015c0 2024-04-27T02:03:03.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.663+0000 7f92e259c700 1 --2- 172.21.15.52:0/908910667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92e40fe540 0x7f92e4195b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T02:03:03.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.663+0000 7f92e1d9b700 1 --2- 172.21.15.52:0/908910667 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92e41015c0 0x7f92e4196050 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f92d8006010 tx=0x7f92d800be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:03.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.663+0000 7f92d37fe700 1 -- 172.21.15.52:0/908910667 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92d801d070 con 0x7f92e41015c0 2024-04-27T02:03:03.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.663+0000 7f92d37fe700 1 -- 172.21.15.52:0/908910667 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f92d8004f50 con 0x7f92e41015c0 2024-04-27T02:03:03.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.663+0000 7f92e8ae0700 1 -- 172.21.15.52:0/908910667 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e419b200 con 0x7f92e41015c0 2024-04-27T02:03:03.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.663+0000 7f92d37fe700 1 -- 172.21.15.52:0/908910667 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92d8021640 con 0x7f92e41015c0 2024-04-27T02:03:03.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.664+0000 7f92e8ae0700 1 -- 172.21.15.52:0/908910667 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e419b6f0 con 0x7f92e41015c0 2024-04-27T02:03:03.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.665+0000 7f92e8ae0700 1 -- 172.21.15.52:0/908910667 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92e4108da0 con 0x7f92e41015c0 2024-04-27T02:03:03.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.665+0000 7f92d37fe700 1 -- 172.21.15.52:0/908910667 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f92d800f460 con 0x7f92e41015c0 2024-04-27T02:03:03.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.665+0000 7f92d37fe700 1 --2- 172.21.15.52:0/908910667 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f92cc06c680 0x7f92cc06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:03.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.666+0000 7f92d37fe700 1 -- 172.21.15.52:0/908910667 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f92d808c6d0 con 0x7f92e41015c0 2024-04-27T02:03:03.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.666+0000 7f92e259c700 1 --2- 172.21.15.52:0/908910667 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f92cc06c680 0x7f92cc06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:03.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.666+0000 7f92e259c700 1 --2- 172.21.15.52:0/908910667 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f92cc06c680 0x7f92cc06eb30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f92d4005fd0 tx=0x7f92d4005f00 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:03.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.670+0000 7f92d37fe700 1 -- 172.21.15.52:0/908910667 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f92d805ac00 con 0x7f92e41015c0 2024-04-27T02:03:03.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.949+0000 7f92e8ae0700 1 -- 172.21.15.52:0/908910667 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f92e4108f60 con 0x7f92e41015c0 2024-04-27T02:03:03.933 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.953+0000 7f92d37fe700 1 -- 172.21.15.52:0/908910667 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7f92d800f710 con 0x7f92e41015c0 2024-04-27T02:03:03.934 INFO:teuthology.orchestra.run.smithi052.stdout:[client.1] 2024-04-27T02:03:03.934 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDXXCxmQcisOBAAYJGhuLSD+fvqeSiOpAILMA== 2024-04-27T02:03:03.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.955+0000 7f92e8ae0700 1 -- 172.21.15.52:0/908910667 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f92cc06c680 msgr2=0x7f92cc06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:03.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.955+0000 7f92e8ae0700 1 --2- 172.21.15.52:0/908910667 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f92cc06c680 0x7f92cc06eb30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f92d4005fd0 tx=0x7f92d4005f00 comp rx=0 tx=0).stop 2024-04-27T02:03:03.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.955+0000 7f92e8ae0700 1 -- 172.21.15.52:0/908910667 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92e41015c0 msgr2=0x7f92e4196050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:03.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.955+0000 7f92e8ae0700 1 --2- 172.21.15.52:0/908910667 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92e41015c0 0x7f92e4196050 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f92d8006010 tx=0x7f92d800be20 comp rx=0 tx=0).stop 2024-04-27T02:03:03.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.955+0000 7f92e8ae0700 1 -- 172.21.15.52:0/908910667 shutdown_connections 2024-04-27T02:03:03.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.955+0000 7f92e8ae0700 1 --2- 172.21.15.52:0/908910667 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f92cc06c680 0x7f92cc06eb30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:03.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.955+0000 7f92e8ae0700 1 --2- 172.21.15.52:0/908910667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92e40fe540 0x7f92e4195b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:03.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.955+0000 7f92e8ae0700 1 --2- 172.21.15.52:0/908910667 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92e41015c0 0x7f92e4196050 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:03.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.955+0000 7f92e8ae0700 1 -- 172.21.15.52:0/908910667 >> 172.21.15.52:0/908910667 conn(0x7f92e40f9e70 msgr2=0x7f92e41047f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:03.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.955+0000 7f92e8ae0700 1 -- 172.21.15.52:0/908910667 shutdown_connections 2024-04-27T02:03:03.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:03:03.955+0000 7f92e8ae0700 1 -- 172.21.15.52:0/908910667 wait complete. 2024-04-27T02:03:04.555 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-27T02:03:04.555 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-27T02:03:04.555 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-27T02:03:04.596 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-27T02:03:04.596 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-27T02:03:04.596 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-27T02:03:04.821 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:03:04.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:04 smithi029 ceph-mon[26769]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:04.944 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:04 smithi029 ceph-mon[26769]: from='client.? 172.21.15.52:0/908910667' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T02:03:04.944 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:04 smithi029 ceph-mon[26769]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T02:03:04.944 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:04 smithi029 ceph-mon[26769]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-27T02:03:05.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:04 smithi052 ceph-mon[32586]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:05.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:04 smithi052 ceph-mon[32586]: from='client.? 172.21.15.52:0/908910667' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T02:03:05.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:04 smithi052 ceph-mon[32586]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T02:03:05.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:04 smithi052 ceph-mon[32586]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-27T02:03:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.517+0000 7f9bf5faa700 1 -- 172.21.15.29:0/2016053209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf00ff350 msgr2=0x7f9bf00ff720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.517+0000 7f9bf5faa700 1 --2- 172.21.15.29:0/2016053209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf00ff350 0x7f9bf00ff720 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9be0009910 tx=0x7f9be0009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.517+0000 7f9bf5faa700 1 -- 172.21.15.29:0/2016053209 shutdown_connections 2024-04-27T02:03:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.517+0000 7f9bf5faa700 1 --2- 172.21.15.29:0/2016053209 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9bf00ffc60 0x7f9bf010a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.517+0000 7f9bf5faa700 1 --2- 172.21.15.29:0/2016053209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf00ff350 0x7f9bf00ff720 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.517+0000 7f9bf5faa700 1 -- 172.21.15.29:0/2016053209 >> 172.21.15.29:0/2016053209 conn(0x7f9bf00748b0 msgr2=0x7f9bf0074cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.517+0000 7f9bf5faa700 1 -- 172.21.15.29:0/2016053209 shutdown_connections 2024-04-27T02:03:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.517+0000 7f9bf5faa700 1 -- 172.21.15.29:0/2016053209 wait complete. 2024-04-27T02:03:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.517+0000 7f9bf5faa700 1 Processor -- start 2024-04-27T02:03:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.517+0000 7f9bf5faa700 1 -- start start 2024-04-27T02:03:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.518+0000 7f9bf5faa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf00ff350 0x7f9bf0195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.518+0000 7f9bf5faa700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9bf00ffc60 0x7f9bf0196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.518+0000 7f9bf5faa700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf0196a50 con 0x7f9bf00ff350 2024-04-27T02:03:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.518+0000 7f9bf5faa700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf019a7e0 con 0x7f9bf00ffc60 2024-04-27T02:03:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.518+0000 7f9bef7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf00ff350 0x7f9bf0195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.518+0000 7f9bef7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf00ff350 0x7f9bf0195e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46192/0 (socket says 172.21.15.29:46192) 2024-04-27T02:03:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.518+0000 7f9bef7fe700 1 -- 172.21.15.29:0/509205137 learned_addr learned my addr 172.21.15.29:0/509205137 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:03:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.518+0000 7f9beeffd700 1 --2- 172.21.15.29:0/509205137 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9bf00ffc60 0x7f9bf0196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.518+0000 7f9bef7fe700 1 -- 172.21.15.29:0/509205137 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9bf00ffc60 msgr2=0x7f9bf0196370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.518+0000 7f9bef7fe700 1 --2- 172.21.15.29:0/509205137 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9bf00ffc60 0x7f9bf0196370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.518+0000 7f9bef7fe700 1 -- 172.21.15.29:0/509205137 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be00095f0 con 0x7f9bf00ff350 2024-04-27T02:03:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.518+0000 7f9bef7fe700 1 --2- 172.21.15.29:0/509205137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf00ff350 0x7f9bf0195e30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9be00092e0 tx=0x7f9be000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.518+0000 7f9becff9700 1 -- 172.21.15.29:0/509205137 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9be001d070 con 0x7f9bf00ff350 2024-04-27T02:03:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.519+0000 7f9becff9700 1 -- 172.21.15.29:0/509205137 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9be0005dc0 con 0x7f9bf00ff350 2024-04-27T02:03:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.519+0000 7f9bf5faa700 1 -- 172.21.15.29:0/509205137 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf019aa60 con 0x7f9bf00ff350 2024-04-27T02:03:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.519+0000 7f9becff9700 1 -- 172.21.15.29:0/509205137 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9be0017400 con 0x7f9bf00ff350 2024-04-27T02:03:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.519+0000 7f9bf5faa700 1 -- 172.21.15.29:0/509205137 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf019af50 con 0x7f9bf00ff350 2024-04-27T02:03:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.520+0000 7f9bf5faa700 1 -- 172.21.15.29:0/509205137 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf0107dd0 con 0x7f9bf00ff350 2024-04-27T02:03:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.520+0000 7f9becff9700 1 -- 172.21.15.29:0/509205137 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f9be000f460 con 0x7f9bf00ff350 2024-04-27T02:03:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.520+0000 7f9becff9700 1 --2- 172.21.15.29:0/509205137 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f9bd806c680 0x7f9bd806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.521+0000 7f9becff9700 1 -- 172.21.15.29:0/509205137 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f9be008c9b0 con 0x7f9bf00ff350 2024-04-27T02:03:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.521+0000 7f9beeffd700 1 --2- 172.21.15.29:0/509205137 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f9bd806c680 0x7f9bd806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.521+0000 7f9beeffd700 1 --2- 172.21.15.29:0/509205137 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f9bd806c680 0x7f9bd806eb30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0197450 tx=0x7f9be400b3c0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:06.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.525+0000 7f9becff9700 1 -- 172.21.15.29:0/509205137 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9be00573d0 con 0x7f9bf00ff350 2024-04-27T02:03:06.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.782+0000 7f9bf5faa700 1 -- 172.21.15.29:0/509205137 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f9bf0197290 con 0x7f9bf00ff350 2024-04-27T02:03:06.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.785+0000 7f9becff9700 1 -- 172.21.15.29:0/509205137 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+172851 (secure 0 0 0) 0x7f9be005a9f0 con 0x7f9bf00ff350 2024-04-27T02:03:06.768 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:03:06.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.788+0000 7f9bf5faa700 1 -- 172.21.15.29:0/509205137 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f9bd806c680 msgr2=0x7f9bd806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:06.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.788+0000 7f9bf5faa700 1 --2- 172.21.15.29:0/509205137 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f9bd806c680 0x7f9bd806eb30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0197450 tx=0x7f9be400b3c0 comp rx=0 tx=0).stop 2024-04-27T02:03:06.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.788+0000 7f9bf5faa700 1 -- 172.21.15.29:0/509205137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf00ff350 msgr2=0x7f9bf0195e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:06.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.788+0000 7f9bf5faa700 1 --2- 172.21.15.29:0/509205137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf00ff350 0x7f9bf0195e30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9be00092e0 tx=0x7f9be000bd70 comp rx=0 tx=0).stop 2024-04-27T02:03:06.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.788+0000 7f9bf5faa700 1 -- 172.21.15.29:0/509205137 shutdown_connections 2024-04-27T02:03:06.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.788+0000 7f9bf5faa700 1 --2- 172.21.15.29:0/509205137 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f9bd806c680 0x7f9bd806eb30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:06.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.788+0000 7f9bf5faa700 1 --2- 172.21.15.29:0/509205137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf00ff350 0x7f9bf0195e30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:06.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.788+0000 7f9bf5faa700 1 --2- 172.21.15.29:0/509205137 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9bf00ffc60 0x7f9bf0196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:06.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.788+0000 7f9bf5faa700 1 -- 172.21.15.29:0/509205137 >> 172.21.15.29:0/509205137 conn(0x7f9bf00748b0 msgr2=0x7f9bf0104f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:06.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.788+0000 7f9bf5faa700 1 -- 172.21.15.29:0/509205137 shutdown_connections 2024-04-27T02:03:06.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:06.788+0000 7f9bf5faa700 1 -- 172.21.15.29:0/509205137 wait complete. 2024-04-27T02:03:07.149 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:06 smithi029 ceph-mon[26769]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:07.149 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:06 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/509205137' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-27T02:03:07.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:06 smithi052 ceph-mon[32586]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:07.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:06 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/509205137' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-27T02:03:07.555 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":19,"active_gid":14215,"active_name":"smithi029.ewvdzs","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":298222579},{"type":"v1","addr":"172.21.15.29:6801","nonce":298222579}]},"active_addr":"172.21.15.29:6801/298222579","active_change":"2024-04-27T01:59:07.772914+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14240,"name":"smithi052.gbsftx","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.29:8443/","prometheus":"http://172.21.15.29:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":740961899}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":1844184351}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":3434352881}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":3079292856}]}]} 2024-04-27T02:03:07.559 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-27T02:03:07.559 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-27T02:03:07.560 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-27T02:03:07.788 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:03:08.151 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:07 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:03:08.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:07 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:03:09.148 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:08 smithi029 ceph-mon[26769]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:09.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:08 smithi052 ceph-mon[32586]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:09.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.509+0000 7f2981b7c700 1 -- 172.21.15.29:0/2242427700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f297c104130 msgr2=0x7f297c104500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:09.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.509+0000 7f2981b7c700 1 --2- 172.21.15.29:0/2242427700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f297c104130 0x7f297c104500 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f296c009910 tx=0x7f296c009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:09.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.509+0000 7f2981b7c700 1 -- 172.21.15.29:0/2242427700 shutdown_connections 2024-04-27T02:03:09.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.509+0000 7f2981b7c700 1 --2- 172.21.15.29:0/2242427700 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f297c0ffbd0 0x7f297c100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:09.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.509+0000 7f2981b7c700 1 --2- 172.21.15.29:0/2242427700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f297c104130 0x7f297c104500 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:09.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.509+0000 7f2981b7c700 1 -- 172.21.15.29:0/2242427700 >> 172.21.15.29:0/2242427700 conn(0x7f297c075130 msgr2=0x7f297c075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:09.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.509+0000 7f2981b7c700 1 -- 172.21.15.29:0/2242427700 shutdown_connections 2024-04-27T02:03:09.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.509+0000 7f2981b7c700 1 -- 172.21.15.29:0/2242427700 wait complete. 2024-04-27T02:03:09.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.510+0000 7f2981b7c700 1 Processor -- start 2024-04-27T02:03:09.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.510+0000 7f2981b7c700 1 -- start start 2024-04-27T02:03:09.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.510+0000 7f2981b7c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f297c0ffbd0 0x7f297c100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:09.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.510+0000 7f2981b7c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f297c104130 0x7f297c101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:09.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.510+0000 7f2981b7c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297c1018b0 con 0x7f297c104130 2024-04-27T02:03:09.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.510+0000 7f2981b7c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297c1019f0 con 0x7f297c0ffbd0 2024-04-27T02:03:09.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.510+0000 7f297affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f297c104130 0x7f297c101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:09.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.510+0000 7f297affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f297c104130 0x7f297c101370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40806/0 (socket says 172.21.15.29:40806) 2024-04-27T02:03:09.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.510+0000 7f297affd700 1 -- 172.21.15.29:0/206488962 learned_addr learned my addr 172.21.15.29:0/206488962 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:03:09.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.510+0000 7f297b7fe700 1 --2- 172.21.15.29:0/206488962 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f297c0ffbd0 0x7f297c100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:09.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.511+0000 7f297affd700 1 -- 172.21.15.29:0/206488962 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f297c0ffbd0 msgr2=0x7f297c100e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:09.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.511+0000 7f297affd700 1 --2- 172.21.15.29:0/206488962 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f297c0ffbd0 0x7f297c100e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:09.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.511+0000 7f297affd700 1 -- 172.21.15.29:0/206488962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296c0095f0 con 0x7f297c104130 2024-04-27T02:03:09.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.511+0000 7f297affd700 1 --2- 172.21.15.29:0/206488962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f297c104130 0x7f297c101370 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f297000e920 tx=0x7f297000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:09.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.511+0000 7f2978ff9700 1 -- 172.21.15.29:0/206488962 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f297000c9d0 con 0x7f297c104130 2024-04-27T02:03:09.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.511+0000 7f2978ff9700 1 -- 172.21.15.29:0/206488962 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2970010430 con 0x7f297c104130 2024-04-27T02:03:09.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.511+0000 7f2981b7c700 1 -- 172.21.15.29:0/206488962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f297c1a2020 con 0x7f297c104130 2024-04-27T02:03:09.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.511+0000 7f2978ff9700 1 -- 172.21.15.29:0/206488962 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2970018650 con 0x7f297c104130 2024-04-27T02:03:09.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.511+0000 7f2981b7c700 1 -- 172.21.15.29:0/206488962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f297c1a2430 con 0x7f297c104130 2024-04-27T02:03:09.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.512+0000 7f2981b7c700 1 -- 172.21.15.29:0/206488962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f297c1086a0 con 0x7f297c104130 2024-04-27T02:03:09.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.513+0000 7f2978ff9700 1 -- 172.21.15.29:0/206488962 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f297000cb30 con 0x7f297c104130 2024-04-27T02:03:09.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.513+0000 7f2978ff9700 1 --2- 172.21.15.29:0/206488962 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f296406c680 0x7f296406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:09.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.513+0000 7f2978ff9700 1 -- 172.21.15.29:0/206488962 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f2970014070 con 0x7f297c104130 2024-04-27T02:03:09.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.513+0000 7f297b7fe700 1 --2- 172.21.15.29:0/206488962 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f296406c680 0x7f296406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:09.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.514+0000 7f297b7fe700 1 --2- 172.21.15.29:0/206488962 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f296406c680 0x7f296406eb30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f296c000c00 tx=0x7f296c009f70 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:09.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.517+0000 7f2978ff9700 1 -- 172.21.15.29:0/206488962 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2970051720 con 0x7f297c104130 2024-04-27T02:03:09.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.720+0000 7f2981b7c700 1 -- 172.21.15.29:0/206488962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f297c101bb0 con 0x7f297c104130 2024-04-27T02:03:09.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.721+0000 7f2978ff9700 1 -- 172.21.15.29:0/206488962 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11299 (secure 0 0 0) 0x7f297c101bb0 con 0x7f297c104130 2024-04-27T02:03:09.701 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:03:09.702 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":40,"fsid":"4c587076-0439-11ef-bc93-c7b262605968","created":"2024-04-27T01:56:52.758582+0000","modified":"2024-04-27T02:02:49.850602+0000","last_up_change":"2024-04-27T02:02:48.845017+0000","last_in_change":"2024-04-27T02:02:29.359236+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-27T02:01:30.306687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3686f584-4bb4-4d6a-8daa-93b9184e4bc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":1138569897},{"type":"v1","addr":"172.21.15.29:6803","nonce":1138569897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":1138569897},{"type":"v1","addr":"172.21.15.29:6805","nonce":1138569897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1138569897},{"type":"v1","addr":"172.21.15.29:6809","nonce":1138569897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":1138569897},{"type":"v1","addr":"172.21.15.29:6807","nonce":1138569897}]},"public_addr":"172.21.15.29:6803/1138569897","cluster_addr":"172.21.15.29:6805/1138569897","heartbeat_back_addr":"172.21.15.29:6809/1138569897","heartbeat_front_addr":"172.21.15.29:6807/1138569897","state":["exists","up"]},{"osd":1,"uuid":"ad9bc7ef-74a2-46c2-a201-a2820c9c8923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3705295680},{"type":"v1","addr":"172.21.15.29:6811","nonce":3705295680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3705295680},{"type":"v1","addr":"172.21.15.29:6813","nonce":3705295680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3705295680},{"type":"v1","addr":"172.21.15.29:6817","nonce":3705295680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3705295680},{"type":"v1","addr":"172.21.15.29:6815","nonce":3705295680}]},"public_addr":"172.21.15.29:6811/3705295680","cluster_addr":"172.21.15.29:6813/3705295680","heartbeat_back_addr":"172.21.15.29:6817/3705295680","heartbeat_front_addr":"172.21.15.29:6815/3705295680","state":["exists","up"]},{"osd":2,"uuid":"315cd205-cf98-4743-8b62-9a158d99a4e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3013669823},{"type":"v1","addr":"172.21.15.29:6819","nonce":3013669823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":3013669823},{"type":"v1","addr":"172.21.15.29:6821","nonce":3013669823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3013669823},{"type":"v1","addr":"172.21.15.29:6825","nonce":3013669823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":3013669823},{"type":"v1","addr":"172.21.15.29:6823","nonce":3013669823}]},"public_addr":"172.21.15.29:6819/3013669823","cluster_addr":"172.21.15.29:6821/3013669823","heartbeat_back_addr":"172.21.15.29:6825/3013669823","heartbeat_front_addr":"172.21.15.29:6823/3013669823","state":["exists","up"]},{"osd":3,"uuid":"59548c0e-dc14-4d71-940c-58802631725b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":263370941},{"type":"v1","addr":"172.21.15.52:6801","nonce":263370941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":263370941},{"type":"v1","addr":"172.21.15.52:6803","nonce":263370941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":263370941},{"type":"v1","addr":"172.21.15.52:6807","nonce":263370941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":263370941},{"type":"v1","addr":"172.21.15.52:6805","nonce":263370941}]},"public_addr":"172.21.15.52:6801/263370941","cluster_addr":"172.21.15.52:6803/263370941","heartbeat_back_addr":"172.21.15.52:6807/263370941","heartbeat_front_addr":"172.21.15.52:6805/263370941","state":["exists","up"]},{"osd":4,"uuid":"c7a75f26-ffa7-447e-8205-478e69d303b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":2191623063},{"type":"v1","addr":"172.21.15.52:6809","nonce":2191623063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":2191623063},{"type":"v1","addr":"172.21.15.52:6811","nonce":2191623063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":2191623063},{"type":"v1","addr":"172.21.15.52:6815","nonce":2191623063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":2191623063},{"type":"v1","addr":"172.21.15.52:6813","nonce":2191623063}]},"public_addr":"172.21.15.52:6809/2191623063","cluster_addr":"172.21.15.52:6811/2191623063","heartbeat_back_addr":"172.21.15.52:6815/2191623063","heartbeat_front_addr":"172.21.15.52:6813/2191623063","state":["exists","up"]},{"osd":5,"uuid":"5cad5f7f-079e-476a-aab5-d5328eda9c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":2667246548},{"type":"v1","addr":"172.21.15.52:6817","nonce":2667246548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":2667246548},{"type":"v1","addr":"172.21.15.52:6819","nonce":2667246548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":2667246548},{"type":"v1","addr":"172.21.15.52:6823","nonce":2667246548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":2667246548},{"type":"v1","addr":"172.21.15.52:6821","nonce":2667246548}]},"public_addr":"172.21.15.52:6817/2667246548","cluster_addr":"172.21.15.52:6819/2667246548","heartbeat_back_addr":"172.21.15.52:6823/2667246548","heartbeat_front_addr":"172.21.15.52:6821/2667246548","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:00:30.289475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:00:59.350372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:01:27.504097+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:01:54.076675+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:02:20.069253+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:02:46.979018+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6801/2469926221":"2024-04-28T01:59:07.772795+0000","172.21.15.29:6801/3873642219":"2024-04-28T01:57:57.788454+0000","172.21.15.29:6800/3873642219":"2024-04-28T01:57:57.788454+0000","172.21.15.29:0/202142181":"2024-04-28T01:57:57.788454+0000","172.21.15.29:0/3675815378":"2024-04-28T01:57:57.788454+0000","172.21.15.29:0/1899826075":"2024-04-28T01:59:07.772795+0000","172.21.15.29:0/297079675":"2024-04-28T01:59:07.772795+0000","172.21.15.29:6800/3122572856":"2024-04-28T01:57:23.638376+0000","172.21.15.29:6800/2469926221":"2024-04-28T01:59:07.772795+0000","172.21.15.29:0/3724837509":"2024-04-28T01:57:23.638376+0000","172.21.15.29:0/909831728":"2024-04-28T01:57:23.638376+0000","172.21.15.29:0/3346693028":"2024-04-28T01:59:07.772795+0000","172.21.15.29:6801/3122572856":"2024-04-28T01:57:23.638376+0000","172.21.15.29:0/1232596821":"2024-04-28T01:57:23.638376+0000","172.21.15.29:0/611503923":"2024-04-28T01:57:57.788454+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T02:03:09.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.723+0000 7f2981b7c700 1 -- 172.21.15.29:0/206488962 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f296406c680 msgr2=0x7f296406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:09.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.723+0000 7f2981b7c700 1 --2- 172.21.15.29:0/206488962 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f296406c680 0x7f296406eb30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f296c000c00 tx=0x7f296c009f70 comp rx=0 tx=0).stop 2024-04-27T02:03:09.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.723+0000 7f2981b7c700 1 -- 172.21.15.29:0/206488962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f297c104130 msgr2=0x7f297c101370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:09.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.723+0000 7f2981b7c700 1 --2- 172.21.15.29:0/206488962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f297c104130 0x7f297c101370 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f297000e920 tx=0x7f297000ec30 comp rx=0 tx=0).stop 2024-04-27T02:03:09.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.723+0000 7f2981b7c700 1 -- 172.21.15.29:0/206488962 shutdown_connections 2024-04-27T02:03:09.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.723+0000 7f2981b7c700 1 --2- 172.21.15.29:0/206488962 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f296406c680 0x7f296406eb30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:09.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.723+0000 7f2981b7c700 1 --2- 172.21.15.29:0/206488962 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f297c0ffbd0 0x7f297c100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:09.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.723+0000 7f2981b7c700 1 --2- 172.21.15.29:0/206488962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f297c104130 0x7f297c101370 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:09.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.723+0000 7f2981b7c700 1 -- 172.21.15.29:0/206488962 >> 172.21.15.29:0/206488962 conn(0x7f297c075130 msgr2=0x7f297c0fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:09.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.723+0000 7f2981b7c700 1 -- 172.21.15.29:0/206488962 shutdown_connections 2024-04-27T02:03:09.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:09.723+0000 7f2981b7c700 1 -- 172.21.15.29:0/206488962 wait complete. 2024-04-27T02:03:10.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:09 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/206488962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T02:03:10.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:09 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/206488962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T02:03:10.535 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-27T02:03:10.535 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-27T02:03:10.749 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:03:11.123 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:10 smithi029 ceph-mon[26769]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:11.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:10 smithi052 ceph-mon[32586]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:12.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.491+0000 7f7d3d4ae700 1 -- 172.21.15.29:0/270608046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d380ffc60 msgr2=0x7f7d3810a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:12.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.491+0000 7f7d3d4ae700 1 --2- 172.21.15.29:0/270608046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d380ffc60 0x7f7d3810a6d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009910 tx=0x7f7d2c009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:12.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.491+0000 7f7d3d4ae700 1 -- 172.21.15.29:0/270608046 shutdown_connections 2024-04-27T02:03:12.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.491+0000 7f7d3d4ae700 1 --2- 172.21.15.29:0/270608046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d380ffc60 0x7f7d3810a6d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:12.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.491+0000 7f7d3d4ae700 1 --2- 172.21.15.29:0/270608046 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d380ff350 0x7f7d380ff720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:12.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.491+0000 7f7d3d4ae700 1 -- 172.21.15.29:0/270608046 >> 172.21.15.29:0/270608046 conn(0x7f7d380748b0 msgr2=0x7f7d38074cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:12.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.491+0000 7f7d3d4ae700 1 -- 172.21.15.29:0/270608046 shutdown_connections 2024-04-27T02:03:12.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.491+0000 7f7d3d4ae700 1 -- 172.21.15.29:0/270608046 wait complete. 2024-04-27T02:03:12.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.492+0000 7f7d3d4ae700 1 Processor -- start 2024-04-27T02:03:12.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.492+0000 7f7d3d4ae700 1 -- start start 2024-04-27T02:03:12.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.492+0000 7f7d3d4ae700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d380ff350 0x7f7d38195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:12.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.492+0000 7f7d3d4ae700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d380ffc60 0x7f7d38196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:12.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.492+0000 7f7d3d4ae700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d38196b50 con 0x7f7d380ffc60 2024-04-27T02:03:12.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.492+0000 7f7d3d4ae700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3819a8e0 con 0x7f7d380ff350 2024-04-27T02:03:12.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.492+0000 7f7d367fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d380ffc60 0x7f7d38196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:12.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.492+0000 7f7d367fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d380ffc60 0x7f7d38196470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40818/0 (socket says 172.21.15.29:40818) 2024-04-27T02:03:12.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.492+0000 7f7d367fc700 1 -- 172.21.15.29:0/66127577 learned_addr learned my addr 172.21.15.29:0/66127577 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:03:12.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.492+0000 7f7d36ffd700 1 --2- 172.21.15.29:0/66127577 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d380ff350 0x7f7d38195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:12.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.492+0000 7f7d367fc700 1 -- 172.21.15.29:0/66127577 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d380ff350 msgr2=0x7f7d38195f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:12.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.492+0000 7f7d367fc700 1 --2- 172.21.15.29:0/66127577 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d380ff350 0x7f7d38195f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:12.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.492+0000 7f7d367fc700 1 -- 172.21.15.29:0/66127577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2c0095f0 con 0x7f7d380ffc60 2024-04-27T02:03:12.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.493+0000 7f7d367fc700 1 --2- 172.21.15.29:0/66127577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d380ffc60 0x7f7d38196470 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c006010 tx=0x7f7d2c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:12.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.493+0000 7f7d1ffff700 1 -- 172.21.15.29:0/66127577 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d2c01d070 con 0x7f7d380ffc60 2024-04-27T02:03:12.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.493+0000 7f7d3d4ae700 1 -- 172.21.15.29:0/66127577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3819ab60 con 0x7f7d380ffc60 2024-04-27T02:03:12.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.493+0000 7f7d1ffff700 1 -- 172.21.15.29:0/66127577 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7d2c004fa0 con 0x7f7d380ffc60 2024-04-27T02:03:12.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.493+0000 7f7d1ffff700 1 -- 172.21.15.29:0/66127577 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d2c00f690 con 0x7f7d380ffc60 2024-04-27T02:03:12.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.493+0000 7f7d3d4ae700 1 -- 172.21.15.29:0/66127577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d3819b050 con 0x7f7d380ffc60 2024-04-27T02:03:12.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.494+0000 7f7d3d4ae700 1 -- 172.21.15.29:0/66127577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d38107dd0 con 0x7f7d380ffc60 2024-04-27T02:03:12.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.494+0000 7f7d1ffff700 1 -- 172.21.15.29:0/66127577 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f7d2c022470 con 0x7f7d380ffc60 2024-04-27T02:03:12.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.495+0000 7f7d1ffff700 1 --2- 172.21.15.29:0/66127577 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f7d2006c680 0x7f7d2006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:12.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.495+0000 7f7d1ffff700 1 -- 172.21.15.29:0/66127577 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7d2c08cd10 con 0x7f7d380ffc60 2024-04-27T02:03:12.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.495+0000 7f7d36ffd700 1 --2- 172.21.15.29:0/66127577 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f7d2006c680 0x7f7d2006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:12.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.495+0000 7f7d36ffd700 1 --2- 172.21.15.29:0/66127577 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f7d2006c680 0x7f7d2006eb30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7d3806a640 tx=0x7f7d28009190 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:12.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.499+0000 7f7d1ffff700 1 -- 172.21.15.29:0/66127577 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7d2c057730 con 0x7f7d380ffc60 2024-04-27T02:03:12.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.699+0000 7f7d3d4ae700 1 -- 172.21.15.29:0/66127577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d3804f2a0 con 0x7f7d380ffc60 2024-04-27T02:03:12.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.700+0000 7f7d1ffff700 1 -- 172.21.15.29:0/66127577 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11299 (secure 0 0 0) 0x7f7d2c05ad50 con 0x7f7d380ffc60 2024-04-27T02:03:12.680 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:03:12.681 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":40,"fsid":"4c587076-0439-11ef-bc93-c7b262605968","created":"2024-04-27T01:56:52.758582+0000","modified":"2024-04-27T02:02:49.850602+0000","last_up_change":"2024-04-27T02:02:48.845017+0000","last_in_change":"2024-04-27T02:02:29.359236+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-27T02:01:30.306687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3686f584-4bb4-4d6a-8daa-93b9184e4bc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":1138569897},{"type":"v1","addr":"172.21.15.29:6803","nonce":1138569897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":1138569897},{"type":"v1","addr":"172.21.15.29:6805","nonce":1138569897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1138569897},{"type":"v1","addr":"172.21.15.29:6809","nonce":1138569897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":1138569897},{"type":"v1","addr":"172.21.15.29:6807","nonce":1138569897}]},"public_addr":"172.21.15.29:6803/1138569897","cluster_addr":"172.21.15.29:6805/1138569897","heartbeat_back_addr":"172.21.15.29:6809/1138569897","heartbeat_front_addr":"172.21.15.29:6807/1138569897","state":["exists","up"]},{"osd":1,"uuid":"ad9bc7ef-74a2-46c2-a201-a2820c9c8923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3705295680},{"type":"v1","addr":"172.21.15.29:6811","nonce":3705295680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3705295680},{"type":"v1","addr":"172.21.15.29:6813","nonce":3705295680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3705295680},{"type":"v1","addr":"172.21.15.29:6817","nonce":3705295680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3705295680},{"type":"v1","addr":"172.21.15.29:6815","nonce":3705295680}]},"public_addr":"172.21.15.29:6811/3705295680","cluster_addr":"172.21.15.29:6813/3705295680","heartbeat_back_addr":"172.21.15.29:6817/3705295680","heartbeat_front_addr":"172.21.15.29:6815/3705295680","state":["exists","up"]},{"osd":2,"uuid":"315cd205-cf98-4743-8b62-9a158d99a4e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3013669823},{"type":"v1","addr":"172.21.15.29:6819","nonce":3013669823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":3013669823},{"type":"v1","addr":"172.21.15.29:6821","nonce":3013669823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3013669823},{"type":"v1","addr":"172.21.15.29:6825","nonce":3013669823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":3013669823},{"type":"v1","addr":"172.21.15.29:6823","nonce":3013669823}]},"public_addr":"172.21.15.29:6819/3013669823","cluster_addr":"172.21.15.29:6821/3013669823","heartbeat_back_addr":"172.21.15.29:6825/3013669823","heartbeat_front_addr":"172.21.15.29:6823/3013669823","state":["exists","up"]},{"osd":3,"uuid":"59548c0e-dc14-4d71-940c-58802631725b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":263370941},{"type":"v1","addr":"172.21.15.52:6801","nonce":263370941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":263370941},{"type":"v1","addr":"172.21.15.52:6803","nonce":263370941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":263370941},{"type":"v1","addr":"172.21.15.52:6807","nonce":263370941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":263370941},{"type":"v1","addr":"172.21.15.52:6805","nonce":263370941}]},"public_addr":"172.21.15.52:6801/263370941","cluster_addr":"172.21.15.52:6803/263370941","heartbeat_back_addr":"172.21.15.52:6807/263370941","heartbeat_front_addr":"172.21.15.52:6805/263370941","state":["exists","up"]},{"osd":4,"uuid":"c7a75f26-ffa7-447e-8205-478e69d303b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":2191623063},{"type":"v1","addr":"172.21.15.52:6809","nonce":2191623063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":2191623063},{"type":"v1","addr":"172.21.15.52:6811","nonce":2191623063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":2191623063},{"type":"v1","addr":"172.21.15.52:6815","nonce":2191623063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":2191623063},{"type":"v1","addr":"172.21.15.52:6813","nonce":2191623063}]},"public_addr":"172.21.15.52:6809/2191623063","cluster_addr":"172.21.15.52:6811/2191623063","heartbeat_back_addr":"172.21.15.52:6815/2191623063","heartbeat_front_addr":"172.21.15.52:6813/2191623063","state":["exists","up"]},{"osd":5,"uuid":"5cad5f7f-079e-476a-aab5-d5328eda9c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":2667246548},{"type":"v1","addr":"172.21.15.52:6817","nonce":2667246548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":2667246548},{"type":"v1","addr":"172.21.15.52:6819","nonce":2667246548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":2667246548},{"type":"v1","addr":"172.21.15.52:6823","nonce":2667246548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":2667246548},{"type":"v1","addr":"172.21.15.52:6821","nonce":2667246548}]},"public_addr":"172.21.15.52:6817/2667246548","cluster_addr":"172.21.15.52:6819/2667246548","heartbeat_back_addr":"172.21.15.52:6823/2667246548","heartbeat_front_addr":"172.21.15.52:6821/2667246548","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:00:30.289475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:00:59.350372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:01:27.504097+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:01:54.076675+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:02:20.069253+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:02:46.979018+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6801/2469926221":"2024-04-28T01:59:07.772795+0000","172.21.15.29:6801/3873642219":"2024-04-28T01:57:57.788454+0000","172.21.15.29:6800/3873642219":"2024-04-28T01:57:57.788454+0000","172.21.15.29:0/202142181":"2024-04-28T01:57:57.788454+0000","172.21.15.29:0/3675815378":"2024-04-28T01:57:57.788454+0000","172.21.15.29:0/1899826075":"2024-04-28T01:59:07.772795+0000","172.21.15.29:0/297079675":"2024-04-28T01:59:07.772795+0000","172.21.15.29:6800/3122572856":"2024-04-28T01:57:23.638376+0000","172.21.15.29:6800/2469926221":"2024-04-28T01:59:07.772795+0000","172.21.15.29:0/3724837509":"2024-04-28T01:57:23.638376+0000","172.21.15.29:0/909831728":"2024-04-28T01:57:23.638376+0000","172.21.15.29:0/3346693028":"2024-04-28T01:59:07.772795+0000","172.21.15.29:6801/3122572856":"2024-04-28T01:57:23.638376+0000","172.21.15.29:0/1232596821":"2024-04-28T01:57:23.638376+0000","172.21.15.29:0/611503923":"2024-04-28T01:57:57.788454+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T02:03:12.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.702+0000 7f7d3d4ae700 1 -- 172.21.15.29:0/66127577 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f7d2006c680 msgr2=0x7f7d2006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:12.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.702+0000 7f7d3d4ae700 1 --2- 172.21.15.29:0/66127577 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f7d2006c680 0x7f7d2006eb30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7d3806a640 tx=0x7f7d28009190 comp rx=0 tx=0).stop 2024-04-27T02:03:12.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.702+0000 7f7d3d4ae700 1 -- 172.21.15.29:0/66127577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d380ffc60 msgr2=0x7f7d38196470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:12.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.702+0000 7f7d3d4ae700 1 --2- 172.21.15.29:0/66127577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d380ffc60 0x7f7d38196470 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c006010 tx=0x7f7d2c00bd70 comp rx=0 tx=0).stop 2024-04-27T02:03:12.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.703+0000 7f7d3d4ae700 1 -- 172.21.15.29:0/66127577 shutdown_connections 2024-04-27T02:03:12.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.703+0000 7f7d3d4ae700 1 --2- 172.21.15.29:0/66127577 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f7d2006c680 0x7f7d2006eb30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:12.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.703+0000 7f7d3d4ae700 1 --2- 172.21.15.29:0/66127577 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d380ff350 0x7f7d38195f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:12.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.703+0000 7f7d3d4ae700 1 --2- 172.21.15.29:0/66127577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d380ffc60 0x7f7d38196470 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:12.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.703+0000 7f7d3d4ae700 1 -- 172.21.15.29:0/66127577 >> 172.21.15.29:0/66127577 conn(0x7f7d380748b0 msgr2=0x7f7d38104f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:12.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.703+0000 7f7d3d4ae700 1 -- 172.21.15.29:0/66127577 shutdown_connections 2024-04-27T02:03:12.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:12.703+0000 7f7d3d4ae700 1 -- 172.21.15.29:0/66127577 wait complete. 2024-04-27T02:03:12.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:12 smithi029 ceph-mon[26769]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:12.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:12 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/66127577' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T02:03:13.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:12 smithi052 ceph-mon[32586]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:13.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:12 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/66127577' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T02:03:13.385 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-27T02:03:13.385 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-27T02:03:13.386 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-27T02:03:13.386 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-27T02:03:13.387 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-27T02:03:13.387 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-27T02:03:13.662 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:03:13.662 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:03:15.115 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:14 smithi029 ceph-mon[26769]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:15.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:14 smithi052 ceph-mon[32586]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:15.988 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:03:16.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.727+0000 7f8a90043700 1 -- 172.21.15.29:0/3898774748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a880687a0 msgr2=0x7f8a88068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:16.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.727+0000 7f8a90043700 1 --2- 172.21.15.29:0/3898774748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a880687a0 0x7f8a88068b70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8a84009910 tx=0x7f8a84009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:16.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.727+0000 7f8a90043700 1 -- 172.21.15.29:0/3898774748 shutdown_connections 2024-04-27T02:03:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.727+0000 7f8a90043700 1 --2- 172.21.15.29:0/3898774748 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a88069140 0x7f8a88104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.727+0000 7f8a90043700 1 --2- 172.21.15.29:0/3898774748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a880687a0 0x7f8a88068b70 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.727+0000 7f8a90043700 1 -- 172.21.15.29:0/3898774748 >> 172.21.15.29:0/3898774748 conn(0x7f8a88075130 msgr2=0x7f8a88075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.727+0000 7f8a90043700 1 -- 172.21.15.29:0/3898774748 shutdown_connections 2024-04-27T02:03:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.727+0000 7f8a90043700 1 -- 172.21.15.29:0/3898774748 wait complete. 2024-04-27T02:03:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.728+0000 7f8a90043700 1 Processor -- start 2024-04-27T02:03:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.728+0000 7f8a90043700 1 -- start start 2024-04-27T02:03:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.728+0000 7f8a90043700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a880687a0 0x7f8a880ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.728+0000 7f8a90043700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a88069140 0x7f8a881001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.728+0000 7f8a90043700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a88100720 con 0x7f8a88069140 2024-04-27T02:03:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.728+0000 7f8a90043700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a88100860 con 0x7f8a880687a0 2024-04-27T02:03:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.728+0000 7f8a8d5de700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a88069140 0x7f8a881001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.728+0000 7f8a8d5de700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a88069140 0x7f8a881001e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45310/0 (socket says 172.21.15.29:45310) 2024-04-27T02:03:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.728+0000 7f8a8d5de700 1 -- 172.21.15.29:0/776494686 learned_addr learned my addr 172.21.15.29:0/776494686 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:03:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.728+0000 7f8a8dddf700 1 --2- 172.21.15.29:0/776494686 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a880687a0 0x7f8a880ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.728+0000 7f8a8d5de700 1 -- 172.21.15.29:0/776494686 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a880687a0 msgr2=0x7f8a880ffca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.728+0000 7f8a8d5de700 1 --2- 172.21.15.29:0/776494686 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a880687a0 0x7f8a880ffca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.729+0000 7f8a8d5de700 1 -- 172.21.15.29:0/776494686 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a840095f0 con 0x7f8a88069140 2024-04-27T02:03:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.729+0000 7f8a8dddf700 1 --2- 172.21.15.29:0/776494686 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a880687a0 0x7f8a880ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T02:03:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.729+0000 7f8a8d5de700 1 --2- 172.21.15.29:0/776494686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a88069140 0x7f8a881001e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8a7800b480 tx=0x7f8a7800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.729+0000 7f8a7effd700 1 -- 172.21.15.29:0/776494686 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a780118c0 con 0x7f8a88069140 2024-04-27T02:03:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.729+0000 7f8a7effd700 1 -- 172.21.15.29:0/776494686 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8a78010380 con 0x7f8a88069140 2024-04-27T02:03:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.729+0000 7f8a90043700 1 -- 172.21.15.29:0/776494686 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a881a2110 con 0x7f8a88069140 2024-04-27T02:03:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.729+0000 7f8a7effd700 1 -- 172.21.15.29:0/776494686 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a7800f4f0 con 0x7f8a88069140 2024-04-27T02:03:16.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.729+0000 7f8a90043700 1 -- 172.21.15.29:0/776494686 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a881a2520 con 0x7f8a88069140 2024-04-27T02:03:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.730+0000 7f8a90043700 1 -- 172.21.15.29:0/776494686 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8a8810d6f0 con 0x7f8a88069140 2024-04-27T02:03:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.730+0000 7f8a7effd700 1 -- 172.21.15.29:0/776494686 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f8a780104f0 con 0x7f8a88069140 2024-04-27T02:03:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.731+0000 7f8a7effd700 1 --2- 172.21.15.29:0/776494686 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f8a7406c680 0x7f8a7406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.731+0000 7f8a8dddf700 1 --2- 172.21.15.29:0/776494686 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f8a7406c680 0x7f8a7406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.731+0000 7f8a7effd700 1 -- 172.21.15.29:0/776494686 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f8a7808b180 con 0x7f8a88069140 2024-04-27T02:03:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.731+0000 7f8a7effd700 1 --2- 172.21.15.29:0/776494686 >> [v2:172.21.15.52:6816/2667246548,v1:172.21.15.52:6817/2667246548] conn(0x7f8a74072350 0x7f8a74074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.731+0000 7f8a7effd700 1 -- 172.21.15.29:0/776494686 --> [v2:172.21.15.52:6816/2667246548,v1:172.21.15.52:6817/2667246548] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8a74074e10 con 0x7f8a74072350 2024-04-27T02:03:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.731+0000 7f8a7effd700 1 -- 172.21.15.29:0/776494686 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a7808d850 con 0x7f8a88069140 2024-04-27T02:03:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.731+0000 7f8a8dddf700 1 --2- 172.21.15.29:0/776494686 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f8a7406c680 0x7f8a7406eb30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8a8400b5c0 tx=0x7f8a84005480 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.731+0000 7f8a8e5e0700 1 --2- 172.21.15.29:0/776494686 >> [v2:172.21.15.52:6816/2667246548,v1:172.21.15.52:6817/2667246548] conn(0x7f8a74072350 0x7f8a74074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.732+0000 7f8a8e5e0700 1 --2- 172.21.15.29:0/776494686 >> [v2:172.21.15.52:6816/2667246548,v1:172.21.15.52:6817/2667246548] conn(0x7f8a74072350 0x7f8a74074760 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.733+0000 7f8a7effd700 1 -- 172.21.15.29:0/776494686 <== osd.5 v2:172.21.15.52:6816/2667246548 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f8a74074e10 con 0x7f8a74072350 2024-04-27T02:03:16.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.748+0000 7f8a90043700 1 -- 172.21.15.29:0/776494686 --> [v2:172.21.15.52:6816/2667246548,v1:172.21.15.52:6817/2667246548] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8a8804f2a0 con 0x7f8a74072350 2024-04-27T02:03:16.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.749+0000 7f8a7effd700 1 -- 172.21.15.29:0/776494686 <== osd.5 v2:172.21.15.52:6816/2667246548 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8a8804f2a0 con 0x7f8a74072350 2024-04-27T02:03:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.749+0000 7f8a90043700 1 -- 172.21.15.29:0/776494686 >> [v2:172.21.15.52:6816/2667246548,v1:172.21.15.52:6817/2667246548] conn(0x7f8a74072350 msgr2=0x7f8a74074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.749+0000 7f8a90043700 1 --2- 172.21.15.29:0/776494686 >> [v2:172.21.15.52:6816/2667246548,v1:172.21.15.52:6817/2667246548] conn(0x7f8a74072350 0x7f8a74074760 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.749+0000 7f8a90043700 1 -- 172.21.15.29:0/776494686 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f8a7406c680 msgr2=0x7f8a7406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.749+0000 7f8a90043700 1 --2- 172.21.15.29:0/776494686 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f8a7406c680 0x7f8a7406eb30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8a8400b5c0 tx=0x7f8a84005480 comp rx=0 tx=0).stop 2024-04-27T02:03:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.749+0000 7f8a90043700 1 -- 172.21.15.29:0/776494686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a88069140 msgr2=0x7f8a881001e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.749+0000 7f8a90043700 1 --2- 172.21.15.29:0/776494686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a88069140 0x7f8a881001e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8a7800b480 tx=0x7f8a7800b840 comp rx=0 tx=0).stop 2024-04-27T02:03:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.750+0000 7f8a90043700 1 -- 172.21.15.29:0/776494686 shutdown_connections 2024-04-27T02:03:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.750+0000 7f8a90043700 1 --2- 172.21.15.29:0/776494686 >> [v2:172.21.15.52:6816/2667246548,v1:172.21.15.52:6817/2667246548] conn(0x7f8a74072350 0x7f8a74074760 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.750+0000 7f8a90043700 1 --2- 172.21.15.29:0/776494686 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f8a7406c680 0x7f8a7406eb30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.750+0000 7f8a90043700 1 --2- 172.21.15.29:0/776494686 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a880687a0 0x7f8a880ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.750+0000 7f8a90043700 1 --2- 172.21.15.29:0/776494686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a88069140 0x7f8a881001e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.750+0000 7f8a90043700 1 -- 172.21.15.29:0/776494686 >> 172.21.15.29:0/776494686 conn(0x7f8a88075130 msgr2=0x7f8a880fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.750+0000 7f8a90043700 1 -- 172.21.15.29:0/776494686 shutdown_connections 2024-04-27T02:03:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:16.750+0000 7f8a90043700 1 -- 172.21.15.29:0/776494686 wait complete. 2024-04-27T02:03:16.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:16 smithi029 ceph-mon[26769]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:17.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:16 smithi052 ceph-mon[32586]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:17.376 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:03:17.376 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:03:17.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.783+0000 7f3588223700 1 -- 172.21.15.29:0/1619659874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3580100e80 msgr2=0x7f3580104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:17.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.783+0000 7f3588223700 1 --2- 172.21.15.29:0/1619659874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3580100e80 0x7f3580104e50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3574009910 tx=0x7f3574009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:17.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.783+0000 7f3588223700 1 -- 172.21.15.29:0/1619659874 shutdown_connections 2024-04-27T02:03:17.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.783+0000 7f3588223700 1 --2- 172.21.15.29:0/1619659874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3580100e80 0x7f3580104e50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:17.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.783+0000 7f3588223700 1 --2- 172.21.15.29:0/1619659874 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35801004e0 0x7f35801008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:17.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.783+0000 7f3588223700 1 -- 172.21.15.29:0/1619659874 >> 172.21.15.29:0/1619659874 conn(0x7f35800fbda0 msgr2=0x7f35800fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:17.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.783+0000 7f3588223700 1 -- 172.21.15.29:0/1619659874 shutdown_connections 2024-04-27T02:03:17.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.783+0000 7f3588223700 1 -- 172.21.15.29:0/1619659874 wait complete. 2024-04-27T02:03:17.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.783+0000 7f3588223700 1 Processor -- start 2024-04-27T02:03:17.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.784+0000 7f3588223700 1 -- start start 2024-04-27T02:03:17.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.784+0000 7f3588223700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35801004e0 0x7f3580073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:17.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.784+0000 7f3588223700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3580100e80 0x7f3580073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:17.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.784+0000 7f3588223700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3580073ad0 con 0x7f35801004e0 2024-04-27T02:03:17.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.784+0000 7f3588223700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3580073c10 con 0x7f3580100e80 2024-04-27T02:03:17.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.784+0000 7f3585fbf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35801004e0 0x7f3580073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:17.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.784+0000 7f3585fbf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35801004e0 0x7f3580073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45330/0 (socket says 172.21.15.29:45330) 2024-04-27T02:03:17.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.784+0000 7f3585fbf700 1 -- 172.21.15.29:0/183727640 learned_addr learned my addr 172.21.15.29:0/183727640 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:03:17.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.784+0000 7f35857be700 1 --2- 172.21.15.29:0/183727640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3580100e80 0x7f3580073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:17.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.784+0000 7f3585fbf700 1 -- 172.21.15.29:0/183727640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3580100e80 msgr2=0x7f3580073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:17.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.784+0000 7f3585fbf700 1 --2- 172.21.15.29:0/183727640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3580100e80 0x7f3580073590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:17.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.784+0000 7f3585fbf700 1 -- 172.21.15.29:0/183727640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35740095f0 con 0x7f35801004e0 2024-04-27T02:03:17.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.784+0000 7f3585fbf700 1 --2- 172.21.15.29:0/183727640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35801004e0 0x7f3580073050 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f35700060b0 tx=0x7f357000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:17.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.785+0000 7f357effd700 1 -- 172.21.15.29:0/183727640 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f357000dda0 con 0x7f35801004e0 2024-04-27T02:03:17.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.785+0000 7f357effd700 1 -- 172.21.15.29:0/183727640 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f357000f040 con 0x7f35801004e0 2024-04-27T02:03:17.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.785+0000 7f3588223700 1 -- 172.21.15.29:0/183727640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35801a2100 con 0x7f35801004e0 2024-04-27T02:03:17.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.785+0000 7f357effd700 1 -- 172.21.15.29:0/183727640 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35700146f0 con 0x7f35801004e0 2024-04-27T02:03:17.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.785+0000 7f3588223700 1 -- 172.21.15.29:0/183727640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35801a24b0 con 0x7f35801004e0 2024-04-27T02:03:17.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.785+0000 7f3588223700 1 -- 172.21.15.29:0/183727640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f358010d650 con 0x7f35801004e0 2024-04-27T02:03:17.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.786+0000 7f357effd700 1 -- 172.21.15.29:0/183727640 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f3570004ad0 con 0x7f35801004e0 2024-04-27T02:03:17.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.787+0000 7f357effd700 1 --2- 172.21.15.29:0/183727640 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f356806c680 0x7f356806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:17.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.787+0000 7f357effd700 1 -- 172.21.15.29:0/183727640 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f357008b3d0 con 0x7f35801004e0 2024-04-27T02:03:17.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.787+0000 7f35857be700 1 --2- 172.21.15.29:0/183727640 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f356806c680 0x7f356806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:17.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.787+0000 7f357effd700 1 --2- 172.21.15.29:0/183727640 >> [v2:172.21.15.29:6802/1138569897,v1:172.21.15.29:6803/1138569897] conn(0x7f3568072350 0x7f3568074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:17.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.787+0000 7f357effd700 1 -- 172.21.15.29:0/183727640 --> [v2:172.21.15.29:6802/1138569897,v1:172.21.15.29:6803/1138569897] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3568074e10 con 0x7f3568072350 2024-04-27T02:03:17.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.787+0000 7f357effd700 1 -- 172.21.15.29:0/183727640 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f357008b830 con 0x7f35801004e0 2024-04-27T02:03:17.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.787+0000 7f35867c0700 1 --2- 172.21.15.29:0/183727640 >> [v2:172.21.15.29:6802/1138569897,v1:172.21.15.29:6803/1138569897] conn(0x7f3568072350 0x7f3568074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:17.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.787+0000 7f35857be700 1 --2- 172.21.15.29:0/183727640 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f356806c680 0x7f356806eb30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3574006010 tx=0x7f357401a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:17.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.787+0000 7f35867c0700 1 --2- 172.21.15.29:0/183727640 >> [v2:172.21.15.29:6802/1138569897,v1:172.21.15.29:6803/1138569897] conn(0x7f3568072350 0x7f3568074760 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:17.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.788+0000 7f357effd700 1 -- 172.21.15.29:0/183727640 <== osd.0 v2:172.21.15.29:6802/1138569897 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f3568074e10 con 0x7f3568072350 2024-04-27T02:03:17.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.804+0000 7f3588223700 1 -- 172.21.15.29:0/183727640 --> [v2:172.21.15.29:6802/1138569897,v1:172.21.15.29:6803/1138569897] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f358004f2a0 con 0x7f3568072350 2024-04-27T02:03:17.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.804+0000 7f357effd700 1 -- 172.21.15.29:0/183727640 <== osd.0 v2:172.21.15.29:6802/1138569897 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f358004f2a0 con 0x7f3568072350 2024-04-27T02:03:17.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.805+0000 7f3588223700 1 -- 172.21.15.29:0/183727640 >> [v2:172.21.15.29:6802/1138569897,v1:172.21.15.29:6803/1138569897] conn(0x7f3568072350 msgr2=0x7f3568074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:17.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.805+0000 7f3588223700 1 --2- 172.21.15.29:0/183727640 >> [v2:172.21.15.29:6802/1138569897,v1:172.21.15.29:6803/1138569897] conn(0x7f3568072350 0x7f3568074760 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:17.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.805+0000 7f3588223700 1 -- 172.21.15.29:0/183727640 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f356806c680 msgr2=0x7f356806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:17.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.805+0000 7f3588223700 1 --2- 172.21.15.29:0/183727640 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f356806c680 0x7f356806eb30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3574006010 tx=0x7f357401a040 comp rx=0 tx=0).stop 2024-04-27T02:03:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.805+0000 7f3588223700 1 -- 172.21.15.29:0/183727640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35801004e0 msgr2=0x7f3580073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.805+0000 7f3588223700 1 --2- 172.21.15.29:0/183727640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35801004e0 0x7f3580073050 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f35700060b0 tx=0x7f357000d3b0 comp rx=0 tx=0).stop 2024-04-27T02:03:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.805+0000 7f3588223700 1 -- 172.21.15.29:0/183727640 shutdown_connections 2024-04-27T02:03:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.805+0000 7f3588223700 1 --2- 172.21.15.29:0/183727640 >> [v2:172.21.15.29:6802/1138569897,v1:172.21.15.29:6803/1138569897] conn(0x7f3568072350 0x7f3568074760 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.805+0000 7f3588223700 1 --2- 172.21.15.29:0/183727640 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f356806c680 0x7f356806eb30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.805+0000 7f3588223700 1 --2- 172.21.15.29:0/183727640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35801004e0 0x7f3580073050 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.805+0000 7f3588223700 1 --2- 172.21.15.29:0/183727640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3580100e80 0x7f3580073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.805+0000 7f3588223700 1 -- 172.21.15.29:0/183727640 >> 172.21.15.29:0/183727640 conn(0x7f35800fbda0 msgr2=0x7f35800fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.805+0000 7f3588223700 1 -- 172.21.15.29:0/183727640 shutdown_connections 2024-04-27T02:03:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:17.805+0000 7f3588223700 1 -- 172.21.15.29:0/183727640 wait complete. 2024-04-27T02:03:18.476 INFO:teuthology.orchestra.run.smithi029.stdout:167503724553 2024-04-27T02:03:18.476 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-27T02:03:18.732 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:03:19.107 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:18 smithi029 ceph-mon[26769]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:19.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:18 smithi052 ceph-mon[32586]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:21.137 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:20 smithi029 ceph-mon[26769]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:21.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:20 smithi052 ceph-mon[32586]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:21.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.358+0000 7fc6f0698700 1 -- 172.21.15.29:0/387210403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6e80ff350 msgr2=0x7fc6e80ff720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:21.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.358+0000 7fc6f0698700 1 --2- 172.21.15.29:0/387210403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6e80ff350 0x7fc6e80ff720 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8009910 tx=0x7fc6d8009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:21.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.358+0000 7fc6f0698700 1 -- 172.21.15.29:0/387210403 shutdown_connections 2024-04-27T02:03:21.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.358+0000 7fc6f0698700 1 --2- 172.21.15.29:0/387210403 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc6e80ffc60 0x7fc6e810a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:21.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.358+0000 7fc6f0698700 1 --2- 172.21.15.29:0/387210403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6e80ff350 0x7fc6e80ff720 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:21.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.359+0000 7fc6f0698700 1 -- 172.21.15.29:0/387210403 >> 172.21.15.29:0/387210403 conn(0x7fc6e80748b0 msgr2=0x7fc6e8074cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:21.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.359+0000 7fc6f0698700 1 -- 172.21.15.29:0/387210403 shutdown_connections 2024-04-27T02:03:21.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.359+0000 7fc6f0698700 1 -- 172.21.15.29:0/387210403 wait complete. 2024-04-27T02:03:21.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.359+0000 7fc6f0698700 1 Processor -- start 2024-04-27T02:03:21.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.359+0000 7fc6f0698700 1 -- start start 2024-04-27T02:03:21.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.360+0000 7fc6f0698700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6e80ff350 0x7fc6e8100530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:21.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.360+0000 7fc6f0698700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc6e80ffc60 0x7fc6e8100a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:21.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.360+0000 7fc6f0698700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e8100fb0 con 0x7fc6e80ff350 2024-04-27T02:03:21.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.360+0000 7fc6f0698700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e81010f0 con 0x7fc6e80ffc60 2024-04-27T02:03:21.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.360+0000 7fc6ee434700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6e80ff350 0x7fc6e8100530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:21.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.360+0000 7fc6ee434700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6e80ff350 0x7fc6e8100530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45346/0 (socket says 172.21.15.29:45346) 2024-04-27T02:03:21.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.360+0000 7fc6ee434700 1 -- 172.21.15.29:0/3779636597 learned_addr learned my addr 172.21.15.29:0/3779636597 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:03:21.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.360+0000 7fc6edc33700 1 --2- 172.21.15.29:0/3779636597 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc6e80ffc60 0x7fc6e8100a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:21.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.360+0000 7fc6ee434700 1 -- 172.21.15.29:0/3779636597 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc6e80ffc60 msgr2=0x7fc6e8100a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:21.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.360+0000 7fc6ee434700 1 --2- 172.21.15.29:0/3779636597 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc6e80ffc60 0x7fc6e8100a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:21.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.360+0000 7fc6ee434700 1 -- 172.21.15.29:0/3779636597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d80095f0 con 0x7fc6e80ff350 2024-04-27T02:03:21.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.360+0000 7fc6ee434700 1 --2- 172.21.15.29:0/3779636597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6e80ff350 0x7fc6e8100530 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8000c00 tx=0x7fc6d80055f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:21.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.360+0000 7fc6df7fe700 1 -- 172.21.15.29:0/3779636597 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6d801d070 con 0x7fc6e80ff350 2024-04-27T02:03:21.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.360+0000 7fc6f0698700 1 -- 172.21.15.29:0/3779636597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e81a2070 con 0x7fc6e80ff350 2024-04-27T02:03:21.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.360+0000 7fc6df7fe700 1 -- 172.21.15.29:0/3779636597 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc6d8005dc0 con 0x7fc6e80ff350 2024-04-27T02:03:21.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.360+0000 7fc6f0698700 1 -- 172.21.15.29:0/3779636597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e81a2480 con 0x7fc6e80ff350 2024-04-27T02:03:21.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.360+0000 7fc6df7fe700 1 -- 172.21.15.29:0/3779636597 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6d8021410 con 0x7fc6e80ff350 2024-04-27T02:03:21.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.361+0000 7fc6f0698700 1 -- 172.21.15.29:0/3779636597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc6e810d5a0 con 0x7fc6e80ff350 2024-04-27T02:03:21.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.362+0000 7fc6df7fe700 1 -- 172.21.15.29:0/3779636597 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fc6d800f460 con 0x7fc6e80ff350 2024-04-27T02:03:21.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.363+0000 7fc6df7fe700 1 --2- 172.21.15.29:0/3779636597 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fc6d406c680 0x7fc6d406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:21.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.363+0000 7fc6edc33700 1 --2- 172.21.15.29:0/3779636597 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fc6d406c680 0x7fc6d406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:21.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.363+0000 7fc6df7fe700 1 -- 172.21.15.29:0/3779636597 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc6d808c800 con 0x7fc6e80ff350 2024-04-27T02:03:21.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.363+0000 7fc6df7fe700 1 --2- 172.21.15.29:0/3779636597 >> [v2:172.21.15.52:6800/263370941,v1:172.21.15.52:6801/263370941] conn(0x7fc6d4072350 0x7fc6d4074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:21.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.363+0000 7fc6df7fe700 1 -- 172.21.15.29:0/3779636597 --> [v2:172.21.15.52:6800/263370941,v1:172.21.15.52:6801/263370941] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6d4074e10 con 0x7fc6d4072350 2024-04-27T02:03:21.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.363+0000 7fc6df7fe700 1 -- 172.21.15.29:0/3779636597 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6d80912a0 con 0x7fc6e80ff350 2024-04-27T02:03:21.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.363+0000 7fc6edc33700 1 --2- 172.21.15.29:0/3779636597 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fc6d406c680 0x7fc6d406eb30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8101d20 tx=0x7fc6e400b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:21.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.363+0000 7fc6eec35700 1 --2- 172.21.15.29:0/3779636597 >> [v2:172.21.15.52:6800/263370941,v1:172.21.15.52:6801/263370941] conn(0x7fc6d4072350 0x7fc6d4074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:21.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.364+0000 7fc6eec35700 1 --2- 172.21.15.29:0/3779636597 >> [v2:172.21.15.52:6800/263370941,v1:172.21.15.52:6801/263370941] conn(0x7fc6d4072350 0x7fc6d4074760 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:21.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.365+0000 7fc6df7fe700 1 -- 172.21.15.29:0/3779636597 <== osd.3 v2:172.21.15.52:6800/263370941 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc6d4074e10 con 0x7fc6d4072350 2024-04-27T02:03:21.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.381+0000 7fc6f0698700 1 -- 172.21.15.29:0/3779636597 --> [v2:172.21.15.52:6800/263370941,v1:172.21.15.52:6801/263370941] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc6e804f2a0 con 0x7fc6d4072350 2024-04-27T02:03:21.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.381+0000 7fc6df7fe700 1 -- 172.21.15.29:0/3779636597 <== osd.3 v2:172.21.15.52:6800/263370941 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc6e804f2a0 con 0x7fc6d4072350 2024-04-27T02:03:21.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.382+0000 7fc6f0698700 1 -- 172.21.15.29:0/3779636597 >> [v2:172.21.15.52:6800/263370941,v1:172.21.15.52:6801/263370941] conn(0x7fc6d4072350 msgr2=0x7fc6d4074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:21.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.382+0000 7fc6f0698700 1 --2- 172.21.15.29:0/3779636597 >> [v2:172.21.15.52:6800/263370941,v1:172.21.15.52:6801/263370941] conn(0x7fc6d4072350 0x7fc6d4074760 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:21.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.382+0000 7fc6f0698700 1 -- 172.21.15.29:0/3779636597 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fc6d406c680 msgr2=0x7fc6d406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:21.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.382+0000 7fc6f0698700 1 --2- 172.21.15.29:0/3779636597 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fc6d406c680 0x7fc6d406eb30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8101d20 tx=0x7fc6e400b410 comp rx=0 tx=0).stop 2024-04-27T02:03:21.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.382+0000 7fc6f0698700 1 -- 172.21.15.29:0/3779636597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6e80ff350 msgr2=0x7fc6e8100530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:21.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.382+0000 7fc6f0698700 1 --2- 172.21.15.29:0/3779636597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6e80ff350 0x7fc6e8100530 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8000c00 tx=0x7fc6d80055f0 comp rx=0 tx=0).stop 2024-04-27T02:03:21.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.382+0000 7fc6f0698700 1 -- 172.21.15.29:0/3779636597 shutdown_connections 2024-04-27T02:03:21.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.382+0000 7fc6f0698700 1 --2- 172.21.15.29:0/3779636597 >> [v2:172.21.15.52:6800/263370941,v1:172.21.15.52:6801/263370941] conn(0x7fc6d4072350 0x7fc6d4074760 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:21.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.382+0000 7fc6f0698700 1 --2- 172.21.15.29:0/3779636597 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fc6d406c680 0x7fc6d406eb30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:21.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.382+0000 7fc6f0698700 1 --2- 172.21.15.29:0/3779636597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6e80ff350 0x7fc6e8100530 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:21.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.382+0000 7fc6f0698700 1 --2- 172.21.15.29:0/3779636597 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc6e80ffc60 0x7fc6e8100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:21.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.382+0000 7fc6f0698700 1 -- 172.21.15.29:0/3779636597 >> 172.21.15.29:0/3779636597 conn(0x7fc6e80748b0 msgr2=0x7fc6e8104f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:21.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.382+0000 7fc6f0698700 1 -- 172.21.15.29:0/3779636597 shutdown_connections 2024-04-27T02:03:21.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:21.382+0000 7fc6f0698700 1 -- 172.21.15.29:0/3779636597 wait complete. 2024-04-27T02:03:22.179 INFO:teuthology.orchestra.run.smithi029.stdout:38654705700 2024-04-27T02:03:22.179 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-27T02:03:23.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:22 smithi029 ceph-mon[26769]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:23.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:22 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:03:23.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:22 smithi052 ceph-mon[32586]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:23.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:22 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:03:23.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.709+0000 7f6b48256700 1 -- 172.21.15.29:0/103725573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b40104130 msgr2=0x7f6b40104500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:23.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.709+0000 7f6b48256700 1 --2- 172.21.15.29:0/103725573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b40104130 0x7f6b40104500 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c009910 tx=0x7f6b3c009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:23.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.710+0000 7f6b48256700 1 -- 172.21.15.29:0/103725573 shutdown_connections 2024-04-27T02:03:23.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.710+0000 7f6b48256700 1 --2- 172.21.15.29:0/103725573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b400ffbd0 0x7f6b40100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:23.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.710+0000 7f6b48256700 1 --2- 172.21.15.29:0/103725573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b40104130 0x7f6b40104500 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:23.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.710+0000 7f6b48256700 1 -- 172.21.15.29:0/103725573 >> 172.21.15.29:0/103725573 conn(0x7f6b40075130 msgr2=0x7f6b40075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:23.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.710+0000 7f6b48256700 1 -- 172.21.15.29:0/103725573 shutdown_connections 2024-04-27T02:03:23.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.710+0000 7f6b48256700 1 -- 172.21.15.29:0/103725573 wait complete. 2024-04-27T02:03:23.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.710+0000 7f6b48256700 1 Processor -- start 2024-04-27T02:03:23.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.710+0000 7f6b48256700 1 -- start start 2024-04-27T02:03:23.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.711+0000 7f6b48256700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b400ffbd0 0x7f6b40195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:23.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.711+0000 7f6b48256700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b40104130 0x7f6b40196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:23.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.711+0000 7f6b48256700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b40196a70 con 0x7f6b400ffbd0 2024-04-27T02:03:23.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.711+0000 7f6b48256700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4019a800 con 0x7f6b40104130 2024-04-27T02:03:23.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.711+0000 7f6b45ff2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b400ffbd0 0x7f6b40195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:23.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.711+0000 7f6b45ff2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b400ffbd0 0x7f6b40195e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45368/0 (socket says 172.21.15.29:45368) 2024-04-27T02:03:23.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.711+0000 7f6b45ff2700 1 -- 172.21.15.29:0/2590040393 learned_addr learned my addr 172.21.15.29:0/2590040393 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:03:23.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.711+0000 7f6b457f1700 1 --2- 172.21.15.29:0/2590040393 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b40104130 0x7f6b40196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:23.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.711+0000 7f6b45ff2700 1 -- 172.21.15.29:0/2590040393 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b40104130 msgr2=0x7f6b40196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:23.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.711+0000 7f6b45ff2700 1 --2- 172.21.15.29:0/2590040393 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b40104130 0x7f6b40196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:23.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.711+0000 7f6b45ff2700 1 -- 172.21.15.29:0/2590040393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3c0095f0 con 0x7f6b400ffbd0 2024-04-27T02:03:23.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.711+0000 7f6b45ff2700 1 --2- 172.21.15.29:0/2590040393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b400ffbd0 0x7f6b40195e50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c00bd70 tx=0x7f6b3c00bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:23.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.711+0000 7f6b36ffd700 1 -- 172.21.15.29:0/2590040393 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b3c01d070 con 0x7f6b400ffbd0 2024-04-27T02:03:23.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.711+0000 7f6b48256700 1 -- 172.21.15.29:0/2590040393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b4019aa80 con 0x7f6b400ffbd0 2024-04-27T02:03:23.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.711+0000 7f6b36ffd700 1 -- 172.21.15.29:0/2590040393 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6b3c00f460 con 0x7f6b400ffbd0 2024-04-27T02:03:23.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.711+0000 7f6b36ffd700 1 -- 172.21.15.29:0/2590040393 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b3c017630 con 0x7f6b400ffbd0 2024-04-27T02:03:23.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.711+0000 7f6b48256700 1 -- 172.21.15.29:0/2590040393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b4019af70 con 0x7f6b400ffbd0 2024-04-27T02:03:23.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.712+0000 7f6b48256700 1 -- 172.21.15.29:0/2590040393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6b4010c5c0 con 0x7f6b400ffbd0 2024-04-27T02:03:23.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.713+0000 7f6b36ffd700 1 -- 172.21.15.29:0/2590040393 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f6b3c003a60 con 0x7f6b400ffbd0 2024-04-27T02:03:23.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.713+0000 7f6b36ffd700 1 --2- 172.21.15.29:0/2590040393 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f6b2c06c680 0x7f6b2c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:23.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.713+0000 7f6b36ffd700 1 -- 172.21.15.29:0/2590040393 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f6b3c08c9a0 con 0x7f6b400ffbd0 2024-04-27T02:03:23.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.713+0000 7f6b457f1700 1 --2- 172.21.15.29:0/2590040393 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f6b2c06c680 0x7f6b2c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:23.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.713+0000 7f6b36ffd700 1 --2- 172.21.15.29:0/2590040393 >> [v2:172.21.15.29:6818/3013669823,v1:172.21.15.29:6819/3013669823] conn(0x7f6b2c072350 0x7f6b2c074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:23.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.713+0000 7f6b36ffd700 1 -- 172.21.15.29:0/2590040393 --> [v2:172.21.15.29:6818/3013669823,v1:172.21.15.29:6819/3013669823] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6b2c074e10 con 0x7f6b2c072350 2024-04-27T02:03:23.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.713+0000 7f6b36ffd700 1 -- 172.21.15.29:0/2590040393 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b3c0912a0 con 0x7f6b400ffbd0 2024-04-27T02:03:23.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.714+0000 7f6b467f3700 1 --2- 172.21.15.29:0/2590040393 >> [v2:172.21.15.29:6818/3013669823,v1:172.21.15.29:6819/3013669823] conn(0x7f6b2c072350 0x7f6b2c074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:23.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.714+0000 7f6b457f1700 1 --2- 172.21.15.29:0/2590040393 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f6b2c06c680 0x7f6b2c06eb30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6b40197470 tx=0x7f6b3000b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:23.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.714+0000 7f6b467f3700 1 --2- 172.21.15.29:0/2590040393 >> [v2:172.21.15.29:6818/3013669823,v1:172.21.15.29:6819/3013669823] conn(0x7f6b2c072350 0x7f6b2c074760 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:23.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.715+0000 7f6b36ffd700 1 -- 172.21.15.29:0/2590040393 <== osd.2 v2:172.21.15.29:6818/3013669823 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f6b2c074e10 con 0x7f6b2c072350 2024-04-27T02:03:23.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.730+0000 7f6b48256700 1 -- 172.21.15.29:0/2590040393 --> [v2:172.21.15.29:6818/3013669823,v1:172.21.15.29:6819/3013669823] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6b4004f2a0 con 0x7f6b2c072350 2024-04-27T02:03:23.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.731+0000 7f6b36ffd700 1 -- 172.21.15.29:0/2590040393 <== osd.2 v2:172.21.15.29:6818/3013669823 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6b4004f2a0 con 0x7f6b2c072350 2024-04-27T02:03:23.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.731+0000 7f6b48256700 1 -- 172.21.15.29:0/2590040393 >> [v2:172.21.15.29:6818/3013669823,v1:172.21.15.29:6819/3013669823] conn(0x7f6b2c072350 msgr2=0x7f6b2c074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:23.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.731+0000 7f6b48256700 1 --2- 172.21.15.29:0/2590040393 >> [v2:172.21.15.29:6818/3013669823,v1:172.21.15.29:6819/3013669823] conn(0x7f6b2c072350 0x7f6b2c074760 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:23.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.731+0000 7f6b48256700 1 -- 172.21.15.29:0/2590040393 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f6b2c06c680 msgr2=0x7f6b2c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:23.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.731+0000 7f6b48256700 1 --2- 172.21.15.29:0/2590040393 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f6b2c06c680 0x7f6b2c06eb30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6b40197470 tx=0x7f6b3000b410 comp rx=0 tx=0).stop 2024-04-27T02:03:23.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.731+0000 7f6b48256700 1 -- 172.21.15.29:0/2590040393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b400ffbd0 msgr2=0x7f6b40195e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:23.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.731+0000 7f6b48256700 1 --2- 172.21.15.29:0/2590040393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b400ffbd0 0x7f6b40195e50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c00bd70 tx=0x7f6b3c00bda0 comp rx=0 tx=0).stop 2024-04-27T02:03:23.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.731+0000 7f6b48256700 1 -- 172.21.15.29:0/2590040393 shutdown_connections 2024-04-27T02:03:23.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.731+0000 7f6b48256700 1 --2- 172.21.15.29:0/2590040393 >> [v2:172.21.15.29:6818/3013669823,v1:172.21.15.29:6819/3013669823] conn(0x7f6b2c072350 0x7f6b2c074760 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:23.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.731+0000 7f6b48256700 1 --2- 172.21.15.29:0/2590040393 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f6b2c06c680 0x7f6b2c06eb30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:23.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.731+0000 7f6b48256700 1 --2- 172.21.15.29:0/2590040393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b400ffbd0 0x7f6b40195e50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:23.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.731+0000 7f6b48256700 1 --2- 172.21.15.29:0/2590040393 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b40104130 0x7f6b40196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:23.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.731+0000 7f6b48256700 1 -- 172.21.15.29:0/2590040393 >> 172.21.15.29:0/2590040393 conn(0x7f6b40075130 msgr2=0x7f6b400fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:23.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.731+0000 7f6b48256700 1 -- 172.21.15.29:0/2590040393 shutdown_connections 2024-04-27T02:03:23.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:23.731+0000 7f6b48256700 1 -- 172.21.15.29:0/2590040393 wait complete. 2024-04-27T02:03:24.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.408+0000 7f168ad26700 1 -- 172.21.15.29:0/3399183890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1684068540 msgr2=0x7f16840689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.408+0000 7f168ad26700 1 --2- 172.21.15.29:0/3399183890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1684068540 0x7f16840689b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1670009910 tx=0x7f1670009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.408+0000 7f168ad26700 1 -- 172.21.15.29:0/3399183890 shutdown_connections 2024-04-27T02:03:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.408+0000 7f168ad26700 1 --2- 172.21.15.29:0/3399183890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1684068540 0x7f16840689b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.408+0000 7f168ad26700 1 --2- 172.21.15.29:0/3399183890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1684106320 0x7f16841066f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.408+0000 7f168ad26700 1 -- 172.21.15.29:0/3399183890 >> 172.21.15.29:0/3399183890 conn(0x7f1684075130 msgr2=0x7f1684075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.408+0000 7f168ad26700 1 -- 172.21.15.29:0/3399183890 shutdown_connections 2024-04-27T02:03:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.408+0000 7f168ad26700 1 -- 172.21.15.29:0/3399183890 wait complete. 2024-04-27T02:03:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.409+0000 7f168ad26700 1 Processor -- start 2024-04-27T02:03:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.409+0000 7f168ad26700 1 -- start start 2024-04-27T02:03:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.409+0000 7f168ad26700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1684068540 0x7f16840ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.409+0000 7f168ad26700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1684106320 0x7f16841001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.409+0000 7f168ad26700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1684100720 con 0x7f1684068540 2024-04-27T02:03:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.409+0000 7f168ad26700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1684100860 con 0x7f1684106320 2024-04-27T02:03:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.409+0000 7f1688ac2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1684068540 0x7f16840ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.409+0000 7f1688ac2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1684068540 0x7f16840ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45388/0 (socket says 172.21.15.29:45388) 2024-04-27T02:03:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.409+0000 7f1688ac2700 1 -- 172.21.15.29:0/2296812260 learned_addr learned my addr 172.21.15.29:0/2296812260 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:03:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.409+0000 7f1688ac2700 1 -- 172.21.15.29:0/2296812260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1684106320 msgr2=0x7f16841001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.409+0000 7f1683fff700 1 --2- 172.21.15.29:0/2296812260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1684106320 0x7f16841001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.409+0000 7f1688ac2700 1 --2- 172.21.15.29:0/2296812260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1684106320 0x7f16841001e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.409+0000 7f1688ac2700 1 -- 172.21.15.29:0/2296812260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16700095f0 con 0x7f1684068540 2024-04-27T02:03:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.409+0000 7f1688ac2700 1 --2- 172.21.15.29:0/2296812260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1684068540 0x7f16840ffca0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f167800e920 tx=0x7f167800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.410+0000 7f1681ffb700 1 -- 172.21.15.29:0/2296812260 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1678009790 con 0x7f1684068540 2024-04-27T02:03:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.410+0000 7f168ad26700 1 -- 172.21.15.29:0/2296812260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16841a2030 con 0x7f1684068540 2024-04-27T02:03:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.410+0000 7f1681ffb700 1 -- 172.21.15.29:0/2296812260 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f167800cba0 con 0x7f1684068540 2024-04-27T02:03:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.410+0000 7f1681ffb700 1 -- 172.21.15.29:0/2296812260 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1678018630 con 0x7f1684068540 2024-04-27T02:03:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.410+0000 7f168ad26700 1 -- 172.21.15.29:0/2296812260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16841a2390 con 0x7f1684068540 2024-04-27T02:03:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.411+0000 7f168ad26700 1 -- 172.21.15.29:0/2296812260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f168410d610 con 0x7f1684068540 2024-04-27T02:03:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.411+0000 7f1681ffb700 1 -- 172.21.15.29:0/2296812260 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f167800cd10 con 0x7f1684068540 2024-04-27T02:03:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.412+0000 7f1681ffb700 1 --2- 172.21.15.29:0/2296812260 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f167406c680 0x7f167406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:24.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.412+0000 7f1681ffb700 1 -- 172.21.15.29:0/2296812260 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f1678014070 con 0x7f1684068540 2024-04-27T02:03:24.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.412+0000 7f1683fff700 1 --2- 172.21.15.29:0/2296812260 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f167406c680 0x7f167406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:24.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.412+0000 7f1681ffb700 1 --2- 172.21.15.29:0/2296812260 >> [v2:172.21.15.52:6808/2191623063,v1:172.21.15.52:6809/2191623063] conn(0x7f1674072350 0x7f1674074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:24.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.412+0000 7f1681ffb700 1 -- 172.21.15.29:0/2296812260 --> [v2:172.21.15.52:6808/2191623063,v1:172.21.15.52:6809/2191623063] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1674074e10 con 0x7f1674072350 2024-04-27T02:03:24.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.412+0000 7f1681ffb700 1 -- 172.21.15.29:0/2296812260 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16780902a0 con 0x7f1684068540 2024-04-27T02:03:24.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.412+0000 7f1683fff700 1 --2- 172.21.15.29:0/2296812260 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f167406c680 0x7f167406eb30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1684101420 tx=0x7f1670009f70 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:24.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.412+0000 7f16892c3700 1 --2- 172.21.15.29:0/2296812260 >> [v2:172.21.15.52:6808/2191623063,v1:172.21.15.52:6809/2191623063] conn(0x7f1674072350 0x7f1674074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:24.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.413+0000 7f16892c3700 1 --2- 172.21.15.29:0/2296812260 >> [v2:172.21.15.52:6808/2191623063,v1:172.21.15.52:6809/2191623063] conn(0x7f1674072350 0x7f1674074760 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:24.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.413+0000 7f1681ffb700 1 -- 172.21.15.29:0/2296812260 <== osd.4 v2:172.21.15.52:6808/2191623063 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f1674074e10 con 0x7f1674072350 2024-04-27T02:03:24.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.431+0000 7f168ad26700 1 -- 172.21.15.29:0/2296812260 --> [v2:172.21.15.52:6808/2191623063,v1:172.21.15.52:6809/2191623063] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f168404f2a0 con 0x7f1674072350 2024-04-27T02:03:24.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.431+0000 7f1681ffb700 1 -- 172.21.15.29:0/2296812260 <== osd.4 v2:172.21.15.52:6808/2191623063 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f168404f2a0 con 0x7f1674072350 2024-04-27T02:03:24.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.431+0000 7f168ad26700 1 -- 172.21.15.29:0/2296812260 >> [v2:172.21.15.52:6808/2191623063,v1:172.21.15.52:6809/2191623063] conn(0x7f1674072350 msgr2=0x7f1674074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:24.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.431+0000 7f168ad26700 1 --2- 172.21.15.29:0/2296812260 >> [v2:172.21.15.52:6808/2191623063,v1:172.21.15.52:6809/2191623063] conn(0x7f1674072350 0x7f1674074760 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:24.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.432+0000 7f168ad26700 1 -- 172.21.15.29:0/2296812260 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f167406c680 msgr2=0x7f167406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:24.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.432+0000 7f168ad26700 1 --2- 172.21.15.29:0/2296812260 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f167406c680 0x7f167406eb30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1684101420 tx=0x7f1670009f70 comp rx=0 tx=0).stop 2024-04-27T02:03:24.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.432+0000 7f168ad26700 1 -- 172.21.15.29:0/2296812260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1684068540 msgr2=0x7f16840ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:24.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.432+0000 7f168ad26700 1 --2- 172.21.15.29:0/2296812260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1684068540 0x7f16840ffca0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f167800e920 tx=0x7f167800ece0 comp rx=0 tx=0).stop 2024-04-27T02:03:24.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.432+0000 7f168ad26700 1 -- 172.21.15.29:0/2296812260 shutdown_connections 2024-04-27T02:03:24.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.432+0000 7f168ad26700 1 --2- 172.21.15.29:0/2296812260 >> [v2:172.21.15.52:6808/2191623063,v1:172.21.15.52:6809/2191623063] conn(0x7f1674072350 0x7f1674074760 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:24.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.432+0000 7f168ad26700 1 --2- 172.21.15.29:0/2296812260 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f167406c680 0x7f167406eb30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:24.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.432+0000 7f168ad26700 1 --2- 172.21.15.29:0/2296812260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1684068540 0x7f16840ffca0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:24.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.432+0000 7f168ad26700 1 --2- 172.21.15.29:0/2296812260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1684106320 0x7f16841001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:24.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.432+0000 7f168ad26700 1 -- 172.21.15.29:0/2296812260 >> 172.21.15.29:0/2296812260 conn(0x7f1684075130 msgr2=0x7f16840febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:24.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.432+0000 7f168ad26700 1 -- 172.21.15.29:0/2296812260 shutdown_connections 2024-04-27T02:03:24.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:24.432+0000 7f168ad26700 1 -- 172.21.15.29:0/2296812260 wait complete. 2024-04-27T02:03:24.968 INFO:teuthology.orchestra.run.smithi029.stdout:81604378650 2024-04-27T02:03:24.968 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-27T02:03:25.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:24 smithi029 ceph-mon[26769]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:25.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:24 smithi052 ceph-mon[32586]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:25.671 INFO:teuthology.orchestra.run.smithi029.stdout:146028888079 2024-04-27T02:03:25.671 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-27T02:03:26.133 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:03:26.529 INFO:teuthology.orchestra.run.smithi029.stdout:111669149716 2024-04-27T02:03:26.529 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-27T02:03:26.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.004+0000 7ff9cc336700 1 -- 172.21.15.29:0/4287122466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9c41004e0 msgr2=0x7ff9c41008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:26.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.004+0000 7ff9cc336700 1 --2- 172.21.15.29:0/4287122466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9c41004e0 0x7ff9c41008b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0009910 tx=0x7ff9c0009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:26.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.004+0000 7ff9cc336700 1 -- 172.21.15.29:0/4287122466 shutdown_connections 2024-04-27T02:03:26.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.004+0000 7ff9cc336700 1 --2- 172.21.15.29:0/4287122466 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff9c4100e80 0x7ff9c4104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:26.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.004+0000 7ff9cc336700 1 --2- 172.21.15.29:0/4287122466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9c41004e0 0x7ff9c41008b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:26.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.004+0000 7ff9cc336700 1 -- 172.21.15.29:0/4287122466 >> 172.21.15.29:0/4287122466 conn(0x7ff9c40fbda0 msgr2=0x7ff9c40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:26.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.004+0000 7ff9cc336700 1 -- 172.21.15.29:0/4287122466 shutdown_connections 2024-04-27T02:03:26.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.004+0000 7ff9cc336700 1 -- 172.21.15.29:0/4287122466 wait complete. 2024-04-27T02:03:26.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.005+0000 7ff9cc336700 1 Processor -- start 2024-04-27T02:03:26.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.005+0000 7ff9cc336700 1 -- start start 2024-04-27T02:03:26.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.005+0000 7ff9cc336700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9c41004e0 0x7ff9c4195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:26.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.005+0000 7ff9cc336700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff9c4100e80 0x7ff9c4196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:26.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.005+0000 7ff9cc336700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c4196b10 con 0x7ff9c41004e0 2024-04-27T02:03:26.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.005+0000 7ff9cc336700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c419a8a0 con 0x7ff9c4100e80 2024-04-27T02:03:26.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.005+0000 7ff9ca0d2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9c41004e0 0x7ff9c4195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:26.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.005+0000 7ff9ca0d2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9c41004e0 0x7ff9c4195ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33104/0 (socket says 172.21.15.29:33104) 2024-04-27T02:03:26.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.005+0000 7ff9ca0d2700 1 -- 172.21.15.29:0/4031719202 learned_addr learned my addr 172.21.15.29:0/4031719202 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:03:26.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.005+0000 7ff9c98d1700 1 --2- 172.21.15.29:0/4031719202 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff9c4100e80 0x7ff9c4196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:26.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.005+0000 7ff9ca0d2700 1 -- 172.21.15.29:0/4031719202 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff9c4100e80 msgr2=0x7ff9c4196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:26.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.005+0000 7ff9ca0d2700 1 --2- 172.21.15.29:0/4031719202 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff9c4100e80 0x7ff9c4196430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:26.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.006+0000 7ff9ca0d2700 1 -- 172.21.15.29:0/4031719202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c00095f0 con 0x7ff9c41004e0 2024-04-27T02:03:26.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.006+0000 7ff9c98d1700 1 --2- 172.21.15.29:0/4031719202 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff9c4100e80 0x7ff9c4196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T02:03:26.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.006+0000 7ff9ca0d2700 1 --2- 172.21.15.29:0/4031719202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9c41004e0 0x7ff9c4195ef0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0000c00 tx=0x7ff9c000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:26.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.006+0000 7ff9bb7fe700 1 -- 172.21.15.29:0/4031719202 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9c001d070 con 0x7ff9c41004e0 2024-04-27T02:03:26.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.006+0000 7ff9bb7fe700 1 -- 172.21.15.29:0/4031719202 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff9c0005dc0 con 0x7ff9c41004e0 2024-04-27T02:03:26.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.006+0000 7ff9bb7fe700 1 -- 172.21.15.29:0/4031719202 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9c0017400 con 0x7ff9c41004e0 2024-04-27T02:03:26.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.006+0000 7ff9cc336700 1 -- 172.21.15.29:0/4031719202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c419ab20 con 0x7ff9c41004e0 2024-04-27T02:03:26.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.006+0000 7ff9cc336700 1 -- 172.21.15.29:0/4031719202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c419b010 con 0x7ff9c41004e0 2024-04-27T02:03:26.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.007+0000 7ff9cc336700 1 -- 172.21.15.29:0/4031719202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff9c410d670 con 0x7ff9c41004e0 2024-04-27T02:03:26.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.007+0000 7ff9bb7fe700 1 -- 172.21.15.29:0/4031719202 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7ff9c000f7f0 con 0x7ff9c41004e0 2024-04-27T02:03:26.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.008+0000 7ff9bb7fe700 1 --2- 172.21.15.29:0/4031719202 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ff9b006c680 0x7ff9b006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:26.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.008+0000 7ff9bb7fe700 1 -- 172.21.15.29:0/4031719202 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff9c008ca20 con 0x7ff9c41004e0 2024-04-27T02:03:26.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.008+0000 7ff9c98d1700 1 --2- 172.21.15.29:0/4031719202 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ff9b006c680 0x7ff9b006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:26.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.008+0000 7ff9bb7fe700 1 --2- 172.21.15.29:0/4031719202 >> [v2:172.21.15.29:6810/3705295680,v1:172.21.15.29:6811/3705295680] conn(0x7ff9b0072350 0x7ff9b0074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:26.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.008+0000 7ff9bb7fe700 1 -- 172.21.15.29:0/4031719202 --> [v2:172.21.15.29:6810/3705295680,v1:172.21.15.29:6811/3705295680] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9b0074e10 con 0x7ff9b0072350 2024-04-27T02:03:26.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.008+0000 7ff9bb7fe700 1 -- 172.21.15.29:0/4031719202 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9c00912a0 con 0x7ff9c41004e0 2024-04-27T02:03:26.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.008+0000 7ff9ca8d3700 1 --2- 172.21.15.29:0/4031719202 >> [v2:172.21.15.29:6810/3705295680,v1:172.21.15.29:6811/3705295680] conn(0x7ff9b0072350 0x7ff9b0074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:26.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.008+0000 7ff9c98d1700 1 --2- 172.21.15.29:0/4031719202 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ff9b006c680 0x7ff9b006eb30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff9b4005fd0 tx=0x7ff9b4005ee0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:26.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.008+0000 7ff9ca8d3700 1 --2- 172.21.15.29:0/4031719202 >> [v2:172.21.15.29:6810/3705295680,v1:172.21.15.29:6811/3705295680] conn(0x7ff9b0072350 0x7ff9b0074760 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:26.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.009+0000 7ff9bb7fe700 1 -- 172.21.15.29:0/4031719202 <== osd.1 v2:172.21.15.29:6810/3705295680 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7ff9b0074e10 con 0x7ff9b0072350 2024-04-27T02:03:27.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.025+0000 7ff9cc336700 1 -- 172.21.15.29:0/4031719202 --> [v2:172.21.15.29:6810/3705295680,v1:172.21.15.29:6811/3705295680] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff9c404f2a0 con 0x7ff9b0072350 2024-04-27T02:03:27.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.025+0000 7ff9bb7fe700 1 -- 172.21.15.29:0/4031719202 <== osd.1 v2:172.21.15.29:6810/3705295680 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff9c404f2a0 con 0x7ff9b0072350 2024-04-27T02:03:27.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.026+0000 7ff9cc336700 1 -- 172.21.15.29:0/4031719202 >> [v2:172.21.15.29:6810/3705295680,v1:172.21.15.29:6811/3705295680] conn(0x7ff9b0072350 msgr2=0x7ff9b0074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:27.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.026+0000 7ff9cc336700 1 --2- 172.21.15.29:0/4031719202 >> [v2:172.21.15.29:6810/3705295680,v1:172.21.15.29:6811/3705295680] conn(0x7ff9b0072350 0x7ff9b0074760 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:27.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.026+0000 7ff9cc336700 1 -- 172.21.15.29:0/4031719202 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ff9b006c680 msgr2=0x7ff9b006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:27.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.026+0000 7ff9cc336700 1 --2- 172.21.15.29:0/4031719202 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ff9b006c680 0x7ff9b006eb30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff9b4005fd0 tx=0x7ff9b4005ee0 comp rx=0 tx=0).stop 2024-04-27T02:03:27.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.026+0000 7ff9cc336700 1 -- 172.21.15.29:0/4031719202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9c41004e0 msgr2=0x7ff9c4195ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:27.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.026+0000 7ff9cc336700 1 --2- 172.21.15.29:0/4031719202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9c41004e0 0x7ff9c4195ef0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0000c00 tx=0x7ff9c000be20 comp rx=0 tx=0).stop 2024-04-27T02:03:27.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.026+0000 7ff9cc336700 1 -- 172.21.15.29:0/4031719202 shutdown_connections 2024-04-27T02:03:27.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.026+0000 7ff9cc336700 1 --2- 172.21.15.29:0/4031719202 >> [v2:172.21.15.29:6810/3705295680,v1:172.21.15.29:6811/3705295680] conn(0x7ff9b0072350 0x7ff9b0074760 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:27.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.026+0000 7ff9cc336700 1 --2- 172.21.15.29:0/4031719202 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ff9b006c680 0x7ff9b006eb30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:27.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.026+0000 7ff9cc336700 1 --2- 172.21.15.29:0/4031719202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9c41004e0 0x7ff9c4195ef0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:27.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.026+0000 7ff9cc336700 1 --2- 172.21.15.29:0/4031719202 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff9c4100e80 0x7ff9c4196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:27.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.026+0000 7ff9cc336700 1 -- 172.21.15.29:0/4031719202 >> 172.21.15.29:0/4031719202 conn(0x7ff9c40fbda0 msgr2=0x7ff9c40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:27.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.026+0000 7ff9cc336700 1 -- 172.21.15.29:0/4031719202 shutdown_connections 2024-04-27T02:03:27.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:27.026+0000 7ff9cc336700 1 -- 172.21.15.29:0/4031719202 wait complete. 2024-04-27T02:03:27.018 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:26 smithi029 ceph-mon[26769]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:27.022 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:03:27.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:26 smithi052 ceph-mon[32586]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:27.686 INFO:teuthology.orchestra.run.smithi029.stdout:60129542177 2024-04-27T02:03:27.687 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-27T02:03:28.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:28 smithi029 ceph-mon[26769]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:29.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:28 smithi052 ceph-mon[32586]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:29.558 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:03:29.559 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:03:29.564 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:03:30.966 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:03:31.161 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:30 smithi029 ceph-mon[26769]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:31.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:30 smithi052 ceph-mon[32586]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:32.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.896+0000 7fd6bb141700 1 -- 172.21.15.29:0/886491267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6b4101420 msgr2=0x7fd6b41017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:32.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.896+0000 7fd6bb141700 1 --2- 172.21.15.29:0/886491267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6b4101420 0x7fd6b41017f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd69c009910 tx=0x7fd69c009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:32.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.896+0000 7fd6bb141700 1 -- 172.21.15.29:0/886491267 shutdown_connections 2024-04-27T02:03:32.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.896+0000 7fd6bb141700 1 --2- 172.21.15.29:0/886491267 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6b4101d30 0x7fd6b410a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:32.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.896+0000 7fd6bb141700 1 --2- 172.21.15.29:0/886491267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6b4101420 0x7fd6b41017f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:32.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.896+0000 7fd6bb141700 1 -- 172.21.15.29:0/886491267 >> 172.21.15.29:0/886491267 conn(0x7fd6b40fac60 msgr2=0x7fd6b40fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:32.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.896+0000 7fd6bb141700 1 -- 172.21.15.29:0/886491267 shutdown_connections 2024-04-27T02:03:32.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.896+0000 7fd6bb141700 1 -- 172.21.15.29:0/886491267 wait complete. 2024-04-27T02:03:32.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.896+0000 7fd6bb141700 1 Processor -- start 2024-04-27T02:03:32.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.897+0000 7fd6bb141700 1 -- start start 2024-04-27T02:03:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.897+0000 7fd6bb141700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6b4101420 0x7fd6b40ff080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.897+0000 7fd6bb141700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6b4101d30 0x7fd6b40ff5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.897+0000 7fd6bb141700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b40ffb00 con 0x7fd6b4101d30 2024-04-27T02:03:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.897+0000 7fd6bb141700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b40ffc40 con 0x7fd6b4101420 2024-04-27T02:03:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.897+0000 7fd6b3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6b4101d30 0x7fd6b40ff5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.897+0000 7fd6b3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6b4101d30 0x7fd6b40ff5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33122/0 (socket says 172.21.15.29:33122) 2024-04-27T02:03:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.897+0000 7fd6b3fff700 1 -- 172.21.15.29:0/4012712802 learned_addr learned my addr 172.21.15.29:0/4012712802 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:03:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.897+0000 7fd6b8edd700 1 --2- 172.21.15.29:0/4012712802 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6b4101420 0x7fd6b40ff080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.897+0000 7fd6b3fff700 1 -- 172.21.15.29:0/4012712802 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6b4101420 msgr2=0x7fd6b40ff080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.897+0000 7fd6b3fff700 1 --2- 172.21.15.29:0/4012712802 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6b4101420 0x7fd6b40ff080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.897+0000 7fd6b3fff700 1 -- 172.21.15.29:0/4012712802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69c0095f0 con 0x7fd6b4101d30 2024-04-27T02:03:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.897+0000 7fd6b3fff700 1 --2- 172.21.15.29:0/4012712802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6b4101d30 0x7fd6b40ff5c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400d8d0 tx=0x7fd6a400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.898+0000 7fd6b1ffb700 1 -- 172.21.15.29:0/4012712802 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6a40098a0 con 0x7fd6b4101d30 2024-04-27T02:03:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.898+0000 7fd6b1ffb700 1 -- 172.21.15.29:0/4012712802 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd6a4010460 con 0x7fd6b4101d30 2024-04-27T02:03:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.898+0000 7fd6b1ffb700 1 -- 172.21.15.29:0/4012712802 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6a400f5f0 con 0x7fd6b4101d30 2024-04-27T02:03:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.898+0000 7fd6bb141700 1 -- 172.21.15.29:0/4012712802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b41a2070 con 0x7fd6b4101d30 2024-04-27T02:03:32.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.898+0000 7fd6bb141700 1 -- 172.21.15.29:0/4012712802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b41a2420 con 0x7fd6b4101d30 2024-04-27T02:03:32.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.899+0000 7fd6bb141700 1 -- 172.21.15.29:0/4012712802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6b4107900 con 0x7fd6b4101d30 2024-04-27T02:03:32.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.899+0000 7fd6b1ffb700 1 -- 172.21.15.29:0/4012712802 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fd6a4009a00 con 0x7fd6b4101d30 2024-04-27T02:03:32.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.900+0000 7fd6b1ffb700 1 --2- 172.21.15.29:0/4012712802 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fd6a006c680 0x7fd6a006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:32.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.900+0000 7fd6b8edd700 1 --2- 172.21.15.29:0/4012712802 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fd6a006c680 0x7fd6a006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:32.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.900+0000 7fd6b1ffb700 1 -- 172.21.15.29:0/4012712802 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd6a408b420 con 0x7fd6b4101d30 2024-04-27T02:03:32.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.900+0000 7fd6b8edd700 1 --2- 172.21.15.29:0/4012712802 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fd6a006c680 0x7fd6a006eb30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd69c006010 tx=0x7fd69c01a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:32.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:32.904+0000 7fd6b1ffb700 1 -- 172.21.15.29:0/4012712802 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd6a4055e40 con 0x7fd6b4101d30 2024-04-27T02:03:33.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:33.108+0000 7fd6bb141700 1 -- 172.21.15.29:0/4012712802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd6b404f2a0 con 0x7fd6b4101d30 2024-04-27T02:03:33.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:33.108+0000 7fd6b1ffb700 1 -- 172.21.15.29:0/4012712802 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd6a4059460 con 0x7fd6b4101d30 2024-04-27T02:03:33.090 INFO:teuthology.orchestra.run.smithi029.stdout:167503724556 2024-04-27T02:03:33.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:33.110+0000 7fd6bb141700 1 -- 172.21.15.29:0/4012712802 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fd6a006c680 msgr2=0x7fd6a006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:33.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:33.110+0000 7fd6bb141700 1 --2- 172.21.15.29:0/4012712802 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fd6a006c680 0x7fd6a006eb30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd69c006010 tx=0x7fd69c01a040 comp rx=0 tx=0).stop 2024-04-27T02:03:33.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:33.110+0000 7fd6bb141700 1 -- 172.21.15.29:0/4012712802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6b4101d30 msgr2=0x7fd6b40ff5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:33.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:33.110+0000 7fd6bb141700 1 --2- 172.21.15.29:0/4012712802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6b4101d30 0x7fd6b40ff5c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400d8d0 tx=0x7fd6a400dbe0 comp rx=0 tx=0).stop 2024-04-27T02:03:33.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:33.110+0000 7fd6bb141700 1 -- 172.21.15.29:0/4012712802 shutdown_connections 2024-04-27T02:03:33.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:33.110+0000 7fd6bb141700 1 --2- 172.21.15.29:0/4012712802 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fd6a006c680 0x7fd6a006eb30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:33.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:33.110+0000 7fd6bb141700 1 --2- 172.21.15.29:0/4012712802 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6b4101420 0x7fd6b40ff080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:33.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:33.110+0000 7fd6bb141700 1 --2- 172.21.15.29:0/4012712802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6b4101d30 0x7fd6b40ff5c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:33.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:33.110+0000 7fd6bb141700 1 -- 172.21.15.29:0/4012712802 >> 172.21.15.29:0/4012712802 conn(0x7fd6b40fac60 msgr2=0x7fd6b4104a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:33.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:33.110+0000 7fd6bb141700 1 -- 172.21.15.29:0/4012712802 shutdown_connections 2024-04-27T02:03:33.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:33.110+0000 7fd6bb141700 1 -- 172.21.15.29:0/4012712802 wait complete. 2024-04-27T02:03:33.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:32 smithi029 ceph-mon[26769]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:33.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:32 smithi052 ceph-mon[32586]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:34.180 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:33 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/4012712802' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-27T02:03:34.205 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724553 got 167503724556 for osd.5 2024-04-27T02:03:34.205 DEBUG:teuthology.parallel:result is None 2024-04-27T02:03:34.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:33 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/4012712802' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-27T02:03:35.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:34 smithi029 ceph-mon[26769]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:35.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:34 smithi052 ceph-mon[32586]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:35.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.574+0000 7f03cf980700 1 -- 172.21.15.29:0/253179959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03c80ff3d0 msgr2=0x7f03c80ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:35.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.574+0000 7f03cf980700 1 --2- 172.21.15.29:0/253179959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03c80ff3d0 0x7f03c80ff7a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009910 tx=0x7f03c4009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:35.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.574+0000 7f03cf980700 1 -- 172.21.15.29:0/253179959 shutdown_connections 2024-04-27T02:03:35.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.574+0000 7f03cf980700 1 --2- 172.21.15.29:0/253179959 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03c80ffce0 0x7f03c810a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:35.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.574+0000 7f03cf980700 1 --2- 172.21.15.29:0/253179959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03c80ff3d0 0x7f03c80ff7a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:35.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.574+0000 7f03cf980700 1 -- 172.21.15.29:0/253179959 >> 172.21.15.29:0/253179959 conn(0x7f03c8074930 msgr2=0x7f03c8074d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:35.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.574+0000 7f03cf980700 1 -- 172.21.15.29:0/253179959 shutdown_connections 2024-04-27T02:03:35.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.574+0000 7f03cf980700 1 -- 172.21.15.29:0/253179959 wait complete. 2024-04-27T02:03:35.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.575+0000 7f03cf980700 1 Processor -- start 2024-04-27T02:03:35.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.575+0000 7f03cf980700 1 -- start start 2024-04-27T02:03:35.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.575+0000 7f03cf980700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03c80ff3d0 0x7f03c81005b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:35.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.575+0000 7f03cf980700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03c80ffce0 0x7f03c8100af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:35.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.575+0000 7f03cf980700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c8101030 con 0x7f03c80ff3d0 2024-04-27T02:03:35.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.575+0000 7f03cf980700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c8101170 con 0x7f03c80ffce0 2024-04-27T02:03:35.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.575+0000 7f03cd71c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03c80ff3d0 0x7f03c81005b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:35.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.575+0000 7f03cd71c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03c80ff3d0 0x7f03c81005b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33130/0 (socket says 172.21.15.29:33130) 2024-04-27T02:03:35.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.575+0000 7f03cd71c700 1 -- 172.21.15.29:0/2061344578 learned_addr learned my addr 172.21.15.29:0/2061344578 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:03:35.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.576+0000 7f03ccf1b700 1 --2- 172.21.15.29:0/2061344578 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03c80ffce0 0x7f03c8100af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:35.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.576+0000 7f03cd71c700 1 -- 172.21.15.29:0/2061344578 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03c80ffce0 msgr2=0x7f03c8100af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:35.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.576+0000 7f03cd71c700 1 --2- 172.21.15.29:0/2061344578 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03c80ffce0 0x7f03c8100af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:35.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.576+0000 7f03cd71c700 1 -- 172.21.15.29:0/2061344578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c40095f0 con 0x7f03c80ff3d0 2024-04-27T02:03:35.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.576+0000 7f03cd71c700 1 --2- 172.21.15.29:0/2061344578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03c80ff3d0 0x7f03c81005b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f03c4006010 tx=0x7f03c400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:35.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.576+0000 7f03be7fc700 1 -- 172.21.15.29:0/2061344578 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03c401d070 con 0x7f03c80ff3d0 2024-04-27T02:03:35.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.576+0000 7f03be7fc700 1 -- 172.21.15.29:0/2061344578 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f03c400f460 con 0x7f03c80ff3d0 2024-04-27T02:03:35.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.576+0000 7f03cf980700 1 -- 172.21.15.29:0/2061344578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c81013f0 con 0x7f03c80ff3d0 2024-04-27T02:03:35.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.576+0000 7f03cf980700 1 -- 172.21.15.29:0/2061344578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c81a2400 con 0x7f03c80ff3d0 2024-04-27T02:03:35.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.576+0000 7f03be7fc700 1 -- 172.21.15.29:0/2061344578 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03c4017630 con 0x7f03c80ff3d0 2024-04-27T02:03:35.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.577+0000 7f03cf980700 1 -- 172.21.15.29:0/2061344578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c8107e50 con 0x7f03c80ff3d0 2024-04-27T02:03:35.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.578+0000 7f03be7fc700 1 -- 172.21.15.29:0/2061344578 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f03c400f5d0 con 0x7f03c80ff3d0 2024-04-27T02:03:35.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.578+0000 7f03be7fc700 1 --2- 172.21.15.29:0/2061344578 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f03b406c680 0x7f03b406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:35.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.579+0000 7f03be7fc700 1 -- 172.21.15.29:0/2061344578 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f03c408c950 con 0x7f03c80ff3d0 2024-04-27T02:03:35.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.579+0000 7f03ccf1b700 1 --2- 172.21.15.29:0/2061344578 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f03b406c680 0x7f03b406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:35.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.579+0000 7f03ccf1b700 1 --2- 172.21.15.29:0/2061344578 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f03b406c680 0x7f03b406eb30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f03c8101da0 tx=0x7f03b800b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:35.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.583+0000 7f03be7fc700 1 -- 172.21.15.29:0/2061344578 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f03c4057370 con 0x7f03c80ff3d0 2024-04-27T02:03:35.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.789+0000 7f03cf980700 1 -- 172.21.15.29:0/2061344578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f03c8101ae0 con 0x7f03c80ff3d0 2024-04-27T02:03:35.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.790+0000 7f03be7fc700 1 -- 172.21.15.29:0/2061344578 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f03c405a990 con 0x7f03c80ff3d0 2024-04-27T02:03:35.771 INFO:teuthology.orchestra.run.smithi029.stdout:146028888081 2024-04-27T02:03:35.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.792+0000 7f03cf980700 1 -- 172.21.15.29:0/2061344578 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f03b406c680 msgr2=0x7f03b406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:35.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.792+0000 7f03cf980700 1 --2- 172.21.15.29:0/2061344578 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f03b406c680 0x7f03b406eb30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f03c8101da0 tx=0x7f03b800b410 comp rx=0 tx=0).stop 2024-04-27T02:03:35.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.792+0000 7f03cf980700 1 -- 172.21.15.29:0/2061344578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03c80ff3d0 msgr2=0x7f03c81005b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:35.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.792+0000 7f03cf980700 1 --2- 172.21.15.29:0/2061344578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03c80ff3d0 0x7f03c81005b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f03c4006010 tx=0x7f03c400bd70 comp rx=0 tx=0).stop 2024-04-27T02:03:35.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.792+0000 7f03cf980700 1 -- 172.21.15.29:0/2061344578 shutdown_connections 2024-04-27T02:03:35.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.792+0000 7f03cf980700 1 --2- 172.21.15.29:0/2061344578 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f03b406c680 0x7f03b406eb30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:35.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.792+0000 7f03cf980700 1 --2- 172.21.15.29:0/2061344578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03c80ff3d0 0x7f03c81005b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:35.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.792+0000 7f03cf980700 1 --2- 172.21.15.29:0/2061344578 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03c80ffce0 0x7f03c8100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:35.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.792+0000 7f03cf980700 1 -- 172.21.15.29:0/2061344578 >> 172.21.15.29:0/2061344578 conn(0x7f03c8074930 msgr2=0x7f03c8104f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:35.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.792+0000 7f03cf980700 1 -- 172.21.15.29:0/2061344578 shutdown_connections 2024-04-27T02:03:35.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:35.792+0000 7f03cf980700 1 -- 172.21.15.29:0/2061344578 wait complete. 2024-04-27T02:03:36.147 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:35 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/2061344578' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-27T02:03:36.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:35 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/2061344578' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-27T02:03:36.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.797+0000 7fa761f24700 1 -- 172.21.15.29:0/2962354463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa75c101420 msgr2=0x7fa75c1017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:36.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.797+0000 7fa761f24700 1 --2- 172.21.15.29:0/2962354463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa75c101420 0x7fa75c1017f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009910 tx=0x7fa74c009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:36.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.797+0000 7fa761f24700 1 -- 172.21.15.29:0/2962354463 shutdown_connections 2024-04-27T02:03:36.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.797+0000 7fa761f24700 1 --2- 172.21.15.29:0/2962354463 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa75c101d30 0x7fa75c10a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:36.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.797+0000 7fa761f24700 1 --2- 172.21.15.29:0/2962354463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa75c101420 0x7fa75c1017f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:36.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.797+0000 7fa761f24700 1 -- 172.21.15.29:0/2962354463 >> 172.21.15.29:0/2962354463 conn(0x7fa75c0fac80 msgr2=0x7fa75c0fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:36.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.797+0000 7fa761f24700 1 -- 172.21.15.29:0/2962354463 shutdown_connections 2024-04-27T02:03:36.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.797+0000 7fa761f24700 1 -- 172.21.15.29:0/2962354463 wait complete. 2024-04-27T02:03:36.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.797+0000 7fa761f24700 1 Processor -- start 2024-04-27T02:03:36.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.798+0000 7fa761f24700 1 -- start start 2024-04-27T02:03:36.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.798+0000 7fa761f24700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa75c101420 0x7fa75c195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:36.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.798+0000 7fa761f24700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa75c101d30 0x7fa75c1963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:36.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.798+0000 7fa761f24700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c196a90 con 0x7fa75c101d30 2024-04-27T02:03:36.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.798+0000 7fa761f24700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c19a820 con 0x7fa75c101420 2024-04-27T02:03:36.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.798+0000 7fa75affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa75c101d30 0x7fa75c1963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:36.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.798+0000 7fa75affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa75c101d30 0x7fa75c1963b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55170/0 (socket says 172.21.15.29:55170) 2024-04-27T02:03:36.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.798+0000 7fa75affd700 1 -- 172.21.15.29:0/2537836871 learned_addr learned my addr 172.21.15.29:0/2537836871 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:03:36.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.798+0000 7fa75b7fe700 1 --2- 172.21.15.29:0/2537836871 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa75c101420 0x7fa75c195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:36.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.798+0000 7fa75affd700 1 -- 172.21.15.29:0/2537836871 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa75c101420 msgr2=0x7fa75c195e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:36.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.798+0000 7fa75affd700 1 --2- 172.21.15.29:0/2537836871 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa75c101420 0x7fa75c195e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:36.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.798+0000 7fa75affd700 1 -- 172.21.15.29:0/2537836871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74c0095f0 con 0x7fa75c101d30 2024-04-27T02:03:36.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.799+0000 7fa75affd700 1 --2- 172.21.15.29:0/2537836871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa75c101d30 0x7fa75c1963b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa75000d8d0 tx=0x7fa75000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:36.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.799+0000 7fa758ff9700 1 -- 172.21.15.29:0/2537836871 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7500098a0 con 0x7fa75c101d30 2024-04-27T02:03:36.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.799+0000 7fa761f24700 1 -- 172.21.15.29:0/2537836871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa75c19ab00 con 0x7fa75c101d30 2024-04-27T02:03:36.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.799+0000 7fa758ff9700 1 -- 172.21.15.29:0/2537836871 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa750010460 con 0x7fa75c101d30 2024-04-27T02:03:36.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.799+0000 7fa758ff9700 1 -- 172.21.15.29:0/2537836871 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa75000f5f0 con 0x7fa75c101d30 2024-04-27T02:03:36.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.799+0000 7fa761f24700 1 -- 172.21.15.29:0/2537836871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa75c19b050 con 0x7fa75c101d30 2024-04-27T02:03:36.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.800+0000 7fa761f24700 1 -- 172.21.15.29:0/2537836871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa75c107900 con 0x7fa75c101d30 2024-04-27T02:03:36.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.800+0000 7fa758ff9700 1 -- 172.21.15.29:0/2537836871 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fa750009a00 con 0x7fa75c101d30 2024-04-27T02:03:36.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.801+0000 7fa758ff9700 1 --2- 172.21.15.29:0/2537836871 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa74406c680 0x7fa74406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:36.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.801+0000 7fa758ff9700 1 -- 172.21.15.29:0/2537836871 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa75008b420 con 0x7fa75c101d30 2024-04-27T02:03:36.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.801+0000 7fa75b7fe700 1 --2- 172.21.15.29:0/2537836871 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa74406c680 0x7fa74406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:36.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.801+0000 7fa75b7fe700 1 --2- 172.21.15.29:0/2537836871 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa74406c680 0x7fa74406eb30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa74c00b5c0 tx=0x7fa74c01a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:36.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:36.805+0000 7fa758ff9700 1 -- 172.21.15.29:0/2537836871 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa750055e40 con 0x7fa75c101d30 2024-04-27T02:03:36.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:36 smithi029 ceph-mon[26769]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:36.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.010+0000 7fa761f24700 1 -- 172.21.15.29:0/2537836871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fa75c19b270 con 0x7fa75c101d30 2024-04-27T02:03:36.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.010+0000 7fa758ff9700 1 -- 172.21.15.29:0/2537836871 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa750059460 con 0x7fa75c101d30 2024-04-27T02:03:36.992 INFO:teuthology.orchestra.run.smithi029.stdout:111669149718 2024-04-27T02:03:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.012+0000 7fa761f24700 1 -- 172.21.15.29:0/2537836871 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa74406c680 msgr2=0x7fa74406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.012+0000 7fa761f24700 1 --2- 172.21.15.29:0/2537836871 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa74406c680 0x7fa74406eb30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa74c00b5c0 tx=0x7fa74c01a040 comp rx=0 tx=0).stop 2024-04-27T02:03:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.012+0000 7fa761f24700 1 -- 172.21.15.29:0/2537836871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa75c101d30 msgr2=0x7fa75c1963b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.012+0000 7fa761f24700 1 --2- 172.21.15.29:0/2537836871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa75c101d30 0x7fa75c1963b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa75000d8d0 tx=0x7fa75000dbe0 comp rx=0 tx=0).stop 2024-04-27T02:03:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.012+0000 7fa761f24700 1 -- 172.21.15.29:0/2537836871 shutdown_connections 2024-04-27T02:03:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.012+0000 7fa761f24700 1 --2- 172.21.15.29:0/2537836871 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa74406c680 0x7fa74406eb30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.012+0000 7fa761f24700 1 --2- 172.21.15.29:0/2537836871 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa75c101420 0x7fa75c195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.012+0000 7fa761f24700 1 --2- 172.21.15.29:0/2537836871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa75c101d30 0x7fa75c1963b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.012+0000 7fa761f24700 1 -- 172.21.15.29:0/2537836871 >> 172.21.15.29:0/2537836871 conn(0x7fa75c0fac80 msgr2=0x7fa75c104a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.012+0000 7fa761f24700 1 -- 172.21.15.29:0/2537836871 shutdown_connections 2024-04-27T02:03:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.012+0000 7fa761f24700 1 -- 172.21.15.29:0/2537836871 wait complete. 2024-04-27T02:03:37.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:36 smithi052 ceph-mon[32586]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.382+0000 7ff8440c5700 1 -- 172.21.15.29:0/4112956072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff83c104130 msgr2=0x7ff83c104500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.382+0000 7ff8440c5700 1 --2- 172.21.15.29:0/4112956072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff83c104130 0x7ff83c104500 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff838009910 tx=0x7ff838009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.382+0000 7ff8440c5700 1 -- 172.21.15.29:0/4112956072 shutdown_connections 2024-04-27T02:03:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.382+0000 7ff8440c5700 1 --2- 172.21.15.29:0/4112956072 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff83c0ffbd0 0x7ff83c100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.382+0000 7ff8440c5700 1 --2- 172.21.15.29:0/4112956072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff83c104130 0x7ff83c104500 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.382+0000 7ff8440c5700 1 -- 172.21.15.29:0/4112956072 >> 172.21.15.29:0/4112956072 conn(0x7ff83c075130 msgr2=0x7ff83c075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.382+0000 7ff8440c5700 1 -- 172.21.15.29:0/4112956072 shutdown_connections 2024-04-27T02:03:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.382+0000 7ff8440c5700 1 -- 172.21.15.29:0/4112956072 wait complete. 2024-04-27T02:03:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.382+0000 7ff8440c5700 1 Processor -- start 2024-04-27T02:03:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.382+0000 7ff8440c5700 1 -- start start 2024-04-27T02:03:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.383+0000 7ff8440c5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff83c0ffbd0 0x7ff83c194d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.383+0000 7ff8440c5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff83c104130 0x7ff83c195280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.383+0000 7ff8440c5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c195960 con 0x7ff83c0ffbd0 2024-04-27T02:03:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.383+0000 7ff8440c5700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c1996f0 con 0x7ff83c104130 2024-04-27T02:03:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.383+0000 7ff841e61700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff83c0ffbd0 0x7ff83c194d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.383+0000 7ff841e61700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff83c0ffbd0 0x7ff83c194d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55186/0 (socket says 172.21.15.29:55186) 2024-04-27T02:03:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.383+0000 7ff841e61700 1 -- 172.21.15.29:0/4155718008 learned_addr learned my addr 172.21.15.29:0/4155718008 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:03:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.383+0000 7ff841660700 1 --2- 172.21.15.29:0/4155718008 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff83c104130 0x7ff83c195280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.383+0000 7ff841e61700 1 -- 172.21.15.29:0/4155718008 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff83c104130 msgr2=0x7ff83c195280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.383+0000 7ff841e61700 1 --2- 172.21.15.29:0/4155718008 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff83c104130 0x7ff83c195280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.383+0000 7ff841e61700 1 -- 172.21.15.29:0/4155718008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8380095f0 con 0x7ff83c0ffbd0 2024-04-27T02:03:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.383+0000 7ff841e61700 1 --2- 172.21.15.29:0/4155718008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff83c0ffbd0 0x7ff83c194d40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff83800bd70 tx=0x7ff83800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.383+0000 7ff832ffd700 1 -- 172.21.15.29:0/4155718008 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff83801d070 con 0x7ff83c0ffbd0 2024-04-27T02:03:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.384+0000 7ff832ffd700 1 -- 172.21.15.29:0/4155718008 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff83800f460 con 0x7ff83c0ffbd0 2024-04-27T02:03:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.384+0000 7ff8440c5700 1 -- 172.21.15.29:0/4155718008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff83c199970 con 0x7ff83c0ffbd0 2024-04-27T02:03:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.384+0000 7ff832ffd700 1 -- 172.21.15.29:0/4155718008 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff838017630 con 0x7ff83c0ffbd0 2024-04-27T02:03:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.384+0000 7ff8440c5700 1 -- 172.21.15.29:0/4155718008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff83c199e60 con 0x7ff83c0ffbd0 2024-04-27T02:03:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.385+0000 7ff8440c5700 1 -- 172.21.15.29:0/4155718008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff83c1076a0 con 0x7ff83c0ffbd0 2024-04-27T02:03:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.385+0000 7ff832ffd700 1 -- 172.21.15.29:0/4155718008 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7ff838003a60 con 0x7ff83c0ffbd0 2024-04-27T02:03:37.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.386+0000 7ff832ffd700 1 --2- 172.21.15.29:0/4155718008 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ff82806c680 0x7ff82806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:37.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.386+0000 7ff832ffd700 1 -- 172.21.15.29:0/4155718008 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff83808c970 con 0x7ff83c0ffbd0 2024-04-27T02:03:37.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.386+0000 7ff841660700 1 --2- 172.21.15.29:0/4155718008 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ff82806c680 0x7ff82806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:37.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.386+0000 7ff841660700 1 --2- 172.21.15.29:0/4155718008 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ff82806c680 0x7ff82806eb30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff83c196360 tx=0x7ff82c00b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:37.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.390+0000 7ff832ffd700 1 -- 172.21.15.29:0/4155718008 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff838057390 con 0x7ff83c0ffbd0 2024-04-27T02:03:37.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.592+0000 7ff8440c5700 1 -- 172.21.15.29:0/4155718008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ff83c1961a0 con 0x7ff83c0ffbd0 2024-04-27T02:03:37.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.593+0000 7ff832ffd700 1 -- 172.21.15.29:0/4155718008 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff83805a9b0 con 0x7ff83c0ffbd0 2024-04-27T02:03:37.574 INFO:teuthology.orchestra.run.smithi029.stdout:38654705704 2024-04-27T02:03:37.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.595+0000 7ff8440c5700 1 -- 172.21.15.29:0/4155718008 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ff82806c680 msgr2=0x7ff82806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:37.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.595+0000 7ff8440c5700 1 --2- 172.21.15.29:0/4155718008 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ff82806c680 0x7ff82806eb30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff83c196360 tx=0x7ff82c00b410 comp rx=0 tx=0).stop 2024-04-27T02:03:37.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.595+0000 7ff8440c5700 1 -- 172.21.15.29:0/4155718008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff83c0ffbd0 msgr2=0x7ff83c194d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:37.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.595+0000 7ff8440c5700 1 --2- 172.21.15.29:0/4155718008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff83c0ffbd0 0x7ff83c194d40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff83800bd70 tx=0x7ff83800bda0 comp rx=0 tx=0).stop 2024-04-27T02:03:37.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.595+0000 7ff8440c5700 1 -- 172.21.15.29:0/4155718008 shutdown_connections 2024-04-27T02:03:37.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.595+0000 7ff8440c5700 1 --2- 172.21.15.29:0/4155718008 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ff82806c680 0x7ff82806eb30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:37.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.595+0000 7ff8440c5700 1 --2- 172.21.15.29:0/4155718008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff83c0ffbd0 0x7ff83c194d40 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:37.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.595+0000 7ff8440c5700 1 --2- 172.21.15.29:0/4155718008 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff83c104130 0x7ff83c195280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:37.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.595+0000 7ff8440c5700 1 -- 172.21.15.29:0/4155718008 >> 172.21.15.29:0/4155718008 conn(0x7ff83c075130 msgr2=0x7ff83c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:37.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.595+0000 7ff8440c5700 1 -- 172.21.15.29:0/4155718008 shutdown_connections 2024-04-27T02:03:37.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:37.595+0000 7ff8440c5700 1 -- 172.21.15.29:0/4155718008 wait complete. 2024-04-27T02:03:37.922 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149716 got 111669149718 for osd.3 2024-04-27T02:03:37.922 DEBUG:teuthology.parallel:result is None 2024-04-27T02:03:38.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:37 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/2537836871' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-27T02:03:38.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:37 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/4155718008' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-27T02:03:38.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:37 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:03:38.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:37 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/2537836871' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-27T02:03:38.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:37 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/4155718008' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-27T02:03:38.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:37 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:03:38.546 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705700 got 38654705704 for osd.0 2024-04-27T02:03:38.546 DEBUG:teuthology.parallel:result is None 2024-04-27T02:03:38.988 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888079 got 146028888081 for osd.4 2024-04-27T02:03:38.988 DEBUG:teuthology.parallel:result is None 2024-04-27T02:03:39.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:38 smithi029 ceph-mon[26769]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:39.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:38 smithi052 ceph-mon[32586]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:39.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.545+0000 7f16de2be700 1 -- 172.21.15.29:0/1442721560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16d8104130 msgr2=0x7f16d8104500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:39.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.545+0000 7f16de2be700 1 --2- 172.21.15.29:0/1442721560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16d8104130 0x7f16d8104500 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f16c8009910 tx=0x7f16c8009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:39.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.546+0000 7f16de2be700 1 -- 172.21.15.29:0/1442721560 shutdown_connections 2024-04-27T02:03:39.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.546+0000 7f16de2be700 1 --2- 172.21.15.29:0/1442721560 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16d80ffbd0 0x7f16d8100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:39.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.546+0000 7f16de2be700 1 --2- 172.21.15.29:0/1442721560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16d8104130 0x7f16d8104500 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:39.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.546+0000 7f16de2be700 1 -- 172.21.15.29:0/1442721560 >> 172.21.15.29:0/1442721560 conn(0x7f16d8075130 msgr2=0x7f16d8075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:39.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.546+0000 7f16de2be700 1 -- 172.21.15.29:0/1442721560 shutdown_connections 2024-04-27T02:03:39.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.546+0000 7f16de2be700 1 -- 172.21.15.29:0/1442721560 wait complete. 2024-04-27T02:03:39.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.546+0000 7f16de2be700 1 Processor -- start 2024-04-27T02:03:39.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.546+0000 7f16de2be700 1 -- start start 2024-04-27T02:03:39.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.547+0000 7f16de2be700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16d80ffbd0 0x7f16d8196c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:39.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.547+0000 7f16de2be700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16d8104130 0x7f16d81971d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:39.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.547+0000 7f16de2be700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d81977f0 con 0x7f16d8104130 2024-04-27T02:03:39.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.547+0000 7f16de2be700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d8197960 con 0x7f16d80ffbd0 2024-04-27T02:03:39.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.547+0000 7f16d77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16d8104130 0x7f16d81971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:39.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.547+0000 7f16d77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16d8104130 0x7f16d81971d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55212/0 (socket says 172.21.15.29:55212) 2024-04-27T02:03:39.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.547+0000 7f16d77fe700 1 -- 172.21.15.29:0/1964314032 learned_addr learned my addr 172.21.15.29:0/1964314032 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:03:39.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.547+0000 7f16d7fff700 1 --2- 172.21.15.29:0/1964314032 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16d80ffbd0 0x7f16d8196c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:39.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.547+0000 7f16d77fe700 1 -- 172.21.15.29:0/1964314032 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16d80ffbd0 msgr2=0x7f16d8196c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:39.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.547+0000 7f16d77fe700 1 --2- 172.21.15.29:0/1964314032 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16d80ffbd0 0x7f16d8196c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:39.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.547+0000 7f16d77fe700 1 -- 172.21.15.29:0/1964314032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c80095f0 con 0x7f16d8104130 2024-04-27T02:03:39.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.548+0000 7f16d77fe700 1 --2- 172.21.15.29:0/1964314032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16d8104130 0x7f16d81971d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f16cc00c8a0 tx=0x7f16cc00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:39.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.548+0000 7f16d57fa700 1 -- 172.21.15.29:0/1964314032 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16cc0078e0 con 0x7f16d8104130 2024-04-27T02:03:39.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.548+0000 7f16d57fa700 1 -- 172.21.15.29:0/1964314032 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f16cc00f450 con 0x7f16d8104130 2024-04-27T02:03:39.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.548+0000 7f16de2be700 1 -- 172.21.15.29:0/1964314032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16d819fd00 con 0x7f16d8104130 2024-04-27T02:03:39.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.548+0000 7f16d57fa700 1 -- 172.21.15.29:0/1964314032 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16cc00e5e0 con 0x7f16d8104130 2024-04-27T02:03:39.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.548+0000 7f16de2be700 1 -- 172.21.15.29:0/1964314032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16d81a01f0 con 0x7f16d8104130 2024-04-27T02:03:39.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.549+0000 7f16de2be700 1 -- 172.21.15.29:0/1964314032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16d81076a0 con 0x7f16d8104130 2024-04-27T02:03:39.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.549+0000 7f16d57fa700 1 -- 172.21.15.29:0/1964314032 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f16cc00f5c0 con 0x7f16d8104130 2024-04-27T02:03:39.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.550+0000 7f16d57fa700 1 --2- 172.21.15.29:0/1964314032 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f16c006c680 0x7f16c006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:39.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.550+0000 7f16d57fa700 1 -- 172.21.15.29:0/1964314032 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f16cc08b4d0 con 0x7f16d8104130 2024-04-27T02:03:39.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.550+0000 7f16d7fff700 1 --2- 172.21.15.29:0/1964314032 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f16c006c680 0x7f16c006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:39.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.550+0000 7f16d7fff700 1 --2- 172.21.15.29:0/1964314032 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f16c006c680 0x7f16c006eb30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f16c8006010 tx=0x7f16c801a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:39.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.554+0000 7f16d57fa700 1 -- 172.21.15.29:0/1964314032 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f16cc055ef0 con 0x7f16d8104130 2024-04-27T02:03:39.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.749+0000 7f3f35868700 1 -- 172.21.15.29:0/647760218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f301004c0 msgr2=0x7f3f30100890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:39.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.749+0000 7f3f35868700 1 --2- 172.21.15.29:0/647760218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f301004c0 0x7f3f30100890 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009910 tx=0x7f3f24009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:39.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.749+0000 7f3f35868700 1 -- 172.21.15.29:0/647760218 shutdown_connections 2024-04-27T02:03:39.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.749+0000 7f3f35868700 1 --2- 172.21.15.29:0/647760218 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f30100e60 0x7f3f30104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:39.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.749+0000 7f3f35868700 1 --2- 172.21.15.29:0/647760218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f301004c0 0x7f3f30100890 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:39.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.749+0000 7f3f35868700 1 -- 172.21.15.29:0/647760218 >> 172.21.15.29:0/647760218 conn(0x7f3f300fbd80 msgr2=0x7f3f300fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:39.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.749+0000 7f3f35868700 1 -- 172.21.15.29:0/647760218 shutdown_connections 2024-04-27T02:03:39.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.749+0000 7f3f35868700 1 -- 172.21.15.29:0/647760218 wait complete. 2024-04-27T02:03:39.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.749+0000 7f3f35868700 1 Processor -- start 2024-04-27T02:03:39.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.750+0000 7f3f35868700 1 -- start start 2024-04-27T02:03:39.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.750+0000 7f3f35868700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f301004c0 0x7f3f30073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:39.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.750+0000 7f3f35868700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f30100e60 0x7f3f30073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:39.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.750+0000 7f3f35868700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f30073ad0 con 0x7f3f30100e60 2024-04-27T02:03:39.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.750+0000 7f3f35868700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f30073c10 con 0x7f3f301004c0 2024-04-27T02:03:39.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.750+0000 7f3f2e7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f30100e60 0x7f3f30073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:39.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.750+0000 7f3f2e7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f30100e60 0x7f3f30073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55232/0 (socket says 172.21.15.29:55232) 2024-04-27T02:03:39.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.750+0000 7f3f2e7fc700 1 -- 172.21.15.29:0/3063529452 learned_addr learned my addr 172.21.15.29:0/3063529452 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:03:39.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.750+0000 7f3f2effd700 1 --2- 172.21.15.29:0/3063529452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f301004c0 0x7f3f30073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:39.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.750+0000 7f3f2e7fc700 1 -- 172.21.15.29:0/3063529452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f301004c0 msgr2=0x7f3f30073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:39.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.750+0000 7f3f2e7fc700 1 --2- 172.21.15.29:0/3063529452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f301004c0 0x7f3f30073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:39.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.750+0000 7f3f2e7fc700 1 -- 172.21.15.29:0/3063529452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f240095f0 con 0x7f3f30100e60 2024-04-27T02:03:39.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.751+0000 7f3f2e7fc700 1 --2- 172.21.15.29:0/3063529452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f30100e60 0x7f3f30073590 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3f2000c8f0 tx=0x7f3f2000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:39.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.751+0000 7f3f34866700 1 -- 172.21.15.29:0/3063529452 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f200078a0 con 0x7f3f30100e60 2024-04-27T02:03:39.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.751+0000 7f3f34866700 1 -- 172.21.15.29:0/3063529452 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3f2000f450 con 0x7f3f30100e60 2024-04-27T02:03:39.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.751+0000 7f3f35868700 1 -- 172.21.15.29:0/3063529452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f301a2100 con 0x7f3f30100e60 2024-04-27T02:03:39.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.751+0000 7f3f34866700 1 -- 172.21.15.29:0/3063529452 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f20018610 con 0x7f3f30100e60 2024-04-27T02:03:39.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.751+0000 7f3f35868700 1 -- 172.21.15.29:0/3063529452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f301a2510 con 0x7f3f30100e60 2024-04-27T02:03:39.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.752+0000 7f3f35868700 1 -- 172.21.15.29:0/3063529452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f301087d0 con 0x7f3f30100e60 2024-04-27T02:03:39.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.753+0000 7f3f34866700 1 -- 172.21.15.29:0/3063529452 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f3f2000f5c0 con 0x7f3f30100e60 2024-04-27T02:03:39.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.753+0000 7f3f34866700 1 --2- 172.21.15.29:0/3063529452 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3f1806c680 0x7f3f1806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:39.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.754+0000 7f3f34866700 1 -- 172.21.15.29:0/3063529452 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f3f2008b570 con 0x7f3f30100e60 2024-04-27T02:03:39.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.754+0000 7f3f2effd700 1 --2- 172.21.15.29:0/3063529452 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3f1806c680 0x7f3f1806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:39.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.754+0000 7f3f2effd700 1 --2- 172.21.15.29:0/3063529452 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3f1806c680 0x7f3f1806eb30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3f24006010 tx=0x7f3f2401a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:39.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.758+0000 7f3f34866700 1 -- 172.21.15.29:0/3063529452 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3f20055ef0 con 0x7f3f30100e60 2024-04-27T02:03:39.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.760+0000 7f16de2be700 1 -- 172.21.15.29:0/1964314032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f16d804f2a0 con 0x7f16d8104130 2024-04-27T02:03:39.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.760+0000 7f16d57fa700 1 -- 172.21.15.29:0/1964314032 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f16cc059510 con 0x7f16d8104130 2024-04-27T02:03:39.742 INFO:teuthology.orchestra.run.smithi029.stdout:81604378653 2024-04-27T02:03:39.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.762+0000 7f16de2be700 1 -- 172.21.15.29:0/1964314032 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f16c006c680 msgr2=0x7f16c006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:39.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.762+0000 7f16de2be700 1 --2- 172.21.15.29:0/1964314032 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f16c006c680 0x7f16c006eb30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f16c8006010 tx=0x7f16c801a040 comp rx=0 tx=0).stop 2024-04-27T02:03:39.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.762+0000 7f16de2be700 1 -- 172.21.15.29:0/1964314032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16d8104130 msgr2=0x7f16d81971d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:39.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.762+0000 7f16de2be700 1 --2- 172.21.15.29:0/1964314032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16d8104130 0x7f16d81971d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f16cc00c8a0 tx=0x7f16cc00cbb0 comp rx=0 tx=0).stop 2024-04-27T02:03:39.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.763+0000 7f16de2be700 1 -- 172.21.15.29:0/1964314032 shutdown_connections 2024-04-27T02:03:39.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.763+0000 7f16de2be700 1 --2- 172.21.15.29:0/1964314032 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f16c006c680 0x7f16c006eb30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:39.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.763+0000 7f16de2be700 1 --2- 172.21.15.29:0/1964314032 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16d80ffbd0 0x7f16d8196c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:39.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.763+0000 7f16de2be700 1 --2- 172.21.15.29:0/1964314032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16d8104130 0x7f16d81971d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:39.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.763+0000 7f16de2be700 1 -- 172.21.15.29:0/1964314032 >> 172.21.15.29:0/1964314032 conn(0x7f16d8075130 msgr2=0x7f16d80fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:39.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.763+0000 7f16de2be700 1 -- 172.21.15.29:0/1964314032 shutdown_connections 2024-04-27T02:03:39.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.763+0000 7f16de2be700 1 -- 172.21.15.29:0/1964314032 wait complete. 2024-04-27T02:03:39.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.968+0000 7f3f35868700 1 -- 172.21.15.29:0/3063529452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3f3004f2a0 con 0x7f3f30100e60 2024-04-27T02:03:39.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.968+0000 7f3f34866700 1 -- 172.21.15.29:0/3063529452 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3f20059510 con 0x7f3f30100e60 2024-04-27T02:03:39.950 INFO:teuthology.orchestra.run.smithi029.stdout:60129542179 2024-04-27T02:03:39.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.970+0000 7f3f35868700 1 -- 172.21.15.29:0/3063529452 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3f1806c680 msgr2=0x7f3f1806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:39.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.970+0000 7f3f35868700 1 --2- 172.21.15.29:0/3063529452 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3f1806c680 0x7f3f1806eb30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3f24006010 tx=0x7f3f2401a040 comp rx=0 tx=0).stop 2024-04-27T02:03:39.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.970+0000 7f3f35868700 1 -- 172.21.15.29:0/3063529452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f30100e60 msgr2=0x7f3f30073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:39.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.970+0000 7f3f35868700 1 --2- 172.21.15.29:0/3063529452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f30100e60 0x7f3f30073590 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3f2000c8f0 tx=0x7f3f2000cc00 comp rx=0 tx=0).stop 2024-04-27T02:03:39.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.971+0000 7f3f35868700 1 -- 172.21.15.29:0/3063529452 shutdown_connections 2024-04-27T02:03:39.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.971+0000 7f3f35868700 1 --2- 172.21.15.29:0/3063529452 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3f1806c680 0x7f3f1806eb30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:39.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.971+0000 7f3f35868700 1 --2- 172.21.15.29:0/3063529452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f301004c0 0x7f3f30073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:39.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.971+0000 7f3f35868700 1 --2- 172.21.15.29:0/3063529452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f30100e60 0x7f3f30073590 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:39.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.971+0000 7f3f35868700 1 -- 172.21.15.29:0/3063529452 >> 172.21.15.29:0/3063529452 conn(0x7f3f300fbd80 msgr2=0x7f3f300fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:39.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.971+0000 7f3f35868700 1 -- 172.21.15.29:0/3063529452 shutdown_connections 2024-04-27T02:03:39.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:39.971+0000 7f3f35868700 1 -- 172.21.15.29:0/3063529452 wait complete. 2024-04-27T02:03:40.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:39 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/1964314032' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-27T02:03:40.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:39 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/1964314032' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-27T02:03:40.592 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378650 got 81604378653 for osd.2 2024-04-27T02:03:40.592 DEBUG:teuthology.parallel:result is None 2024-04-27T02:03:41.049 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542177 got 60129542179 for osd.1 2024-04-27T02:03:41.049 DEBUG:teuthology.parallel:result is None 2024-04-27T02:03:41.049 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-27T02:03:41.049 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-27T02:03:41.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:40 smithi029 ceph-mon[26769]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:41.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:40 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/3063529452' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-27T02:03:41.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:40 smithi052 ceph-mon[32586]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:41.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:40 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/3063529452' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-27T02:03:41.266 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:03:42.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.842+0000 7fd6a61d9700 1 -- 172.21.15.29:0/227672992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6a0100e60 msgr2=0x7fd6a0104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:42.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.842+0000 7fd6a61d9700 1 --2- 172.21.15.29:0/227672992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6a0100e60 0x7fd6a0104e30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd694009910 tx=0x7fd694009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:42.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.842+0000 7fd6a61d9700 1 -- 172.21.15.29:0/227672992 shutdown_connections 2024-04-27T02:03:42.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.842+0000 7fd6a61d9700 1 --2- 172.21.15.29:0/227672992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6a0100e60 0x7fd6a0104e30 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:42.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.842+0000 7fd6a61d9700 1 --2- 172.21.15.29:0/227672992 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6a01004c0 0x7fd6a0100890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:42.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.842+0000 7fd6a61d9700 1 -- 172.21.15.29:0/227672992 >> 172.21.15.29:0/227672992 conn(0x7fd6a00fbd80 msgr2=0x7fd6a00fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:42.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.842+0000 7fd6a61d9700 1 -- 172.21.15.29:0/227672992 shutdown_connections 2024-04-27T02:03:42.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.842+0000 7fd6a61d9700 1 -- 172.21.15.29:0/227672992 wait complete. 2024-04-27T02:03:42.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.843+0000 7fd6a61d9700 1 Processor -- start 2024-04-27T02:03:42.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.843+0000 7fd6a61d9700 1 -- start start 2024-04-27T02:03:42.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.843+0000 7fd6a61d9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6a01004c0 0x7fd6a0195f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:42.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.843+0000 7fd6a61d9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6a0100e60 0x7fd6a0196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:42.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.843+0000 7fd6a61d9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a0196b70 con 0x7fd6a0100e60 2024-04-27T02:03:42.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.843+0000 7fd6a61d9700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a019a900 con 0x7fd6a01004c0 2024-04-27T02:03:42.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.843+0000 7fd69effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6a0100e60 0x7fd6a0196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:42.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.843+0000 7fd69effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6a0100e60 0x7fd6a0196490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55244/0 (socket says 172.21.15.29:55244) 2024-04-27T02:03:42.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.843+0000 7fd69effd700 1 -- 172.21.15.29:0/2022146399 learned_addr learned my addr 172.21.15.29:0/2022146399 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:03:42.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.843+0000 7fd69f7fe700 1 --2- 172.21.15.29:0/2022146399 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6a01004c0 0x7fd6a0195f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:42.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.843+0000 7fd69effd700 1 -- 172.21.15.29:0/2022146399 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6a01004c0 msgr2=0x7fd6a0195f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:42.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.843+0000 7fd69effd700 1 --2- 172.21.15.29:0/2022146399 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6a01004c0 0x7fd6a0195f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:42.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.844+0000 7fd69effd700 1 -- 172.21.15.29:0/2022146399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6940095f0 con 0x7fd6a0100e60 2024-04-27T02:03:42.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.844+0000 7fd69effd700 1 --2- 172.21.15.29:0/2022146399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6a0100e60 0x7fd6a0196490 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd6940092e0 tx=0x7fd69400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:42.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.844+0000 7fd69cff9700 1 -- 172.21.15.29:0/2022146399 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd69401d070 con 0x7fd6a0100e60 2024-04-27T02:03:42.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.844+0000 7fd6a61d9700 1 -- 172.21.15.29:0/2022146399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a019ab80 con 0x7fd6a0100e60 2024-04-27T02:03:42.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.844+0000 7fd69cff9700 1 -- 172.21.15.29:0/2022146399 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd69400f460 con 0x7fd6a0100e60 2024-04-27T02:03:42.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.844+0000 7fd69cff9700 1 -- 172.21.15.29:0/2022146399 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd694017630 con 0x7fd6a0100e60 2024-04-27T02:03:42.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.844+0000 7fd6a61d9700 1 -- 172.21.15.29:0/2022146399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a019b070 con 0x7fd6a0100e60 2024-04-27T02:03:42.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.845+0000 7fd6a61d9700 1 -- 172.21.15.29:0/2022146399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a0108830 con 0x7fd6a0100e60 2024-04-27T02:03:42.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.845+0000 7fd69cff9700 1 -- 172.21.15.29:0/2022146399 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fd69400f5d0 con 0x7fd6a0100e60 2024-04-27T02:03:42.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.846+0000 7fd69cff9700 1 --2- 172.21.15.29:0/2022146399 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fd68806c680 0x7fd68806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:42.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.846+0000 7fd69cff9700 1 -- 172.21.15.29:0/2022146399 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd69408c9f0 con 0x7fd6a0100e60 2024-04-27T02:03:42.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.846+0000 7fd69f7fe700 1 --2- 172.21.15.29:0/2022146399 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fd68806c680 0x7fd68806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:42.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.847+0000 7fd69f7fe700 1 --2- 172.21.15.29:0/2022146399 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fd68806c680 0x7fd68806eb30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd690005950 tx=0x7fd6900058e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:42.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:42.851+0000 7fd69cff9700 1 -- 172.21.15.29:0/2022146399 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd69408ee60 con 0x7fd6a0100e60 2024-04-27T02:03:43.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:43.049+0000 7fd6a61d9700 1 -- 172.21.15.29:0/2022146399 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd6a019b290 con 0x7fd68806c680 2024-04-27T02:03:43.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:43.050+0000 7fd69cff9700 1 -- 172.21.15.29:0/2022146399 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28240 (secure 0 0 0) 0x7fd6a019b290 con 0x7fd68806c680 2024-04-27T02:03:43.032 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:03:43.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:43.052+0000 7fd6a61d9700 1 -- 172.21.15.29:0/2022146399 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fd68806c680 msgr2=0x7fd68806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:43.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:43.052+0000 7fd6a61d9700 1 --2- 172.21.15.29:0/2022146399 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fd68806c680 0x7fd68806eb30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd690005950 tx=0x7fd6900058e0 comp rx=0 tx=0).stop 2024-04-27T02:03:43.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:43.052+0000 7fd6a61d9700 1 -- 172.21.15.29:0/2022146399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6a0100e60 msgr2=0x7fd6a0196490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:43.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:43.052+0000 7fd6a61d9700 1 --2- 172.21.15.29:0/2022146399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6a0100e60 0x7fd6a0196490 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd6940092e0 tx=0x7fd69400be20 comp rx=0 tx=0).stop 2024-04-27T02:03:43.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:43.053+0000 7fd6a61d9700 1 -- 172.21.15.29:0/2022146399 shutdown_connections 2024-04-27T02:03:43.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:43.053+0000 7fd6a61d9700 1 --2- 172.21.15.29:0/2022146399 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fd68806c680 0x7fd68806eb30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:43.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:43.053+0000 7fd6a61d9700 1 --2- 172.21.15.29:0/2022146399 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6a01004c0 0x7fd6a0195f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:43.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:43.053+0000 7fd6a61d9700 1 --2- 172.21.15.29:0/2022146399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6a0100e60 0x7fd6a0196490 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:43.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:43.053+0000 7fd6a61d9700 1 -- 172.21.15.29:0/2022146399 >> 172.21.15.29:0/2022146399 conn(0x7fd6a00fbd80 msgr2=0x7fd6a00fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:43.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:43.053+0000 7fd6a61d9700 1 -- 172.21.15.29:0/2022146399 shutdown_connections 2024-04-27T02:03:43.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:43.053+0000 7fd6a61d9700 1 -- 172.21.15.29:0/2022146399 wait complete. 2024-04-27T02:03:43.035 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-04-27T02:03:43.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:42 smithi029 ceph-mon[26769]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:43.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:42 smithi052 ceph-mon[32586]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:43.747 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-04-27T02:03:41.825207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165584,"kb_used_data":3872,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305328,"statfs":{"total":575970213888,"available":575800655872,"internally_reserved":0,"allocated":3964928,"data_stored":2608497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001959"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":185,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-27T02:02:48.852201+0000","last_change":"2024-04-27T02:02:21.874768+0000","last_active":"2024-04-27T02:02:48.852201+0000","last_peered":"2024-04-27T02:02:48.852201+0000","last_clean":"2024-04-27T02:02:48.852201+0000","last_became_active":"2024-04-27T02:02:21.874400+0000","last_became_peered":"2024-04-27T02:02:21.874400+0000","last_unstale":"2024-04-27T02:02:48.852201+0000","last_undegraded":"2024-04-27T02:02:48.852201+0000","last_fullsized":"2024-04-27T02:02:48.852201+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T02:01:30.307083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T02:01:30.307083+0000","last_clean_scrub_stamp":"2024-04-27T02:01:30.307083+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:44:38.492854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 02:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.57899999999999996}]},{"osd":1,"last update":"Sat Apr 27 02:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.60199999999999998}]},{"osd":2,"last update":"Sat Apr 27 02:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.63900000000000001}]},{"osd":3,"last update":"Sat Apr 27 02:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.65300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]}]},{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 02:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.63600000000000001}]},{"osd":1,"last update":"Sat Apr 27 02:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.59699999999999998}]},{"osd":2,"last update":"Sat Apr 27 02:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.61399999999999999}]},{"osd":4,"last update":"Sat Apr 27 02:03:23 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.38100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378654,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27304,"kb_used_data":352,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717848,"statfs":{"total":95995035648,"available":95967076352,"internally_reserved":0,"allocated":360448,"data_stored":139008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 02:03:33 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.38800000000000001,"15min":0.39600000000000002},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.39900000000000002,"15min":0.40600000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.29799999999999999}]},{"osd":1,"last update":"Sat Apr 27 02:03:33 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.42199999999999999,"15min":0.42999999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.38},{"interface":"front","average":{"1min":0.38,"5min":0.41699999999999998,"15min":0.42299999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.28100000000000003}]},{"osd":3,"last update":"Sat Apr 27 02:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.64400000000000002}]},{"osd":4,"last update":"Sat Apr 27 02:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.61799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705705,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 27 02:03:06 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.34300000000000003,"15min":0.34000000000000002},"min":{"1min":0.24199999999999999,"5min":0.219,"15min":0.219},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.32700000000000001,"15min":0.32100000000000001},"min":{"1min":0.27900000000000003,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.50800000000000001,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.23599999999999999}]},{"osd":2,"last update":"Sat Apr 27 02:03:37 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.375,"15min":0.38100000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.436,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.38400000000000001,"15min":0.39000000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.42899999999999999,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.30299999999999999}]},{"osd":3,"last update":"Sat Apr 27 02:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.42899999999999999}]},{"osd":4,"last update":"Sat Apr 27 02:03:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.39400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 02:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.34300000000000003,"15min":0.33600000000000002},"min":{"1min":0.22,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.35499999999999998,"15min":0.34599999999999997},"min":{"1min":0.27700000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.29099999999999998}]},{"osd":2,"last update":"Sat Apr 27 02:03:33 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.39800000000000002,"15min":0.40400000000000003},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.48899999999999999,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.42199999999999999,"15min":0.42899999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.55200000000000005,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.33300000000000002}]},{"osd":3,"last update":"Sat Apr 27 02:02:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.379},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.45800000000000002}]},{"osd":4,"last update":"Sat Apr 27 02:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.41299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T02:03:43.748 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-27T02:03:43.966 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:03:45.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:44 smithi029 ceph-mon[26769]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T02:03:45.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:44 smithi029 ceph-mon[26769]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:45.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:44 smithi052 ceph-mon[32586]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T02:03:45.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:44 smithi052 ceph-mon[32586]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:45.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.618+0000 7fed8a6a0700 1 -- 172.21.15.29:0/1671162798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed84069140 msgr2=0x7fed84104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:45.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.618+0000 7fed8a6a0700 1 --2- 172.21.15.29:0/1671162798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed84069140 0x7fed84104eb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fed78009910 tx=0x7fed78009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:45.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.618+0000 7fed8a6a0700 1 -- 172.21.15.29:0/1671162798 shutdown_connections 2024-04-27T02:03:45.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.618+0000 7fed8a6a0700 1 --2- 172.21.15.29:0/1671162798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed84069140 0x7fed84104eb0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:45.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.618+0000 7fed8a6a0700 1 --2- 172.21.15.29:0/1671162798 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed840687a0 0x7fed84068b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:45.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.618+0000 7fed8a6a0700 1 -- 172.21.15.29:0/1671162798 >> 172.21.15.29:0/1671162798 conn(0x7fed84075130 msgr2=0x7fed84075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:45.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.618+0000 7fed8a6a0700 1 -- 172.21.15.29:0/1671162798 shutdown_connections 2024-04-27T02:03:45.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.618+0000 7fed8a6a0700 1 -- 172.21.15.29:0/1671162798 wait complete. 2024-04-27T02:03:45.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.618+0000 7fed8a6a0700 1 Processor -- start 2024-04-27T02:03:45.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.618+0000 7fed8a6a0700 1 -- start start 2024-04-27T02:03:45.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.619+0000 7fed8a6a0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed840687a0 0x7fed840ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:45.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.619+0000 7fed8a6a0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed84069140 0x7fed841001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:45.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.619+0000 7fed8a6a0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed84100720 con 0x7fed84069140 2024-04-27T02:03:45.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.619+0000 7fed8a6a0700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed84100860 con 0x7fed840687a0 2024-04-27T02:03:45.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.619+0000 7fed837fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed84069140 0x7fed841001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:45.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.619+0000 7fed837fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed84069140 0x7fed841001e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55268/0 (socket says 172.21.15.29:55268) 2024-04-27T02:03:45.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.619+0000 7fed837fe700 1 -- 172.21.15.29:0/1385377404 learned_addr learned my addr 172.21.15.29:0/1385377404 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:03:45.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.619+0000 7fed83fff700 1 --2- 172.21.15.29:0/1385377404 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed840687a0 0x7fed840ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:45.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.619+0000 7fed837fe700 1 -- 172.21.15.29:0/1385377404 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed840687a0 msgr2=0x7fed840ffca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:45.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.619+0000 7fed837fe700 1 --2- 172.21.15.29:0/1385377404 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed840687a0 0x7fed840ffca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:45.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.619+0000 7fed837fe700 1 -- 172.21.15.29:0/1385377404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed780095f0 con 0x7fed84069140 2024-04-27T02:03:45.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.619+0000 7fed837fe700 1 --2- 172.21.15.29:0/1385377404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed84069140 0x7fed841001e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fed780092e0 tx=0x7fed7800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:45.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.619+0000 7fed817fa700 1 -- 172.21.15.29:0/1385377404 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed7801d070 con 0x7fed84069140 2024-04-27T02:03:45.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.620+0000 7fed817fa700 1 -- 172.21.15.29:0/1385377404 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fed78005dc0 con 0x7fed84069140 2024-04-27T02:03:45.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.620+0000 7fed8a6a0700 1 -- 172.21.15.29:0/1385377404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed84100ae0 con 0x7fed84069140 2024-04-27T02:03:45.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.620+0000 7fed817fa700 1 -- 172.21.15.29:0/1385377404 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed78017400 con 0x7fed84069140 2024-04-27T02:03:45.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.620+0000 7fed8a6a0700 1 -- 172.21.15.29:0/1385377404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed841a2360 con 0x7fed84069140 2024-04-27T02:03:45.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.621+0000 7fed8a6a0700 1 -- 172.21.15.29:0/1385377404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed84108750 con 0x7fed84069140 2024-04-27T02:03:45.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.621+0000 7fed817fa700 1 -- 172.21.15.29:0/1385377404 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fed7800f460 con 0x7fed84069140 2024-04-27T02:03:45.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.621+0000 7fed817fa700 1 --2- 172.21.15.29:0/1385377404 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fed6c06c680 0x7fed6c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:45.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.622+0000 7fed83fff700 1 --2- 172.21.15.29:0/1385377404 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fed6c06c680 0x7fed6c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:45.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.622+0000 7fed817fa700 1 -- 172.21.15.29:0/1385377404 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fed7808c960 con 0x7fed84069140 2024-04-27T02:03:45.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.622+0000 7fed83fff700 1 --2- 172.21.15.29:0/1385377404 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fed6c06c680 0x7fed6c06eb30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fed74005950 tx=0x7fed7400b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:45.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.626+0000 7fed817fa700 1 -- 172.21.15.29:0/1385377404 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fed78057380 con 0x7fed84069140 2024-04-27T02:03:45.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.821+0000 7fed8a6a0700 1 -- 172.21.15.29:0/1385377404 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fed84061190 con 0x7fed6c06c680 2024-04-27T02:03:45.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.823+0000 7fed817fa700 1 -- 172.21.15.29:0/1385377404 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28241 (secure 0 0 0) 0x7fed84061190 con 0x7fed6c06c680 2024-04-27T02:03:45.804 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:03:45.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.825+0000 7fed8a6a0700 1 -- 172.21.15.29:0/1385377404 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fed6c06c680 msgr2=0x7fed6c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:45.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.825+0000 7fed8a6a0700 1 --2- 172.21.15.29:0/1385377404 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fed6c06c680 0x7fed6c06eb30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fed74005950 tx=0x7fed7400b410 comp rx=0 tx=0).stop 2024-04-27T02:03:45.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.825+0000 7fed8a6a0700 1 -- 172.21.15.29:0/1385377404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed84069140 msgr2=0x7fed841001e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:45.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.825+0000 7fed8a6a0700 1 --2- 172.21.15.29:0/1385377404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed84069140 0x7fed841001e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fed780092e0 tx=0x7fed7800bd70 comp rx=0 tx=0).stop 2024-04-27T02:03:45.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.825+0000 7fed8a6a0700 1 -- 172.21.15.29:0/1385377404 shutdown_connections 2024-04-27T02:03:45.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.825+0000 7fed8a6a0700 1 --2- 172.21.15.29:0/1385377404 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fed6c06c680 0x7fed6c06eb30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:45.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.825+0000 7fed8a6a0700 1 --2- 172.21.15.29:0/1385377404 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed840687a0 0x7fed840ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:45.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.825+0000 7fed8a6a0700 1 --2- 172.21.15.29:0/1385377404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed84069140 0x7fed841001e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:45.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.825+0000 7fed8a6a0700 1 -- 172.21.15.29:0/1385377404 >> 172.21.15.29:0/1385377404 conn(0x7fed84075130 msgr2=0x7fed84104720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:45.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.825+0000 7fed8a6a0700 1 -- 172.21.15.29:0/1385377404 shutdown_connections 2024-04-27T02:03:45.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:45.825+0000 7fed8a6a0700 1 -- 172.21.15.29:0/1385377404 wait complete. 2024-04-27T02:03:45.806 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-04-27T02:03:46.572 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2024-04-27T02:03:43.825735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165584,"kb_used_data":3872,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305328,"statfs":{"total":575970213888,"available":575800655872,"internally_reserved":0,"allocated":3964928,"data_stored":2608497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002187"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":185,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-27T02:02:48.852201+0000","last_change":"2024-04-27T02:02:21.874768+0000","last_active":"2024-04-27T02:02:48.852201+0000","last_peered":"2024-04-27T02:02:48.852201+0000","last_clean":"2024-04-27T02:02:48.852201+0000","last_became_active":"2024-04-27T02:02:21.874400+0000","last_became_peered":"2024-04-27T02:02:21.874400+0000","last_unstale":"2024-04-27T02:02:48.852201+0000","last_undegraded":"2024-04-27T02:02:48.852201+0000","last_fullsized":"2024-04-27T02:02:48.852201+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T02:01:30.307083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T02:01:30.307083+0000","last_clean_scrub_stamp":"2024-04-27T02:01:30.307083+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:44:38.492854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 02:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.38800000000000001}]},{"osd":1,"last update":"Sat Apr 27 02:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.505},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.45200000000000001}]},{"osd":2,"last update":"Sat Apr 27 02:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.49199999999999999}]},{"osd":3,"last update":"Sat Apr 27 02:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.59199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]}]},{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 02:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.63600000000000001}]},{"osd":1,"last update":"Sat Apr 27 02:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.59699999999999998}]},{"osd":2,"last update":"Sat Apr 27 02:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.61399999999999999}]},{"osd":4,"last update":"Sat Apr 27 02:03:23 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.38100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378654,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27304,"kb_used_data":352,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717848,"statfs":{"total":95995035648,"available":95967076352,"internally_reserved":0,"allocated":360448,"data_stored":139008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 02:03:33 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.38800000000000001,"15min":0.39600000000000002},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.39900000000000002,"15min":0.40600000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.29799999999999999}]},{"osd":1,"last update":"Sat Apr 27 02:03:33 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.42199999999999999,"15min":0.42999999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.38},{"interface":"front","average":{"1min":0.38,"5min":0.41699999999999998,"15min":0.42299999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.28100000000000003}]},{"osd":3,"last update":"Sat Apr 27 02:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.64400000000000002}]},{"osd":4,"last update":"Sat Apr 27 02:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.61799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705706,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 27 02:03:06 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.34300000000000003,"15min":0.34000000000000002},"min":{"1min":0.24199999999999999,"5min":0.219,"15min":0.219},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.32700000000000001,"15min":0.32100000000000001},"min":{"1min":0.27900000000000003,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.50800000000000001,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.495}]},{"osd":2,"last update":"Sat Apr 27 02:03:37 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.375,"15min":0.38100000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.436,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.38400000000000001,"15min":0.39000000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.42899999999999999,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.51800000000000002}]},{"osd":3,"last update":"Sat Apr 27 02:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.78700000000000003}]},{"osd":4,"last update":"Sat Apr 27 02:03:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.83199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]}]},{"osd":1,"up_from":14,"seq":60129542180,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 02:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.34300000000000003,"15min":0.33600000000000002},"min":{"1min":0.22,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.35499999999999998,"15min":0.34599999999999997},"min":{"1min":0.27700000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.35199999999999998}]},{"osd":2,"last update":"Sat Apr 27 02:03:33 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.39800000000000002,"15min":0.40400000000000003},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.48899999999999999,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.42199999999999999,"15min":0.42899999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.55200000000000005,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.36799999999999999}]},{"osd":3,"last update":"Sat Apr 27 02:02:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.53900000000000003}]},{"osd":4,"last update":"Sat Apr 27 02:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.48999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T02:03:46.573 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-27T02:03:46.573 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-27T02:03:46.573 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-27T02:03:46.573 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-27T02:03:46.792 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:03:47.163 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:46 smithi029 ceph-mon[26769]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T02:03:47.163 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:46 smithi029 ceph-mon[26769]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:47.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:46 smithi052 ceph-mon[32586]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T02:03:47.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:46 smithi052 ceph-mon[32586]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.431+0000 7fbd7706d700 1 -- 172.21.15.29:0/1867616146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd70101d30 msgr2=0x7fbd7010a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.431+0000 7fbd7706d700 1 --2- 172.21.15.29:0/1867616146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd70101d30 0x7fbd7010a230 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbd64009910 tx=0x7fbd64009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.431+0000 7fbd7706d700 1 -- 172.21.15.29:0/1867616146 shutdown_connections 2024-04-27T02:03:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.431+0000 7fbd7706d700 1 --2- 172.21.15.29:0/1867616146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd70101d30 0x7fbd7010a230 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.431+0000 7fbd7706d700 1 --2- 172.21.15.29:0/1867616146 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd70101420 0x7fbd701017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.431+0000 7fbd7706d700 1 -- 172.21.15.29:0/1867616146 >> 172.21.15.29:0/1867616146 conn(0x7fbd700fac80 msgr2=0x7fbd700fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.431+0000 7fbd7706d700 1 -- 172.21.15.29:0/1867616146 shutdown_connections 2024-04-27T02:03:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.431+0000 7fbd7706d700 1 -- 172.21.15.29:0/1867616146 wait complete. 2024-04-27T02:03:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.431+0000 7fbd7706d700 1 Processor -- start 2024-04-27T02:03:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.431+0000 7fbd7706d700 1 -- start start 2024-04-27T02:03:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.432+0000 7fbd7706d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd70101420 0x7fbd700ff080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.432+0000 7fbd7706d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd70101d30 0x7fbd700ff5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.432+0000 7fbd7706d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd700ffb00 con 0x7fbd70101420 2024-04-27T02:03:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.432+0000 7fbd7706d700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd700ffc40 con 0x7fbd70101d30 2024-04-27T02:03:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.432+0000 7fbd74e09700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd70101420 0x7fbd700ff080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.432+0000 7fbd74e09700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd70101420 0x7fbd700ff080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39154/0 (socket says 172.21.15.29:39154) 2024-04-27T02:03:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.432+0000 7fbd74e09700 1 -- 172.21.15.29:0/3122825777 learned_addr learned my addr 172.21.15.29:0/3122825777 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:03:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.432+0000 7fbd6ffff700 1 --2- 172.21.15.29:0/3122825777 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd70101d30 0x7fbd700ff5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.432+0000 7fbd74e09700 1 -- 172.21.15.29:0/3122825777 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd70101d30 msgr2=0x7fbd700ff5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.432+0000 7fbd74e09700 1 --2- 172.21.15.29:0/3122825777 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd70101d30 0x7fbd700ff5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.432+0000 7fbd74e09700 1 -- 172.21.15.29:0/3122825777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd640095f0 con 0x7fbd70101420 2024-04-27T02:03:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.432+0000 7fbd6ffff700 1 --2- 172.21.15.29:0/3122825777 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd70101d30 0x7fbd700ff5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T02:03:48.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.432+0000 7fbd74e09700 1 --2- 172.21.15.29:0/3122825777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd70101420 0x7fbd700ff080 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbd6000d8d0 tx=0x7fbd6000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:48.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.432+0000 7fbd6dffb700 1 -- 172.21.15.29:0/3122825777 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd60009940 con 0x7fbd70101420 2024-04-27T02:03:48.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.432+0000 7fbd7706d700 1 -- 172.21.15.29:0/3122825777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd701a2030 con 0x7fbd70101420 2024-04-27T02:03:48.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.432+0000 7fbd6dffb700 1 -- 172.21.15.29:0/3122825777 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbd60010460 con 0x7fbd70101420 2024-04-27T02:03:48.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.432+0000 7fbd6dffb700 1 -- 172.21.15.29:0/3122825777 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd6000f5f0 con 0x7fbd70101420 2024-04-27T02:03:48.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.433+0000 7fbd7706d700 1 -- 172.21.15.29:0/3122825777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd701a2440 con 0x7fbd70101420 2024-04-27T02:03:48.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.434+0000 7fbd7706d700 1 -- 172.21.15.29:0/3122825777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd701078a0 con 0x7fbd70101420 2024-04-27T02:03:48.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.434+0000 7fbd6dffb700 1 -- 172.21.15.29:0/3122825777 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fbd600105d0 con 0x7fbd70101420 2024-04-27T02:03:48.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.435+0000 7fbd6dffb700 1 --2- 172.21.15.29:0/3122825777 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fbd5806c680 0x7fbd5806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:48.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.435+0000 7fbd6dffb700 1 -- 172.21.15.29:0/3122825777 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fbd6008b5e0 con 0x7fbd70101420 2024-04-27T02:03:48.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.435+0000 7fbd6ffff700 1 --2- 172.21.15.29:0/3122825777 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fbd5806c680 0x7fbd5806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:48.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.435+0000 7fbd6ffff700 1 --2- 172.21.15.29:0/3122825777 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fbd5806c680 0x7fbd5806eb30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400b5c0 tx=0x7fbd6401a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:48.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.439+0000 7fbd6dffb700 1 -- 172.21.15.29:0/3122825777 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fbd60055f60 con 0x7fbd70101420 2024-04-27T02:03:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.687+0000 7fbd7706d700 1 -- 172.21.15.29:0/3122825777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbd7004f2a0 con 0x7fbd70101420 2024-04-27T02:03:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.687+0000 7fbd6dffb700 1 -- 172.21.15.29:0/3122825777 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fbd60059580 con 0x7fbd70101420 2024-04-27T02:03:48.668 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:03:48.668 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-27T02:03:48.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.689+0000 7fbd7706d700 1 -- 172.21.15.29:0/3122825777 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fbd5806c680 msgr2=0x7fbd5806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:48.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.689+0000 7fbd7706d700 1 --2- 172.21.15.29:0/3122825777 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fbd5806c680 0x7fbd5806eb30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400b5c0 tx=0x7fbd6401a040 comp rx=0 tx=0).stop 2024-04-27T02:03:48.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.689+0000 7fbd7706d700 1 -- 172.21.15.29:0/3122825777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd70101420 msgr2=0x7fbd700ff080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:48.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.689+0000 7fbd7706d700 1 --2- 172.21.15.29:0/3122825777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd70101420 0x7fbd700ff080 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbd6000d8d0 tx=0x7fbd6000dc90 comp rx=0 tx=0).stop 2024-04-27T02:03:48.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.689+0000 7fbd7706d700 1 -- 172.21.15.29:0/3122825777 shutdown_connections 2024-04-27T02:03:48.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.690+0000 7fbd7706d700 1 --2- 172.21.15.29:0/3122825777 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fbd5806c680 0x7fbd5806eb30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:48.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.690+0000 7fbd7706d700 1 --2- 172.21.15.29:0/3122825777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd70101420 0x7fbd700ff080 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:48.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.690+0000 7fbd7706d700 1 --2- 172.21.15.29:0/3122825777 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd70101d30 0x7fbd700ff5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:48.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.690+0000 7fbd7706d700 1 -- 172.21.15.29:0/3122825777 >> 172.21.15.29:0/3122825777 conn(0x7fbd700fac80 msgr2=0x7fbd701049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:48.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.690+0000 7fbd7706d700 1 -- 172.21.15.29:0/3122825777 shutdown_connections 2024-04-27T02:03:48.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:48.690+0000 7fbd7706d700 1 -- 172.21.15.29:0/3122825777 wait complete. 2024-04-27T02:03:49.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:48 smithi029 ceph-mon[26769]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:49.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:48 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/3122825777' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-27T02:03:49.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:48 smithi052 ceph-mon[32586]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:49.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:48 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/3122825777' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-27T02:03:49.528 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-27T02:03:49.528 INFO:tasks.cephadm:Setup complete, yielding 2024-04-27T02:03:49.529 INFO:teuthology.run_tasks:Running task print... 2024-04-27T02:03:49.538 INFO:teuthology.task.print:**** done end installing v18.2.0 cephadm ... 2024-04-27T02:03:49.538 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T02:03:49.557 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi029.front.sepia.ceph.com 2024-04-27T02:03:49.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-04-27T02:03:49.782 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:03:51.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:50 smithi029 ceph-mon[26769]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:51.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:50 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:03:51.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:50 smithi052 ceph-mon[32586]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:51.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:50 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:03:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.515+0000 7f01669de700 1 -- 172.21.15.29:0/870204065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01600ffb60 msgr2=0x7f01600fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.515+0000 7f01669de700 1 --2- 172.21.15.29:0/870204065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01600ffb60 0x7f01600fffd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0148009910 tx=0x7f0148009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.515+0000 7f01669de700 1 -- 172.21.15.29:0/870204065 shutdown_connections 2024-04-27T02:03:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.515+0000 7f01669de700 1 --2- 172.21.15.29:0/870204065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01600ffb60 0x7f01600fffd0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.515+0000 7f01669de700 1 --2- 172.21.15.29:0/870204065 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01600fe8c0 0x7f01600fecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.515+0000 7f01669de700 1 -- 172.21.15.29:0/870204065 >> 172.21.15.29:0/870204065 conn(0x7f01600f9f10 msgr2=0x7f01600fc340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.515+0000 7f01669de700 1 -- 172.21.15.29:0/870204065 shutdown_connections 2024-04-27T02:03:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.515+0000 7f01669de700 1 -- 172.21.15.29:0/870204065 wait complete. 2024-04-27T02:03:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.516+0000 7f01669de700 1 Processor -- start 2024-04-27T02:03:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.516+0000 7f01669de700 1 -- start start 2024-04-27T02:03:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.516+0000 7f01669de700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01600fe8c0 0x7f0160105900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.516+0000 7f01669de700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01600ffb60 0x7f016019a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.516+0000 7f01669de700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016019ade0 con 0x7f01600fe8c0 2024-04-27T02:03:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.516+0000 7f01669de700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016019af20 con 0x7f01600ffb60 2024-04-27T02:03:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.516+0000 7f015ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01600fe8c0 0x7f0160105900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.516+0000 7f015ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01600fe8c0 0x7f0160105900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39166/0 (socket says 172.21.15.29:39166) 2024-04-27T02:03:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.516+0000 7f015ffff700 1 -- 172.21.15.29:0/2382213794 learned_addr learned my addr 172.21.15.29:0/2382213794 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:03:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.516+0000 7f015f7fe700 1 --2- 172.21.15.29:0/2382213794 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01600ffb60 0x7f016019a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.517+0000 7f015ffff700 1 -- 172.21.15.29:0/2382213794 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01600ffb60 msgr2=0x7f016019a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.517+0000 7f015ffff700 1 --2- 172.21.15.29:0/2382213794 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01600ffb60 0x7f016019a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.517+0000 7f015ffff700 1 -- 172.21.15.29:0/2382213794 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01480095f0 con 0x7f01600fe8c0 2024-04-27T02:03:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.517+0000 7f015f7fe700 1 --2- 172.21.15.29:0/2382213794 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01600ffb60 0x7f016019a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T02:03:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.517+0000 7f015ffff700 1 --2- 172.21.15.29:0/2382213794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01600fe8c0 0x7f0160105900 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f015000d8d0 tx=0x7f015000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.517+0000 7f015d7fa700 1 -- 172.21.15.29:0/2382213794 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0150009940 con 0x7f01600fe8c0 2024-04-27T02:03:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.517+0000 7f015d7fa700 1 -- 172.21.15.29:0/2382213794 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0150010460 con 0x7f01600fe8c0 2024-04-27T02:03:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.517+0000 7f01669de700 1 -- 172.21.15.29:0/2382213794 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f016019b200 con 0x7f01600fe8c0 2024-04-27T02:03:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.517+0000 7f015d7fa700 1 -- 172.21.15.29:0/2382213794 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f015000f5d0 con 0x7f01600fe8c0 2024-04-27T02:03:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.517+0000 7f01669de700 1 -- 172.21.15.29:0/2382213794 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f016019b750 con 0x7f01600fe8c0 2024-04-27T02:03:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.518+0000 7f01669de700 1 -- 172.21.15.29:0/2382213794 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0160107c30 con 0x7f01600fe8c0 2024-04-27T02:03:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.519+0000 7f015d7fa700 1 -- 172.21.15.29:0/2382213794 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f01500105d0 con 0x7f01600fe8c0 2024-04-27T02:03:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.519+0000 7f015d7fa700 1 --2- 172.21.15.29:0/2382213794 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f014c06c680 0x7f014c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.519+0000 7f015d7fa700 1 -- 172.21.15.29:0/2382213794 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f015008b540 con 0x7f01600fe8c0 2024-04-27T02:03:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.519+0000 7f015f7fe700 1 --2- 172.21.15.29:0/2382213794 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f014c06c680 0x7f014c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.519+0000 7f015f7fe700 1 --2- 172.21.15.29:0/2382213794 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f014c06c680 0x7f014c06eb30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f014800b5c0 tx=0x7f0148005e20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:51.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.524+0000 7f015d7fa700 1 -- 172.21.15.29:0/2382213794 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f01500599f0 con 0x7f01600fe8c0 2024-04-27T02:03:51.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.723+0000 7f01669de700 1 -- 172.21.15.29:0/2382213794 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f0160107e50 con 0x7f01600fe8c0 2024-04-27T02:03:51.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.727+0000 7f015d7fa700 1 -- 172.21.15.29:0/2382213794 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f0150059580 con 0x7f01600fe8c0 2024-04-27T02:03:51.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.729+0000 7f01669de700 1 -- 172.21.15.29:0/2382213794 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f014c06c680 msgr2=0x7f014c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:51.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.729+0000 7f01669de700 1 --2- 172.21.15.29:0/2382213794 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f014c06c680 0x7f014c06eb30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f014800b5c0 tx=0x7f0148005e20 comp rx=0 tx=0).stop 2024-04-27T02:03:51.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.729+0000 7f01669de700 1 -- 172.21.15.29:0/2382213794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01600fe8c0 msgr2=0x7f0160105900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:51.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.729+0000 7f01669de700 1 --2- 172.21.15.29:0/2382213794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01600fe8c0 0x7f0160105900 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f015000d8d0 tx=0x7f015000dc90 comp rx=0 tx=0).stop 2024-04-27T02:03:51.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.729+0000 7f01669de700 1 -- 172.21.15.29:0/2382213794 shutdown_connections 2024-04-27T02:03:51.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.729+0000 7f01669de700 1 --2- 172.21.15.29:0/2382213794 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f014c06c680 0x7f014c06eb30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:51.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.729+0000 7f01669de700 1 --2- 172.21.15.29:0/2382213794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01600fe8c0 0x7f0160105900 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:51.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.729+0000 7f01669de700 1 --2- 172.21.15.29:0/2382213794 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01600ffb60 0x7f016019a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:51.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.729+0000 7f01669de700 1 -- 172.21.15.29:0/2382213794 >> 172.21.15.29:0/2382213794 conn(0x7f01600f9f10 msgr2=0x7f0160102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:51.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.729+0000 7f01669de700 1 -- 172.21.15.29:0/2382213794 shutdown_connections 2024-04-27T02:03:51.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:51.729+0000 7f01669de700 1 -- 172.21.15.29:0/2382213794 wait complete. 2024-04-27T02:03:52.122 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:51 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:03:52.122 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:51 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:03:52.122 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:51 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:03:52.122 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:51 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/2382213794' entity='client.admin' 2024-04-27T02:03:52.122 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:51 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:03:52.122 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:51 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:03:52.123 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:51 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:03:52.123 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:51 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:03:52.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:51 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:03:52.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:51 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:03:52.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:51 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:03:52.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:51 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/2382213794' entity='client.admin' 2024-04-27T02:03:52.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:51 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:03:52.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:51 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:03:52.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:51 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:03:52.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:51 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:03:52.356 INFO:teuthology.run_tasks:Running task print... 2024-04-27T02:03:52.367 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-04-27T02:03:52.368 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T02:03:52.390 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi029.front.sepia.ceph.com 2024-04-27T02:03:52.390 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-27T02:03:52.612 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:03:53.073 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:52 smithi029 ceph-mon[26769]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:53.074 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:52 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:03:53.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:52 smithi052 ceph-mon[32586]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:53.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:52 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:03:54.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.306+0000 7fb722e46700 1 -- 172.21.15.29:0/1365446117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c069b00 msgr2=0x7fb71c102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:54.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.306+0000 7fb722e46700 1 --2- 172.21.15.29:0/1365446117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c069b00 0x7fb71c102df0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb710009910 tx=0x7fb710009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:54.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.306+0000 7fb722e46700 1 -- 172.21.15.29:0/1365446117 shutdown_connections 2024-04-27T02:03:54.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.306+0000 7fb722e46700 1 --2- 172.21.15.29:0/1365446117 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb71c103330 0x7fb71c1057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:54.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.306+0000 7fb722e46700 1 --2- 172.21.15.29:0/1365446117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c069b00 0x7fb71c102df0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:54.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.306+0000 7fb722e46700 1 -- 172.21.15.29:0/1365446117 >> 172.21.15.29:0/1365446117 conn(0x7fb71c0fa7a0 msgr2=0x7fb71c0fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:54.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.306+0000 7fb722e46700 1 -- 172.21.15.29:0/1365446117 shutdown_connections 2024-04-27T02:03:54.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.306+0000 7fb722e46700 1 -- 172.21.15.29:0/1365446117 wait complete. 2024-04-27T02:03:54.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.306+0000 7fb722e46700 1 Processor -- start 2024-04-27T02:03:54.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.307+0000 7fb722e46700 1 -- start start 2024-04-27T02:03:54.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.307+0000 7fb722e46700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb71c103330 0x7fb71c198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:54.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.307+0000 7fb722e46700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c1986c0 0x7fb71c19d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:54.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.307+0000 7fb722e46700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71c198bc0 con 0x7fb71c1986c0 2024-04-27T02:03:54.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.307+0000 7fb722e46700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71c198d30 con 0x7fb71c103330 2024-04-27T02:03:54.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.307+0000 7fb71bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c1986c0 0x7fb71c19d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:54.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.307+0000 7fb71bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c1986c0 0x7fb71c19d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39176/0 (socket says 172.21.15.29:39176) 2024-04-27T02:03:54.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.307+0000 7fb71bfff700 1 -- 172.21.15.29:0/3001571026 learned_addr learned my addr 172.21.15.29:0/3001571026 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:03:54.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.307+0000 7fb720be2700 1 --2- 172.21.15.29:0/3001571026 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb71c103330 0x7fb71c198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:54.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.307+0000 7fb71bfff700 1 -- 172.21.15.29:0/3001571026 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb71c103330 msgr2=0x7fb71c198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:54.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.307+0000 7fb71bfff700 1 --2- 172.21.15.29:0/3001571026 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb71c103330 0x7fb71c198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:54.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.307+0000 7fb71bfff700 1 -- 172.21.15.29:0/3001571026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7100095f0 con 0x7fb71c1986c0 2024-04-27T02:03:54.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.307+0000 7fb720be2700 1 --2- 172.21.15.29:0/3001571026 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb71c103330 0x7fb71c198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T02:03:54.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.307+0000 7fb71bfff700 1 --2- 172.21.15.29:0/3001571026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c1986c0 0x7fb71c19d6e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb70800b4b0 tx=0x7fb70800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:54.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.307+0000 7fb719ffb700 1 -- 172.21.15.29:0/3001571026 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb708011800 con 0x7fb71c1986c0 2024-04-27T02:03:54.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.308+0000 7fb719ffb700 1 -- 172.21.15.29:0/3001571026 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb708011e10 con 0x7fb71c1986c0 2024-04-27T02:03:54.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.308+0000 7fb722e46700 1 -- 172.21.15.29:0/3001571026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb71c19dc80 con 0x7fb71c1986c0 2024-04-27T02:03:54.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.308+0000 7fb719ffb700 1 -- 172.21.15.29:0/3001571026 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb70800f2e0 con 0x7fb71c1986c0 2024-04-27T02:03:54.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.308+0000 7fb722e46700 1 -- 172.21.15.29:0/3001571026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb71c19e1a0 con 0x7fb71c1986c0 2024-04-27T02:03:54.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.309+0000 7fb722e46700 1 -- 172.21.15.29:0/3001571026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb71c1924f0 con 0x7fb71c1986c0 2024-04-27T02:03:54.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.309+0000 7fb719ffb700 1 -- 172.21.15.29:0/3001571026 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fb708010380 con 0x7fb71c1986c0 2024-04-27T02:03:54.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.310+0000 7fb719ffb700 1 --2- 172.21.15.29:0/3001571026 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb70c06c680 0x7fb70c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:54.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.310+0000 7fb719ffb700 1 -- 172.21.15.29:0/3001571026 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb70808b0a0 con 0x7fb71c1986c0 2024-04-27T02:03:54.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.310+0000 7fb720be2700 1 --2- 172.21.15.29:0/3001571026 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb70c06c680 0x7fb70c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:54.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.310+0000 7fb720be2700 1 --2- 172.21.15.29:0/3001571026 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb70c06c680 0x7fb70c06eb30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb710006010 tx=0x7fb710005c00 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:54.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.314+0000 7fb719ffb700 1 -- 172.21.15.29:0/3001571026 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb7080595d0 con 0x7fb71c1986c0 2024-04-27T02:03:54.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.532+0000 7fb722e46700 1 -- 172.21.15.29:0/3001571026 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb71c061190 con 0x7fb70c06c680 2024-04-27T02:03:54.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.533+0000 7fb719ffb700 1 -- 172.21.15.29:0/3001571026 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fb71c061190 con 0x7fb70c06c680 2024-04-27T02:03:54.514 INFO:teuthology.orchestra.run.smithi029.stdout:Backend: cephadm 2024-04-27T02:03:54.514 INFO:teuthology.orchestra.run.smithi029.stdout:Available: Yes 2024-04-27T02:03:54.514 INFO:teuthology.orchestra.run.smithi029.stdout:Paused: No 2024-04-27T02:03:54.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.535+0000 7fb722e46700 1 -- 172.21.15.29:0/3001571026 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb70c06c680 msgr2=0x7fb70c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:54.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.535+0000 7fb722e46700 1 --2- 172.21.15.29:0/3001571026 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb70c06c680 0x7fb70c06eb30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb710006010 tx=0x7fb710005c00 comp rx=0 tx=0).stop 2024-04-27T02:03:54.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.535+0000 7fb722e46700 1 -- 172.21.15.29:0/3001571026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c1986c0 msgr2=0x7fb71c19d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:54.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.535+0000 7fb722e46700 1 --2- 172.21.15.29:0/3001571026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c1986c0 0x7fb71c19d6e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb70800b4b0 tx=0x7fb70800b870 comp rx=0 tx=0).stop 2024-04-27T02:03:54.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.535+0000 7fb722e46700 1 -- 172.21.15.29:0/3001571026 shutdown_connections 2024-04-27T02:03:54.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.535+0000 7fb722e46700 1 --2- 172.21.15.29:0/3001571026 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb70c06c680 0x7fb70c06eb30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:54.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.535+0000 7fb722e46700 1 --2- 172.21.15.29:0/3001571026 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb71c103330 0x7fb71c198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:54.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.535+0000 7fb722e46700 1 --2- 172.21.15.29:0/3001571026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c1986c0 0x7fb71c19d6e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:54.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.535+0000 7fb722e46700 1 -- 172.21.15.29:0/3001571026 >> 172.21.15.29:0/3001571026 conn(0x7fb71c0fa7a0 msgr2=0x7fb71c102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:54.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.535+0000 7fb722e46700 1 -- 172.21.15.29:0/3001571026 shutdown_connections 2024-04-27T02:03:54.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:54.535+0000 7fb722e46700 1 -- 172.21.15.29:0/3001571026 wait complete. 2024-04-27T02:03:55.165 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:54 smithi029 ceph-mon[26769]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:55.191 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-27T02:03:55.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:54 smithi052 ceph-mon[32586]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:55.407 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:03:56.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:55 smithi029 ceph-mon[26769]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:03:56.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:55 smithi052 ceph-mon[32586]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:03:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.209+0000 7f03802ad700 1 -- 172.21.15.29:0/929364910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03780ff130 msgr2=0x7f03780ff540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.209+0000 7f03802ad700 1 --2- 172.21.15.29:0/929364910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03780ff130 0x7f03780ff540 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0374009910 tx=0x7f0374009c20 comp rx=0 tx=0).stop 2024-04-27T02:03:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.209+0000 7f03802ad700 1 -- 172.21.15.29:0/929364910 shutdown_connections 2024-04-27T02:03:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.209+0000 7f03802ad700 1 --2- 172.21.15.29:0/929364910 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03781003d0 0x7f0378100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.209+0000 7f03802ad700 1 --2- 172.21.15.29:0/929364910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03780ff130 0x7f03780ff540 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.209+0000 7f03802ad700 1 -- 172.21.15.29:0/929364910 >> 172.21.15.29:0/929364910 conn(0x7f03780fa780 msgr2=0x7f03780fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.209+0000 7f03802ad700 1 -- 172.21.15.29:0/929364910 shutdown_connections 2024-04-27T02:03:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.209+0000 7f03802ad700 1 -- 172.21.15.29:0/929364910 wait complete. 2024-04-27T02:03:57.191 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:56 smithi029 ceph-mon[26769]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.210+0000 7f03802ad700 1 Processor -- start 2024-04-27T02:03:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.210+0000 7f03802ad700 1 -- start start 2024-04-27T02:03:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.210+0000 7f03802ad700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03780ff130 0x7f0378197cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.210+0000 7f03802ad700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03781003d0 0x7f03781981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.210+0000 7f03802ad700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03781987f0 con 0x7f03780ff130 2024-04-27T02:03:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.210+0000 7f03802ad700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0378198960 con 0x7f03781003d0 2024-04-27T02:03:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.210+0000 7f037e049700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03780ff130 0x7f0378197cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.210+0000 7f037e049700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03780ff130 0x7f0378197cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45384/0 (socket says 172.21.15.29:45384) 2024-04-27T02:03:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.210+0000 7f037e049700 1 -- 172.21.15.29:0/3558381960 learned_addr learned my addr 172.21.15.29:0/3558381960 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:03:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.211+0000 7f037d848700 1 --2- 172.21.15.29:0/3558381960 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03781003d0 0x7f03781981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.211+0000 7f037e049700 1 -- 172.21.15.29:0/3558381960 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03781003d0 msgr2=0x7f03781981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.211+0000 7f037e049700 1 --2- 172.21.15.29:0/3558381960 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03781003d0 0x7f03781981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.211+0000 7f037e049700 1 -- 172.21.15.29:0/3558381960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03740095f0 con 0x7f03780ff130 2024-04-27T02:03:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.211+0000 7f037e049700 1 --2- 172.21.15.29:0/3558381960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03780ff130 0x7f0378197cb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f037400bd70 tx=0x7f037400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.211+0000 7f036f7fe700 1 -- 172.21.15.29:0/3558381960 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f037401d070 con 0x7f03780ff130 2024-04-27T02:03:57.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.211+0000 7f03802ad700 1 -- 172.21.15.29:0/3558381960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f037819d390 con 0x7f03780ff130 2024-04-27T02:03:57.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.211+0000 7f036f7fe700 1 -- 172.21.15.29:0/3558381960 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0374022470 con 0x7f03780ff130 2024-04-27T02:03:57.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.211+0000 7f036f7fe700 1 -- 172.21.15.29:0/3558381960 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f037400f670 con 0x7f03780ff130 2024-04-27T02:03:57.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.211+0000 7f03802ad700 1 -- 172.21.15.29:0/3558381960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f037819d880 con 0x7f03780ff130 2024-04-27T02:03:57.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.212+0000 7f03802ad700 1 -- 172.21.15.29:0/3558381960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0378107c30 con 0x7f03780ff130 2024-04-27T02:03:57.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.213+0000 7f036f7fe700 1 -- 172.21.15.29:0/3558381960 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f0374022a50 con 0x7f03780ff130 2024-04-27T02:03:57.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.213+0000 7f036f7fe700 1 --2- 172.21.15.29:0/3558381960 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f036406c680 0x7f036406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:03:57.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.214+0000 7f036f7fe700 1 -- 172.21.15.29:0/3558381960 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f037408caa0 con 0x7f03780ff130 2024-04-27T02:03:57.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.214+0000 7f037d848700 1 --2- 172.21.15.29:0/3558381960 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f036406c680 0x7f036406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:03:57.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.214+0000 7f037d848700 1 --2- 172.21.15.29:0/3558381960 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f036406c680 0x7f036406eb30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0368005950 tx=0x7f03680058e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:03:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.218+0000 7f036f7fe700 1 -- 172.21.15.29:0/3558381960 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0374057570 con 0x7f03780ff130 2024-04-27T02:03:57.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:56 smithi052 ceph-mon[32586]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:57.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.427+0000 7f03802ad700 1 -- 172.21.15.29:0/3558381960 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0378068c50 con 0x7f036406c680 2024-04-27T02:03:57.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.436+0000 7f036f7fe700 1 -- 172.21.15.29:0/3558381960 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2860 (secure 0 0 0) 0x7f0378068c50 con 0x7f036406c680 2024-04-27T02:03:57.416 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:03:57.416 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.smithi029 smithi029 *:9093,9094 running (3m) 2m ago 5m 27.0M - 0.25.0 c8568f914cd2 f713bca72a8f 2024-04-27T02:03:57.416 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-exporter.smithi029 smithi029 running (5m) 2m ago 5m 7037k - 18.2.0 dc2bc1663786 d585d22341b7 2024-04-27T02:03:57.416 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-exporter.smithi052 smithi052 running (4m) 70s ago 4m 6962k - 18.2.0 dc2bc1663786 3555a1bdbadc 2024-04-27T02:03:57.417 INFO:teuthology.orchestra.run.smithi029.stdout:crash.smithi029 smithi029 running (5m) 2m ago 5m 6665k - 18.2.0 dc2bc1663786 aa848ffaabfc 2024-04-27T02:03:57.417 INFO:teuthology.orchestra.run.smithi029.stdout:crash.smithi052 smithi052 running (4m) 70s ago 4m 6660k - 18.2.0 dc2bc1663786 460da511ea40 2024-04-27T02:03:57.417 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.smithi029 smithi029 *:3000 running (3m) 2m ago 5m 80.2M - 9.4.7 954c08fa6188 2e3e5d432724 2024-04-27T02:03:57.417 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.smithi029.ewvdzs smithi029 *:9283,8765,8443 running (6m) 2m ago 6m 478M - 18.2.0 dc2bc1663786 76689e16ce1a 2024-04-27T02:03:57.417 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.smithi052.gbsftx smithi052 *:8443,9283,8765 running (4m) 70s ago 4m 438M - 18.2.0 dc2bc1663786 72bb682980a1 2024-04-27T02:03:57.417 INFO:teuthology.orchestra.run.smithi029.stdout:mon.smithi029 smithi029 running (6m) 2m ago 7m 44.9M 2048M 18.2.0 dc2bc1663786 33130493e80d 2024-04-27T02:03:57.417 INFO:teuthology.orchestra.run.smithi029.stdout:mon.smithi052 smithi052 running (4m) 70s ago 4m 40.5M 2048M 18.2.0 dc2bc1663786 5af8b02bce2f 2024-04-27T02:03:57.417 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.smithi029 smithi029 *:9100 running (5m) 2m ago 5m 15.4M - 1.5.0 0da6a335fe13 f65d14d021a2 2024-04-27T02:03:57.417 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.smithi052 smithi052 *:9100 running (4m) 70s ago 4m 16.3M - 1.5.0 0da6a335fe13 71aadb2b8e2e 2024-04-27T02:03:57.417 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (3m) 2m ago 3m 39.9M 3948M 18.2.0 dc2bc1663786 b4688a5b18ec 2024-04-27T02:03:57.417 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (3m) 2m ago 3m 40.7M 3948M 18.2.0 dc2bc1663786 7123e1ad8219 2024-04-27T02:03:57.417 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (2m) 2m ago 2m 39.7M 3948M 18.2.0 dc2bc1663786 f8ecf2655a40 2024-04-27T02:03:57.417 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi052 running (2m) 70s ago 2m 41.0M 4972M 18.2.0 dc2bc1663786 584fb2e408cd 2024-04-27T02:03:57.417 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi052 running (102s) 70s ago 102s 38.0M 4972M 18.2.0 dc2bc1663786 26fdd5746d2a 2024-04-27T02:03:57.418 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi052 running (75s) 70s ago 75s 11.3M 4972M 18.2.0 dc2bc1663786 6fa7665653a4 2024-04-27T02:03:57.418 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.smithi029 smithi029 *:9095 running (3m) 2m ago 4m 42.2M - 2.43.0 a07b618ecd1d a9ebc9badf18 2024-04-27T02:03:57.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.438+0000 7f03802ad700 1 -- 172.21.15.29:0/3558381960 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f036406c680 msgr2=0x7f036406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:57.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.438+0000 7f03802ad700 1 --2- 172.21.15.29:0/3558381960 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f036406c680 0x7f036406eb30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0368005950 tx=0x7f03680058e0 comp rx=0 tx=0).stop 2024-04-27T02:03:57.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.438+0000 7f03802ad700 1 -- 172.21.15.29:0/3558381960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03780ff130 msgr2=0x7f0378197cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:03:57.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.438+0000 7f03802ad700 1 --2- 172.21.15.29:0/3558381960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03780ff130 0x7f0378197cb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f037400bd70 tx=0x7f037400be50 comp rx=0 tx=0).stop 2024-04-27T02:03:57.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.438+0000 7f03802ad700 1 -- 172.21.15.29:0/3558381960 shutdown_connections 2024-04-27T02:03:57.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.438+0000 7f03802ad700 1 --2- 172.21.15.29:0/3558381960 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f036406c680 0x7f036406eb30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:57.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.438+0000 7f03802ad700 1 --2- 172.21.15.29:0/3558381960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03780ff130 0x7f0378197cb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:57.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.438+0000 7f03802ad700 1 --2- 172.21.15.29:0/3558381960 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03781003d0 0x7f03781981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:03:57.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.438+0000 7f03802ad700 1 -- 172.21.15.29:0/3558381960 >> 172.21.15.29:0/3558381960 conn(0x7f03780fa780 msgr2=0x7f0378068540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:03:57.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.438+0000 7f03802ad700 1 -- 172.21.15.29:0/3558381960 shutdown_connections 2024-04-27T02:03:57.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:03:57.439+0000 7f03802ad700 1 -- 172.21.15.29:0/3558381960 wait complete. 2024-04-27T02:03:58.081 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-27T02:03:58.295 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:03:59.127 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:58 smithi029 ceph-mon[26769]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:03:59.127 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:03:58 smithi029 ceph-mon[26769]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:03:59.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:58 smithi052 ceph-mon[32586]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:03:59.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:03:58 smithi052 ceph-mon[32586]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:04:00.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.090+0000 7f35caf74700 1 -- 172.21.15.29:0/1025084108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35c40ff130 msgr2=0x7f35c40ff540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:00.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.090+0000 7f35caf74700 1 --2- 172.21.15.29:0/1025084108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35c40ff130 0x7f35c40ff540 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f35b8009910 tx=0x7f35b8009c20 comp rx=0 tx=0).stop 2024-04-27T02:04:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.090+0000 7f35caf74700 1 -- 172.21.15.29:0/1025084108 shutdown_connections 2024-04-27T02:04:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.090+0000 7f35caf74700 1 --2- 172.21.15.29:0/1025084108 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35c41003d0 0x7f35c4100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.090+0000 7f35caf74700 1 --2- 172.21.15.29:0/1025084108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35c40ff130 0x7f35c40ff540 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.090+0000 7f35caf74700 1 -- 172.21.15.29:0/1025084108 >> 172.21.15.29:0/1025084108 conn(0x7f35c40fa780 msgr2=0x7f35c40fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.090+0000 7f35caf74700 1 -- 172.21.15.29:0/1025084108 shutdown_connections 2024-04-27T02:04:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.090+0000 7f35caf74700 1 -- 172.21.15.29:0/1025084108 wait complete. 2024-04-27T02:04:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.091+0000 7f35caf74700 1 Processor -- start 2024-04-27T02:04:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.091+0000 7f35caf74700 1 -- start start 2024-04-27T02:04:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.091+0000 7f35caf74700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35c40ff130 0x7f35c4195aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.091+0000 7f35caf74700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35c41003d0 0x7f35c4195fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.091+0000 7f35caf74700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c4196600 con 0x7f35c41003d0 2024-04-27T02:04:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.091+0000 7f35caf74700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c4196740 con 0x7f35c40ff130 2024-04-27T02:04:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.091+0000 7f35c3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35c41003d0 0x7f35c4195fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:00.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.091+0000 7f35c3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35c41003d0 0x7f35c4195fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45402/0 (socket says 172.21.15.29:45402) 2024-04-27T02:04:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.091+0000 7f35c3fff700 1 -- 172.21.15.29:0/790850384 learned_addr learned my addr 172.21.15.29:0/790850384 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:04:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.092+0000 7f35c8d10700 1 --2- 172.21.15.29:0/790850384 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35c40ff130 0x7f35c4195aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.092+0000 7f35c3fff700 1 -- 172.21.15.29:0/790850384 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35c40ff130 msgr2=0x7f35c4195aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.092+0000 7f35c3fff700 1 --2- 172.21.15.29:0/790850384 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35c40ff130 0x7f35c4195aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.092+0000 7f35c3fff700 1 -- 172.21.15.29:0/790850384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b80095f0 con 0x7f35c41003d0 2024-04-27T02:04:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.092+0000 7f35c8d10700 1 --2- 172.21.15.29:0/790850384 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35c40ff130 0x7f35c4195aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T02:04:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.092+0000 7f35c3fff700 1 --2- 172.21.15.29:0/790850384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35c41003d0 0x7f35c4195fe0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f35b000e920 tx=0x7f35b000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.092+0000 7f35c1ffb700 1 -- 172.21.15.29:0/790850384 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35b0009790 con 0x7f35c41003d0 2024-04-27T02:04:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.092+0000 7f35caf74700 1 -- 172.21.15.29:0/790850384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c419b1f0 con 0x7f35c41003d0 2024-04-27T02:04:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.092+0000 7f35c1ffb700 1 -- 172.21.15.29:0/790850384 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f35b000cba0 con 0x7f35c41003d0 2024-04-27T02:04:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.092+0000 7f35c1ffb700 1 -- 172.21.15.29:0/790850384 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35b0018630 con 0x7f35c41003d0 2024-04-27T02:04:00.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.092+0000 7f35caf74700 1 -- 172.21.15.29:0/790850384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c419b740 con 0x7f35c41003d0 2024-04-27T02:04:00.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.094+0000 7f35caf74700 1 -- 172.21.15.29:0/790850384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c4107c10 con 0x7f35c41003d0 2024-04-27T02:04:00.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.094+0000 7f35c1ffb700 1 -- 172.21.15.29:0/790850384 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f35b000cd10 con 0x7f35c41003d0 2024-04-27T02:04:00.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.094+0000 7f35c1ffb700 1 --2- 172.21.15.29:0/790850384 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f35b406c680 0x7f35b406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:00.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.094+0000 7f35c1ffb700 1 -- 172.21.15.29:0/790850384 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f35b0014070 con 0x7f35c41003d0 2024-04-27T02:04:00.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.094+0000 7f35c8d10700 1 --2- 172.21.15.29:0/790850384 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f35b406c680 0x7f35b406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:00.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.095+0000 7f35c8d10700 1 --2- 172.21.15.29:0/790850384 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f35b406c680 0x7f35b406eb30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f35b8009fd0 tx=0x7f35b8005940 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:00.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.099+0000 7f35c1ffb700 1 -- 172.21.15.29:0/790850384 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f35b0051720 con 0x7f35c41003d0 2024-04-27T02:04:00.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.307+0000 7f35caf74700 1 -- 172.21.15.29:0/790850384 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f35c4068c50 con 0x7f35b406c680 2024-04-27T02:04:00.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.312+0000 7f35c1ffb700 1 -- 172.21.15.29:0/790850384 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1290 (secure 0 0 0) 0x7f35c4068c50 con 0x7f35b406c680 2024-04-27T02:04:00.292 INFO:teuthology.orchestra.run.smithi029.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-27T02:04:00.292 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-04-27T02:04:00.292 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-exporter 2/2 2m ago 6m * 2024-04-27T02:04:00.292 INFO:teuthology.orchestra.run.smithi029.stdout:crash 2/2 2m ago 6m * 2024-04-27T02:04:00.292 INFO:teuthology.orchestra.run.smithi029.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-04-27T02:04:00.292 INFO:teuthology.orchestra.run.smithi029.stdout:mgr 2/2 2m ago 6m count:2 2024-04-27T02:04:00.292 INFO:teuthology.orchestra.run.smithi029.stdout:mon 2/2 2m ago 5m smithi029:172.21.15.29=smithi029;smithi052:172.21.15.52=smithi052;count:2 2024-04-27T02:04:00.292 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-04-27T02:04:00.293 INFO:teuthology.orchestra.run.smithi029.stdout:osd 6 2m ago - 2024-04-27T02:04:00.293 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-04-27T02:04:00.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.314+0000 7f35caf74700 1 -- 172.21.15.29:0/790850384 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f35b406c680 msgr2=0x7f35b406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:00.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.314+0000 7f35caf74700 1 --2- 172.21.15.29:0/790850384 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f35b406c680 0x7f35b406eb30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f35b8009fd0 tx=0x7f35b8005940 comp rx=0 tx=0).stop 2024-04-27T02:04:00.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.314+0000 7f35caf74700 1 -- 172.21.15.29:0/790850384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35c41003d0 msgr2=0x7f35c4195fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:00.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.314+0000 7f35caf74700 1 --2- 172.21.15.29:0/790850384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35c41003d0 0x7f35c4195fe0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f35b000e920 tx=0x7f35b000ece0 comp rx=0 tx=0).stop 2024-04-27T02:04:00.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.314+0000 7f35caf74700 1 -- 172.21.15.29:0/790850384 shutdown_connections 2024-04-27T02:04:00.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.314+0000 7f35caf74700 1 --2- 172.21.15.29:0/790850384 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f35b406c680 0x7f35b406eb30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:00.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.314+0000 7f35caf74700 1 --2- 172.21.15.29:0/790850384 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35c40ff130 0x7f35c4195aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:00.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.314+0000 7f35caf74700 1 --2- 172.21.15.29:0/790850384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35c41003d0 0x7f35c4195fe0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:00.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.314+0000 7f35caf74700 1 -- 172.21.15.29:0/790850384 >> 172.21.15.29:0/790850384 conn(0x7f35c40fa780 msgr2=0x7f35c4068540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:00.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.314+0000 7f35caf74700 1 -- 172.21.15.29:0/790850384 shutdown_connections 2024-04-27T02:04:00.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:00.314+0000 7f35caf74700 1 -- 172.21.15.29:0/790850384 wait complete. 2024-04-27T02:04:00.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-27T02:04:01.156 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:04:01.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:00 smithi029 ceph-mon[26769]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:04:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:00 smithi052 ceph-mon[32586]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:04:02.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:01 smithi052 ceph-mon[32586]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:04:02.294 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:01 smithi029 ceph-mon[26769]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:04:02.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.832+0000 7f1f884c0700 1 -- 172.21.15.29:0/1298404927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80100240 msgr2=0x7f1f80100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:02.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.832+0000 7f1f884c0700 1 --2- 172.21.15.29:0/1298404927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80100240 0x7f1f80100650 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1f78009910 tx=0x7f1f78009c20 comp rx=0 tx=0).stop 2024-04-27T02:04:02.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.832+0000 7f1f884c0700 1 -- 172.21.15.29:0/1298404927 shutdown_connections 2024-04-27T02:04:02.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.832+0000 7f1f884c0700 1 --2- 172.21.15.29:0/1298404927 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f80101440 0x7f1f80101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:02.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.832+0000 7f1f884c0700 1 --2- 172.21.15.29:0/1298404927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80100240 0x7f1f80100650 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:02.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.832+0000 7f1f884c0700 1 -- 172.21.15.29:0/1298404927 >> 172.21.15.29:0/1298404927 conn(0x7f1f800fb7f0 msgr2=0x7f1f800fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:02.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.832+0000 7f1f884c0700 1 -- 172.21.15.29:0/1298404927 shutdown_connections 2024-04-27T02:04:02.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.832+0000 7f1f884c0700 1 -- 172.21.15.29:0/1298404927 wait complete. 2024-04-27T02:04:02.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.833+0000 7f1f884c0700 1 Processor -- start 2024-04-27T02:04:02.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.833+0000 7f1f884c0700 1 -- start start 2024-04-27T02:04:02.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.833+0000 7f1f884c0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f80100240 0x7f1f80074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:02.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.833+0000 7f1f884c0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80101440 0x7f1f80073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:02.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.833+0000 7f1f884c0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f80073680 con 0x7f1f80101440 2024-04-27T02:04:02.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.833+0000 7f1f884c0700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f800737c0 con 0x7f1f80100240 2024-04-27T02:04:02.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.833+0000 7f1f85a5b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80101440 0x7f1f80073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:02.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.833+0000 7f1f85a5b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80101440 0x7f1f80073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45434/0 (socket says 172.21.15.29:45434) 2024-04-27T02:04:02.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.833+0000 7f1f85a5b700 1 -- 172.21.15.29:0/3590209692 learned_addr learned my addr 172.21.15.29:0/3590209692 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:04:02.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.833+0000 7f1f8625c700 1 --2- 172.21.15.29:0/3590209692 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f80100240 0x7f1f80074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:02.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.834+0000 7f1f85a5b700 1 -- 172.21.15.29:0/3590209692 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f80100240 msgr2=0x7f1f80074af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:02.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.834+0000 7f1f85a5b700 1 --2- 172.21.15.29:0/3590209692 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f80100240 0x7f1f80074af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:02.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.834+0000 7f1f85a5b700 1 -- 172.21.15.29:0/3590209692 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f780095f0 con 0x7f1f80101440 2024-04-27T02:04:02.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.834+0000 7f1f85a5b700 1 --2- 172.21.15.29:0/3590209692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80101440 0x7f1f80073140 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1f7400d8d0 tx=0x7f1f7400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.834+0000 7f1f7f7fe700 1 -- 172.21.15.29:0/3590209692 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f740098a0 con 0x7f1f80101440 2024-04-27T02:04:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.834+0000 7f1f884c0700 1 -- 172.21.15.29:0/3590209692 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f80073aa0 con 0x7f1f80101440 2024-04-27T02:04:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.834+0000 7f1f7f7fe700 1 -- 172.21.15.29:0/3590209692 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1f74010460 con 0x7f1f80101440 2024-04-27T02:04:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.834+0000 7f1f7f7fe700 1 -- 172.21.15.29:0/3590209692 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f7400f5f0 con 0x7f1f80101440 2024-04-27T02:04:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.834+0000 7f1f884c0700 1 -- 172.21.15.29:0/3590209692 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f8019db30 con 0x7f1f80101440 2024-04-27T02:04:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.835+0000 7f1f884c0700 1 -- 172.21.15.29:0/3590209692 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f80108d90 con 0x7f1f80101440 2024-04-27T02:04:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.836+0000 7f1f7f7fe700 1 -- 172.21.15.29:0/3590209692 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f1f74009a00 con 0x7f1f80101440 2024-04-27T02:04:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.836+0000 7f1f7f7fe700 1 --2- 172.21.15.29:0/3590209692 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1f6806c680 0x7f1f6806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.836+0000 7f1f7f7fe700 1 -- 172.21.15.29:0/3590209692 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f1f7408b420 con 0x7f1f80101440 2024-04-27T02:04:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.836+0000 7f1f8625c700 1 --2- 172.21.15.29:0/3590209692 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1f6806c680 0x7f1f6806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.836+0000 7f1f8625c700 1 --2- 172.21.15.29:0/3590209692 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1f6806c680 0x7f1f6806eb30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1f78006010 tx=0x7f1f7800b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:02.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:02.840+0000 7f1f7f7fe700 1 -- 172.21.15.29:0/3590209692 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1f740598d0 con 0x7f1f80101440 2024-04-27T02:04:03.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:03.049+0000 7f1f884c0700 1 -- 172.21.15.29:0/3590209692 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f80105d90 con 0x7f1f6806c680 2024-04-27T02:04:03.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:03.051+0000 7f1f7f7fe700 1 -- 172.21.15.29:0/3590209692 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+145 (secure 0 0 0) 0x7f1f80105d90 con 0x7f1f6806c680 2024-04-27T02:04:03.031 INFO:teuthology.orchestra.run.smithi029.stdout:HOST ADDR LABELS STATUS 2024-04-27T02:04:03.031 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 172.21.15.29 2024-04-27T02:04:03.031 INFO:teuthology.orchestra.run.smithi029.stdout:smithi052 172.21.15.52 2024-04-27T02:04:03.031 INFO:teuthology.orchestra.run.smithi029.stdout:2 hosts in cluster 2024-04-27T02:04:03.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:03.053+0000 7f1f884c0700 1 -- 172.21.15.29:0/3590209692 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1f6806c680 msgr2=0x7f1f6806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:03.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:03.053+0000 7f1f884c0700 1 --2- 172.21.15.29:0/3590209692 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1f6806c680 0x7f1f6806eb30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1f78006010 tx=0x7f1f7800b540 comp rx=0 tx=0).stop 2024-04-27T02:04:03.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:03.053+0000 7f1f884c0700 1 -- 172.21.15.29:0/3590209692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80101440 msgr2=0x7f1f80073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:03.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:03.053+0000 7f1f884c0700 1 --2- 172.21.15.29:0/3590209692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80101440 0x7f1f80073140 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1f7400d8d0 tx=0x7f1f7400dbe0 comp rx=0 tx=0).stop 2024-04-27T02:04:03.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:03.053+0000 7f1f884c0700 1 -- 172.21.15.29:0/3590209692 shutdown_connections 2024-04-27T02:04:03.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:03.053+0000 7f1f884c0700 1 --2- 172.21.15.29:0/3590209692 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1f6806c680 0x7f1f6806eb30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:03.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:03.053+0000 7f1f884c0700 1 --2- 172.21.15.29:0/3590209692 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f80100240 0x7f1f80074af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:03.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:03.053+0000 7f1f884c0700 1 --2- 172.21.15.29:0/3590209692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80101440 0x7f1f80073140 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:03.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:03.053+0000 7f1f884c0700 1 -- 172.21.15.29:0/3590209692 >> 172.21.15.29:0/3590209692 conn(0x7f1f800fb7f0 msgr2=0x7f1f80104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:03.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:03.053+0000 7f1f884c0700 1 -- 172.21.15.29:0/3590209692 shutdown_connections 2024-04-27T02:04:03.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:03.053+0000 7f1f884c0700 1 -- 172.21.15.29:0/3590209692 wait complete. 2024-04-27T02:04:03.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:02 smithi052 ceph-mon[32586]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:04:03.341 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:02 smithi029 ceph-mon[26769]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:04:03.711 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-27T02:04:03.927 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:04:04.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:03 smithi052 ceph-mon[32586]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:04:04.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:03 smithi052 ceph-mon[32586]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:04:04.300 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:03 smithi029 ceph-mon[26769]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:04:04.301 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:03 smithi029 ceph-mon[26769]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:04:05.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.594+0000 7feb2bceb700 1 -- 172.21.15.29:0/1928955930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb24100340 msgr2=0x7feb24100790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:05.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.594+0000 7feb2bceb700 1 --2- 172.21.15.29:0/1928955930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb24100340 0x7feb24100790 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7feb20009910 tx=0x7feb20009c20 comp rx=0 tx=0).stop 2024-04-27T02:04:05.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.594+0000 7feb2bceb700 1 -- 172.21.15.29:0/1928955930 shutdown_connections 2024-04-27T02:04:05.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.594+0000 7feb2bceb700 1 --2- 172.21.15.29:0/1928955930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb24100340 0x7feb24100790 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:05.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.594+0000 7feb2bceb700 1 --2- 172.21.15.29:0/1928955930 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb240ff140 0x7feb240ff550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:05.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.594+0000 7feb2bceb700 1 -- 172.21.15.29:0/1928955930 >> 172.21.15.29:0/1928955930 conn(0x7feb240fa6d0 msgr2=0x7feb240fcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:05.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.595+0000 7feb2bceb700 1 -- 172.21.15.29:0/1928955930 shutdown_connections 2024-04-27T02:04:05.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.595+0000 7feb2bceb700 1 -- 172.21.15.29:0/1928955930 wait complete. 2024-04-27T02:04:05.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.595+0000 7feb2bceb700 1 Processor -- start 2024-04-27T02:04:05.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.595+0000 7feb2bceb700 1 -- start start 2024-04-27T02:04:05.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.595+0000 7feb2bceb700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb240ff140 0x7feb241030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:05.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.595+0000 7feb2bceb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb24100340 0x7feb24101720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:05.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.595+0000 7feb2bceb700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb24101c60 con 0x7feb24100340 2024-04-27T02:04:05.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.595+0000 7feb2bceb700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb24101da0 con 0x7feb240ff140 2024-04-27T02:04:05.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.595+0000 7feb29286700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb24100340 0x7feb24101720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:05.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.596+0000 7feb29286700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb24100340 0x7feb24101720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45446/0 (socket says 172.21.15.29:45446) 2024-04-27T02:04:05.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.596+0000 7feb29286700 1 -- 172.21.15.29:0/3026224187 learned_addr learned my addr 172.21.15.29:0/3026224187 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:04:05.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.596+0000 7feb29a87700 1 --2- 172.21.15.29:0/3026224187 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb240ff140 0x7feb241030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:05.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.596+0000 7feb29286700 1 -- 172.21.15.29:0/3026224187 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb240ff140 msgr2=0x7feb241030d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:05.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.596+0000 7feb29286700 1 --2- 172.21.15.29:0/3026224187 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb240ff140 0x7feb241030d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:05.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.596+0000 7feb29286700 1 -- 172.21.15.29:0/3026224187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb200095f0 con 0x7feb24100340 2024-04-27T02:04:05.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.596+0000 7feb29286700 1 --2- 172.21.15.29:0/3026224187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb24100340 0x7feb24101720 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7feb2000bd70 tx=0x7feb2000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:05.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.596+0000 7feb16ffd700 1 -- 172.21.15.29:0/3026224187 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb2001d070 con 0x7feb24100340 2024-04-27T02:04:05.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.596+0000 7feb2bceb700 1 -- 172.21.15.29:0/3026224187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb24102020 con 0x7feb24100340 2024-04-27T02:04:05.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.596+0000 7feb16ffd700 1 -- 172.21.15.29:0/3026224187 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7feb20022470 con 0x7feb24100340 2024-04-27T02:04:05.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.596+0000 7feb16ffd700 1 -- 172.21.15.29:0/3026224187 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb2000f670 con 0x7feb24100340 2024-04-27T02:04:05.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.596+0000 7feb2bceb700 1 -- 172.21.15.29:0/3026224187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb24102510 con 0x7feb24100340 2024-04-27T02:04:05.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.597+0000 7feb2bceb700 1 -- 172.21.15.29:0/3026224187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb24107c90 con 0x7feb24100340 2024-04-27T02:04:05.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.598+0000 7feb16ffd700 1 -- 172.21.15.29:0/3026224187 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7feb20022a50 con 0x7feb24100340 2024-04-27T02:04:05.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.598+0000 7feb16ffd700 1 --2- 172.21.15.29:0/3026224187 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7feb1006c6d0 0x7feb1006eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:05.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.598+0000 7feb16ffd700 1 -- 172.21.15.29:0/3026224187 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7feb2008cd60 con 0x7feb24100340 2024-04-27T02:04:05.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.598+0000 7feb29a87700 1 --2- 172.21.15.29:0/3026224187 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7feb1006c6d0 0x7feb1006eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:05.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.599+0000 7feb29a87700 1 --2- 172.21.15.29:0/3026224187 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7feb1006c6d0 0x7feb1006eb80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7feb241001a0 tx=0x7feb18005d70 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:05.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.603+0000 7feb16ffd700 1 -- 172.21.15.29:0/3026224187 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7feb2005b170 con 0x7feb24100340 2024-04-27T02:04:05.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.809+0000 7feb2bceb700 1 -- 172.21.15.29:0/3026224187 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7feb24069520 con 0x7feb1006c6d0 2024-04-27T02:04:05.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.811+0000 7feb16ffd700 1 -- 172.21.15.29:0/3026224187 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7feb24069520 con 0x7feb1006c6d0 2024-04-27T02:04:05.791 INFO:teuthology.orchestra.run.smithi029.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-27T02:04:05.791 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7512009B480DGN 447G No 2m ago LVM detected, locked 2024-04-27T02:04:05.791 INFO:teuthology.orchestra.run.smithi029.stdout:smithi052 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513010N480DGN 447G No 75s ago LVM detected, locked 2024-04-27T02:04:05.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.813+0000 7feb2bceb700 1 -- 172.21.15.29:0/3026224187 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7feb1006c6d0 msgr2=0x7feb1006eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:05.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.813+0000 7feb2bceb700 1 --2- 172.21.15.29:0/3026224187 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7feb1006c6d0 0x7feb1006eb80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7feb241001a0 tx=0x7feb18005d70 comp rx=0 tx=0).stop 2024-04-27T02:04:05.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.813+0000 7feb2bceb700 1 -- 172.21.15.29:0/3026224187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb24100340 msgr2=0x7feb24101720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:05.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.813+0000 7feb2bceb700 1 --2- 172.21.15.29:0/3026224187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb24100340 0x7feb24101720 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7feb2000bd70 tx=0x7feb2000be50 comp rx=0 tx=0).stop 2024-04-27T02:04:05.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.813+0000 7feb2bceb700 1 -- 172.21.15.29:0/3026224187 shutdown_connections 2024-04-27T02:04:05.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.813+0000 7feb2bceb700 1 --2- 172.21.15.29:0/3026224187 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7feb1006c6d0 0x7feb1006eb80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:05.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.813+0000 7feb2bceb700 1 --2- 172.21.15.29:0/3026224187 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb240ff140 0x7feb241030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:05.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.813+0000 7feb2bceb700 1 --2- 172.21.15.29:0/3026224187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb24100340 0x7feb24101720 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:05.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.813+0000 7feb2bceb700 1 -- 172.21.15.29:0/3026224187 >> 172.21.15.29:0/3026224187 conn(0x7feb240fa6d0 msgr2=0x7feb24068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:05.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.813+0000 7feb2bceb700 1 -- 172.21.15.29:0/3026224187 shutdown_connections 2024-04-27T02:04:05.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:05.813+0000 7feb2bceb700 1 -- 172.21.15.29:0/3026224187 wait complete. 2024-04-27T02:04:06.416 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T02:04:06.428 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi029.front.sepia.ceph.com 2024-04-27T02:04:06.428 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-04-27T02:04:06.642 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:04:07.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:06 smithi029 ceph-mon[26769]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:04:07.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:06 smithi029 ceph-mon[26769]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:04:07.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:06 smithi052 ceph-mon[32586]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:04:07.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:06 smithi052 ceph-mon[32586]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:04:08.056 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:07 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:04:08.155 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:07 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:04:08.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.301+0000 7f4fc185a700 1 -- 172.21.15.29:0/2319038763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fbc100240 msgr2=0x7f4fbc100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:08.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.301+0000 7f4fc185a700 1 --2- 172.21.15.29:0/2319038763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fbc100240 0x7f4fbc100650 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4fac009910 tx=0x7f4fac009c20 comp rx=0 tx=0).stop 2024-04-27T02:04:08.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.301+0000 7f4fc185a700 1 -- 172.21.15.29:0/2319038763 shutdown_connections 2024-04-27T02:04:08.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.301+0000 7f4fc185a700 1 --2- 172.21.15.29:0/2319038763 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fbc101440 0x7f4fbc101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:08.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.301+0000 7f4fc185a700 1 --2- 172.21.15.29:0/2319038763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fbc100240 0x7f4fbc100650 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:08.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.301+0000 7f4fc185a700 1 -- 172.21.15.29:0/2319038763 >> 172.21.15.29:0/2319038763 conn(0x7f4fbc0fb7f0 msgr2=0x7f4fbc0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:08.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.301+0000 7f4fc185a700 1 -- 172.21.15.29:0/2319038763 shutdown_connections 2024-04-27T02:04:08.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.301+0000 7f4fc185a700 1 -- 172.21.15.29:0/2319038763 wait complete. 2024-04-27T02:04:08.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.302+0000 7f4fc185a700 1 Processor -- start 2024-04-27T02:04:08.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.302+0000 7f4fc185a700 1 -- start start 2024-04-27T02:04:08.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.302+0000 7f4fc185a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fbc100240 0x7f4fbc195b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:08.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.302+0000 7f4fc185a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fbc101440 0x7f4fbc1960a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:08.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.302+0000 7f4fc185a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc1966c0 con 0x7f4fbc100240 2024-04-27T02:04:08.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.302+0000 7f4fc185a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc196800 con 0x7f4fbc101440 2024-04-27T02:04:08.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.302+0000 7f4fbaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fbc100240 0x7f4fbc195b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:08.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.302+0000 7f4fbaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fbc100240 0x7f4fbc195b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48990/0 (socket says 172.21.15.29:48990) 2024-04-27T02:04:08.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.302+0000 7f4fbaffd700 1 -- 172.21.15.29:0/2931973327 learned_addr learned my addr 172.21.15.29:0/2931973327 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:04:08.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.302+0000 7f4fb3fff700 1 --2- 172.21.15.29:0/2931973327 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fbc101440 0x7f4fbc1960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:08.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.302+0000 7f4fbaffd700 1 -- 172.21.15.29:0/2931973327 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fbc101440 msgr2=0x7f4fbc1960a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:08.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.302+0000 7f4fbaffd700 1 --2- 172.21.15.29:0/2931973327 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fbc101440 0x7f4fbc1960a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:08.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.302+0000 7f4fbaffd700 1 -- 172.21.15.29:0/2931973327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fac0095f0 con 0x7f4fbc100240 2024-04-27T02:04:08.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.302+0000 7f4fbaffd700 1 --2- 172.21.15.29:0/2931973327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fbc100240 0x7f4fbc195b60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4fac00bda0 tx=0x7f4fac00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:08.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.303+0000 7f4fc0858700 1 -- 172.21.15.29:0/2931973327 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fac01d070 con 0x7f4fbc100240 2024-04-27T02:04:08.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.303+0000 7f4fc0858700 1 -- 172.21.15.29:0/2931973327 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4fac004c50 con 0x7f4fbc100240 2024-04-27T02:04:08.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.303+0000 7f4fc185a700 1 -- 172.21.15.29:0/2931973327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fbc19b250 con 0x7f4fbc100240 2024-04-27T02:04:08.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.303+0000 7f4fc0858700 1 -- 172.21.15.29:0/2931973327 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fac00f690 con 0x7f4fbc100240 2024-04-27T02:04:08.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.303+0000 7f4fc185a700 1 -- 172.21.15.29:0/2931973327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fbc19b740 con 0x7f4fbc100240 2024-04-27T02:04:08.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.304+0000 7f4fc185a700 1 -- 172.21.15.29:0/2931973327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fbc108e10 con 0x7f4fbc100240 2024-04-27T02:04:08.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.304+0000 7f4fc0858700 1 -- 172.21.15.29:0/2931973327 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f4fac00f900 con 0x7f4fbc100240 2024-04-27T02:04:08.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.305+0000 7f4fc0858700 1 --2- 172.21.15.29:0/2931973327 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f4fa806c680 0x7f4fa806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:08.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.305+0000 7f4fb3fff700 1 --2- 172.21.15.29:0/2931973327 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f4fa806c680 0x7f4fa806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:08.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.305+0000 7f4fc0858700 1 -- 172.21.15.29:0/2931973327 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f4fac08cdf0 con 0x7f4fbc100240 2024-04-27T02:04:08.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.305+0000 7f4fb3fff700 1 --2- 172.21.15.29:0/2931973327 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f4fa806c680 0x7f4fa806eb30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4006fd0 tx=0x7f4fa4008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:08.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.310+0000 7f4fc0858700 1 -- 172.21.15.29:0/2931973327 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4fac05b200 con 0x7f4fbc100240 2024-04-27T02:04:08.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:08.540+0000 7f4fc185a700 1 -- 172.21.15.29:0/2931973327 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f4fbc19bd80 con 0x7f4fa806c680 2024-04-27T02:04:09.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:08 smithi052 ceph-mon[32586]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:04:09.450 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:08 smithi029 ceph-mon[26769]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:04:10.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:10.059+0000 7f4fc0858700 1 -- 172.21.15.29:0/2931973327 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4fbc19bd80 con 0x7f4fa806c680 2024-04-27T02:04:10.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:10.062+0000 7f4fc185a700 1 -- 172.21.15.29:0/2931973327 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f4fa806c680 msgr2=0x7f4fa806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:10.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:10.062+0000 7f4fc185a700 1 --2- 172.21.15.29:0/2931973327 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f4fa806c680 0x7f4fa806eb30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4006fd0 tx=0x7f4fa4008040 comp rx=0 tx=0).stop 2024-04-27T02:04:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:10.062+0000 7f4fc185a700 1 -- 172.21.15.29:0/2931973327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fbc100240 msgr2=0x7f4fbc195b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:10.062+0000 7f4fc185a700 1 --2- 172.21.15.29:0/2931973327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fbc100240 0x7f4fbc195b60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4fac00bda0 tx=0x7f4fac00be80 comp rx=0 tx=0).stop 2024-04-27T02:04:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:10.062+0000 7f4fc185a700 1 -- 172.21.15.29:0/2931973327 shutdown_connections 2024-04-27T02:04:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:10.062+0000 7f4fc185a700 1 --2- 172.21.15.29:0/2931973327 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f4fa806c680 0x7f4fa806eb30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:10.062+0000 7f4fc185a700 1 --2- 172.21.15.29:0/2931973327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fbc100240 0x7f4fbc195b60 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:10.062+0000 7f4fc185a700 1 --2- 172.21.15.29:0/2931973327 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fbc101440 0x7f4fbc1960a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:10.062+0000 7f4fc185a700 1 -- 172.21.15.29:0/2931973327 >> 172.21.15.29:0/2931973327 conn(0x7f4fbc0fb7f0 msgr2=0x7f4fbc104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:10.062+0000 7f4fc185a700 1 -- 172.21.15.29:0/2931973327 shutdown_connections 2024-04-27T02:04:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:10.062+0000 7f4fc185a700 1 -- 172.21.15.29:0/2931973327 wait complete. 2024-04-27T02:04:10.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:09 smithi052 ceph-mon[32586]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:04:10.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:09 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-27T02:04:10.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:09 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-27T02:04:10.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:09 smithi052 ceph-mon[32586]: osdmap e41: 6 total, 6 up, 6 in 2024-04-27T02:04:10.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:09 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-27T02:04:10.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:09 smithi052 ceph-mon[32586]: pgmap v180: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:04:10.422 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:09 smithi029 ceph-mon[26769]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:04:10.423 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:09 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-27T02:04:10.423 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:09 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-27T02:04:10.423 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:09 smithi029 ceph-mon[26769]: osdmap e41: 6 total, 6 up, 6 in 2024-04-27T02:04:10.423 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:09 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-27T02:04:10.423 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:09 smithi029 ceph-mon[26769]: pgmap v180: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:04:10.423 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:04:10.034+0000 7f35f6560700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T02:04:10.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-27T02:04:11.358 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:11 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-27T02:04:11.358 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:11 smithi029 ceph-mon[26769]: osdmap e42: 6 total, 6 up, 6 in 2024-04-27T02:04:11.358 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:11 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-27T02:04:11.358 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:11 smithi029 ceph-mon[26769]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T02:04:11.358 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:11 smithi029 ceph-mon[26769]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-27T02:04:11.358 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:11 smithi029 ceph-mon[26769]: osdmap e43: 6 total, 6 up, 6 in 2024-04-27T02:04:11.358 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:11 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-27T02:04:11.359 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:11 smithi029 ceph-mon[26769]: fsmap cephfs:0 2024-04-27T02:04:11.359 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:11 smithi029 ceph-mon[26769]: Saving service mds.cephfs spec with placement count:4 2024-04-27T02:04:11.359 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:11 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:11.359 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:11 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:04:11.359 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:11 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:04:11.359 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:11 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:04:11.359 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:11 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:11.359 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:11 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi029.dyrjrd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T02:04:11.359 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:11 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi029.dyrjrd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T02:04:11.359 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:11 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:04:11.359 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:11 smithi029 ceph-mon[26769]: Deploying daemon mds.cephfs.smithi029.dyrjrd on smithi029 2024-04-27T02:04:11.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:11 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-27T02:04:11.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:11 smithi052 ceph-mon[32586]: osdmap e42: 6 total, 6 up, 6 in 2024-04-27T02:04:11.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:11 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-27T02:04:11.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:11 smithi052 ceph-mon[32586]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T02:04:11.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:11 smithi052 ceph-mon[32586]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-27T02:04:11.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:11 smithi052 ceph-mon[32586]: osdmap e43: 6 total, 6 up, 6 in 2024-04-27T02:04:11.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:11 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-27T02:04:11.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:11 smithi052 ceph-mon[32586]: fsmap cephfs:0 2024-04-27T02:04:11.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:11 smithi052 ceph-mon[32586]: Saving service mds.cephfs spec with placement count:4 2024-04-27T02:04:11.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:11 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:11.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:11 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:04:11.505 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:11 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:04:11.505 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:11 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:04:11.505 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:11 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:11.505 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:11 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi029.dyrjrd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T02:04:11.505 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:11 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi029.dyrjrd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T02:04:11.505 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:11 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:04:11.505 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:11 smithi052 ceph-mon[32586]: Deploying daemon mds.cephfs.smithi029.dyrjrd on smithi029 2024-04-27T02:04:12.126 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:04:12.249 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:12 smithi029 ceph-mon[26769]: osdmap e44: 6 total, 6 up, 6 in 2024-04-27T02:04:12.249 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:12 smithi029 ceph-mon[26769]: pgmap v184: 65 pgs: 10 active+clean, 16 creating+peering, 39 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:04:12.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:12 smithi052 ceph-mon[32586]: osdmap e44: 6 total, 6 up, 6 in 2024-04-27T02:04:12.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:12 smithi052 ceph-mon[32586]: pgmap v184: 65 pgs: 10 active+clean, 16 creating+peering, 39 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:04:13.400 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:13 smithi029 ceph-mon[26769]: osdmap e45: 6 total, 6 up, 6 in 2024-04-27T02:04:13.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:13 smithi052 ceph-mon[32586]: osdmap e45: 6 total, 6 up, 6 in 2024-04-27T02:04:14.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.198+0000 7f19dda21700 1 -- 172.21.15.29:0/3580118028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19d8100ce0 msgr2=0x7f19d81030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:14.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.198+0000 7f19dda21700 1 --2- 172.21.15.29:0/3580118028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19d8100ce0 0x7f19d81030c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f19c800b0a0 tx=0x7f19c800b3b0 comp rx=0 tx=0).stop 2024-04-27T02:04:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.198+0000 7f19dda21700 1 -- 172.21.15.29:0/3580118028 shutdown_connections 2024-04-27T02:04:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.198+0000 7f19dda21700 1 --2- 172.21.15.29:0/3580118028 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f19d8103600 0x7f19d81059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.198+0000 7f19dda21700 1 --2- 172.21.15.29:0/3580118028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19d8100ce0 0x7f19d81030c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.198+0000 7f19dda21700 1 -- 172.21.15.29:0/3580118028 >> 172.21.15.29:0/3580118028 conn(0x7f19d80fa700 msgr2=0x7f19d80fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.198+0000 7f19dda21700 1 -- 172.21.15.29:0/3580118028 shutdown_connections 2024-04-27T02:04:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.198+0000 7f19dda21700 1 -- 172.21.15.29:0/3580118028 wait complete. 2024-04-27T02:04:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.199+0000 7f19dda21700 1 Processor -- start 2024-04-27T02:04:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.199+0000 7f19dda21700 1 -- start start 2024-04-27T02:04:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.199+0000 7f19dda21700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19d8100ce0 0x7f19d8100640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.199+0000 7f19dda21700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f19d8103600 0x7f19d80fec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.199+0000 7f19dda21700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d80ff1d0 con 0x7f19d8100ce0 2024-04-27T02:04:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.199+0000 7f19dda21700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d80ff310 con 0x7f19d8103600 2024-04-27T02:04:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.199+0000 7f19d6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19d8100ce0 0x7f19d8100640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.199+0000 7f19d6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19d8100ce0 0x7f19d8100640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49028/0 (socket says 172.21.15.29:49028) 2024-04-27T02:04:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.199+0000 7f19d6ffd700 1 -- 172.21.15.29:0/2837961104 learned_addr learned my addr 172.21.15.29:0/2837961104 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:04:14.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.199+0000 7f19d67fc700 1 --2- 172.21.15.29:0/2837961104 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f19d8103600 0x7f19d80fec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:14.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.200+0000 7f19d6ffd700 1 -- 172.21.15.29:0/2837961104 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f19d8103600 msgr2=0x7f19d80fec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:14.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.200+0000 7f19d6ffd700 1 --2- 172.21.15.29:0/2837961104 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f19d8103600 0x7f19d80fec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:14.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.200+0000 7f19d6ffd700 1 -- 172.21.15.29:0/2837961104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c8009d00 con 0x7f19d8100ce0 2024-04-27T02:04:14.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.200+0000 7f19d67fc700 1 --2- 172.21.15.29:0/2837961104 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f19d8103600 0x7f19d80fec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T02:04:14.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.200+0000 7f19d6ffd700 1 --2- 172.21.15.29:0/2837961104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19d8100ce0 0x7f19d8100640 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f19c8009fd0 tx=0x7f19c80096e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:14.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.200+0000 7f19dca1f700 1 -- 172.21.15.29:0/2837961104 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19c800e040 con 0x7f19d8100ce0 2024-04-27T02:04:14.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.200+0000 7f19dda21700 1 -- 172.21.15.29:0/2837961104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d80ff590 con 0x7f19d8100ce0 2024-04-27T02:04:14.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.200+0000 7f19dca1f700 1 -- 172.21.15.29:0/2837961104 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f19c8003de0 con 0x7f19d8100ce0 2024-04-27T02:04:14.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.200+0000 7f19dca1f700 1 -- 172.21.15.29:0/2837961104 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19c801d910 con 0x7f19d8100ce0 2024-04-27T02:04:14.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.200+0000 7f19dda21700 1 -- 172.21.15.29:0/2837961104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d80ffa80 con 0x7f19d8100ce0 2024-04-27T02:04:14.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.201+0000 7f19dda21700 1 -- 172.21.15.29:0/2837961104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d818fc70 con 0x7f19d8100ce0 2024-04-27T02:04:14.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.201+0000 7f19dca1f700 1 -- 172.21.15.29:0/2837961104 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f19c801dbd0 con 0x7f19d8100ce0 2024-04-27T02:04:14.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.202+0000 7f19dca1f700 1 --2- 172.21.15.29:0/2837961104 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f19c406c680 0x7f19c406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:14.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.202+0000 7f19dca1f700 1 -- 172.21.15.29:0/2837961104 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f19c808d340 con 0x7f19d8100ce0 2024-04-27T02:04:14.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.202+0000 7f19d67fc700 1 --2- 172.21.15.29:0/2837961104 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f19c406c680 0x7f19c406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:14.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.202+0000 7f19d67fc700 1 --2- 172.21.15.29:0/2837961104 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f19c406c680 0x7f19c406eb30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f19c0005950 tx=0x7f19c000b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:14.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.207+0000 7f19dca1f700 1 -- 172.21.15.29:0/2837961104 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f19c805b4b0 con 0x7f19d8100ce0 2024-04-27T02:04:14.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:14 smithi029 ceph-mon[26769]: pgmap v186: 65 pgs: 37 active+clean, 16 creating+peering, 12 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:04:14.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:14 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:14.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:14 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:14.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.452+0000 7f19dda21700 1 -- 172.21.15.29:0/2837961104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f19d802cc30 con 0x7f19d8100ce0 2024-04-27T02:04:14.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.452+0000 7f19dca1f700 1 -- 172.21.15.29:0/2837961104 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 4 v4) v1 ==== 75+0+1264 (secure 0 0 0) 0x7f19c805b040 con 0x7f19d8100ce0 2024-04-27T02:04:14.432 INFO:teuthology.orchestra.run.smithi029.stdout:e4 2024-04-27T02:04:14.433 INFO:teuthology.orchestra.run.smithi029.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T02:04:14.433 INFO:teuthology.orchestra.run.smithi029.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T02:04:14.433 INFO:teuthology.orchestra.run.smithi029.stdout:legacy client fscid: 1 2024-04-27T02:04:14.433 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:04:14.433 INFO:teuthology.orchestra.run.smithi029.stdout:Filesystem 'cephfs' (1) 2024-04-27T02:04:14.433 INFO:teuthology.orchestra.run.smithi029.stdout:fs_name cephfs 2024-04-27T02:04:14.433 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 4 2024-04-27T02:04:14.433 INFO:teuthology.orchestra.run.smithi029.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-27T02:04:14.433 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-04-27T02:04:10.035536+0000 2024-04-27T02:04:14.433 INFO:teuthology.orchestra.run.smithi029.stdout:modified 2024-04-27T02:04:14.058864+0000 2024-04-27T02:04:14.433 INFO:teuthology.orchestra.run.smithi029.stdout:tableserver 0 2024-04-27T02:04:14.433 INFO:teuthology.orchestra.run.smithi029.stdout:root 0 2024-04-27T02:04:14.433 INFO:teuthology.orchestra.run.smithi029.stdout:session_timeout 60 2024-04-27T02:04:14.434 INFO:teuthology.orchestra.run.smithi029.stdout:session_autoclose 300 2024-04-27T02:04:14.434 INFO:teuthology.orchestra.run.smithi029.stdout:max_file_size 1099511627776 2024-04-27T02:04:14.434 INFO:teuthology.orchestra.run.smithi029.stdout:required_client_features {} 2024-04-27T02:04:14.434 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure 0 2024-04-27T02:04:14.434 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure_osd_epoch 0 2024-04-27T02:04:14.434 INFO:teuthology.orchestra.run.smithi029.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T02:04:14.434 INFO:teuthology.orchestra.run.smithi029.stdout:max_mds 1 2024-04-27T02:04:14.434 INFO:teuthology.orchestra.run.smithi029.stdout:in 0 2024-04-27T02:04:14.434 INFO:teuthology.orchestra.run.smithi029.stdout:up {0=14468} 2024-04-27T02:04:14.434 INFO:teuthology.orchestra.run.smithi029.stdout:failed 2024-04-27T02:04:14.434 INFO:teuthology.orchestra.run.smithi029.stdout:damaged 2024-04-27T02:04:14.434 INFO:teuthology.orchestra.run.smithi029.stdout:stopped 2024-04-27T02:04:14.434 INFO:teuthology.orchestra.run.smithi029.stdout:data_pools [3] 2024-04-27T02:04:14.434 INFO:teuthology.orchestra.run.smithi029.stdout:metadata_pool 2 2024-04-27T02:04:14.434 INFO:teuthology.orchestra.run.smithi029.stdout:inline_data disabled 2024-04-27T02:04:14.435 INFO:teuthology.orchestra.run.smithi029.stdout:balancer 2024-04-27T02:04:14.435 INFO:teuthology.orchestra.run.smithi029.stdout:bal_rank_mask -1 2024-04-27T02:04:14.435 INFO:teuthology.orchestra.run.smithi029.stdout:standby_count_wanted 0 2024-04-27T02:04:14.435 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi029.dyrjrd{0:14468} state up:creating seq 1 addr [v2:172.21.15.29:6826/2380229866,v1:172.21.15.29:6827/2380229866] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T02:04:14.435 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:04:14.435 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:04:14.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.455+0000 7f19dda21700 1 -- 172.21.15.29:0/2837961104 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f19c406c680 msgr2=0x7f19c406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:14.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.455+0000 7f19dda21700 1 --2- 172.21.15.29:0/2837961104 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f19c406c680 0x7f19c406eb30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f19c0005950 tx=0x7f19c000b410 comp rx=0 tx=0).stop 2024-04-27T02:04:14.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.455+0000 7f19dda21700 1 -- 172.21.15.29:0/2837961104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19d8100ce0 msgr2=0x7f19d8100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:14.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.455+0000 7f19dda21700 1 --2- 172.21.15.29:0/2837961104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19d8100ce0 0x7f19d8100640 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f19c8009fd0 tx=0x7f19c80096e0 comp rx=0 tx=0).stop 2024-04-27T02:04:14.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.455+0000 7f19dda21700 1 -- 172.21.15.29:0/2837961104 shutdown_connections 2024-04-27T02:04:14.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.455+0000 7f19dda21700 1 --2- 172.21.15.29:0/2837961104 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f19c406c680 0x7f19c406eb30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:14.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.455+0000 7f19dda21700 1 --2- 172.21.15.29:0/2837961104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19d8100ce0 0x7f19d8100640 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:14.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.455+0000 7f19dda21700 1 --2- 172.21.15.29:0/2837961104 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f19d8103600 0x7f19d80fec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:14.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.455+0000 7f19dda21700 1 -- 172.21.15.29:0/2837961104 >> 172.21.15.29:0/2837961104 conn(0x7f19d80fa700 msgr2=0x7f19d80fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:14.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.455+0000 7f19dda21700 1 -- 172.21.15.29:0/2837961104 shutdown_connections 2024-04-27T02:04:14.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:14.455+0000 7f19dda21700 1 -- 172.21.15.29:0/2837961104 wait complete. 2024-04-27T02:04:14.436 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 4 2024-04-27T02:04:14.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:14 smithi052 ceph-mon[32586]: pgmap v186: 65 pgs: 37 active+clean, 16 creating+peering, 12 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:04:14.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:14 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:14.504 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:14 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:15.072 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T02:04:15.083 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi029.front.sepia.ceph.com 2024-04-27T02:04:15.083 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- bash -c 'ceph fs set cephfs max_mds 1' 2024-04-27T02:04:15.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:15 smithi052 ceph-mon[32586]: mds.? [v2:172.21.15.29:6826/2380229866,v1:172.21.15.29:6827/2380229866] up:boot 2024-04-27T02:04:15.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:15 smithi052 ceph-mon[32586]: daemon mds.cephfs.smithi029.dyrjrd assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-27T02:04:15.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:15 smithi052 ceph-mon[32586]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-27T02:04:15.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:15 smithi052 ceph-mon[32586]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-27T02:04:15.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:15 smithi052 ceph-mon[32586]: Cluster is now healthy 2024-04-27T02:04:15.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:15 smithi052 ceph-mon[32586]: fsmap cephfs:0 1 up:standby 2024-04-27T02:04:15.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:15 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi029.dyrjrd"}]: dispatch 2024-04-27T02:04:15.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:15 smithi052 ceph-mon[32586]: fsmap cephfs:1 {0=cephfs.smithi029.dyrjrd=up:creating} 2024-04-27T02:04:15.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:15 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:15.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:15 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi052.yylmkb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T02:04:15.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:15 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi052.yylmkb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T02:04:15.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:15 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:04:15.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:15 smithi052 ceph-mon[32586]: Deploying daemon mds.cephfs.smithi052.yylmkb on smithi052 2024-04-27T02:04:15.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:15 smithi052 ceph-mon[32586]: daemon mds.cephfs.smithi029.dyrjrd is now active in filesystem cephfs as rank 0 2024-04-27T02:04:15.255 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:15 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/2837961104' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T02:04:15.300 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:04:15.416 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:15 smithi029 ceph-mon[26769]: mds.? [v2:172.21.15.29:6826/2380229866,v1:172.21.15.29:6827/2380229866] up:boot 2024-04-27T02:04:15.416 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:15 smithi029 ceph-mon[26769]: daemon mds.cephfs.smithi029.dyrjrd assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-27T02:04:15.416 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:15 smithi029 ceph-mon[26769]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-27T02:04:15.417 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:15 smithi029 ceph-mon[26769]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-27T02:04:15.417 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:15 smithi029 ceph-mon[26769]: Cluster is now healthy 2024-04-27T02:04:15.417 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:15 smithi029 ceph-mon[26769]: fsmap cephfs:0 1 up:standby 2024-04-27T02:04:15.417 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:15 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi029.dyrjrd"}]: dispatch 2024-04-27T02:04:15.417 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:15 smithi029 ceph-mon[26769]: fsmap cephfs:1 {0=cephfs.smithi029.dyrjrd=up:creating} 2024-04-27T02:04:15.417 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:15 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:15.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:15 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi052.yylmkb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T02:04:15.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:15 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi052.yylmkb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T02:04:15.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:15 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:04:15.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:15 smithi029 ceph-mon[26769]: Deploying daemon mds.cephfs.smithi052.yylmkb on smithi052 2024-04-27T02:04:15.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:15 smithi029 ceph-mon[26769]: daemon mds.cephfs.smithi029.dyrjrd is now active in filesystem cephfs as rank 0 2024-04-27T02:04:15.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:15 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/2837961104' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T02:04:16.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:16 smithi052 ceph-mon[32586]: mds.? [v2:172.21.15.29:6826/2380229866,v1:172.21.15.29:6827/2380229866] up:active 2024-04-27T02:04:16.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:16 smithi052 ceph-mon[32586]: fsmap cephfs:1 {0=cephfs.smithi029.dyrjrd=up:active} 2024-04-27T02:04:16.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:16 smithi052 ceph-mon[32586]: pgmap v187: 65 pgs: 59 active+clean, 6 creating+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 2 op/s 2024-04-27T02:04:16.336 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:16 smithi029 ceph-mon[26769]: mds.? [v2:172.21.15.29:6826/2380229866,v1:172.21.15.29:6827/2380229866] up:active 2024-04-27T02:04:16.349 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:16 smithi029 ceph-mon[26769]: fsmap cephfs:1 {0=cephfs.smithi029.dyrjrd=up:active} 2024-04-27T02:04:16.349 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:16 smithi029 ceph-mon[26769]: pgmap v187: 65 pgs: 59 active+clean, 6 creating+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 2 op/s 2024-04-27T02:04:16.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.900+0000 7f3dadede700 1 -- 172.21.15.29:0/1132710012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3da81018a0 msgr2=0x7f3da8101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:16.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.900+0000 7f3dadede700 1 --2- 172.21.15.29:0/1132710012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3da81018a0 0x7f3da8101cb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3d98009910 tx=0x7f3d98009c20 comp rx=0 tx=0).stop 2024-04-27T02:04:16.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.900+0000 7f3dadede700 1 -- 172.21.15.29:0/1132710012 shutdown_connections 2024-04-27T02:04:16.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.900+0000 7f3dadede700 1 --2- 172.21.15.29:0/1132710012 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3da81021f0 0x7f3da80fe320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:16.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.900+0000 7f3dadede700 1 --2- 172.21.15.29:0/1132710012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3da81018a0 0x7f3da8101cb0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:16.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.900+0000 7f3dadede700 1 -- 172.21.15.29:0/1132710012 >> 172.21.15.29:0/1132710012 conn(0x7f3da80f9f10 msgr2=0x7f3da80fc340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:16.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.900+0000 7f3dadede700 1 -- 172.21.15.29:0/1132710012 shutdown_connections 2024-04-27T02:04:16.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.900+0000 7f3dadede700 1 -- 172.21.15.29:0/1132710012 wait complete. 2024-04-27T02:04:16.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.900+0000 7f3dadede700 1 Processor -- start 2024-04-27T02:04:16.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.901+0000 7f3dadede700 1 -- start start 2024-04-27T02:04:16.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.901+0000 7f3dadede700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3da81018a0 0x7f3da8195b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:16.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.901+0000 7f3dadede700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3da81021f0 0x7f3da8196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:16.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.901+0000 7f3dadede700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da8196690 con 0x7f3da81018a0 2024-04-27T02:04:16.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.901+0000 7f3dadede700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da81967d0 con 0x7f3da81021f0 2024-04-27T02:04:16.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.901+0000 7f3da77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3da81018a0 0x7f3da8195b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:16.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.901+0000 7f3da77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3da81018a0 0x7f3da8195b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37894/0 (socket says 172.21.15.29:37894) 2024-04-27T02:04:16.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.901+0000 7f3da77fe700 1 -- 172.21.15.29:0/2752134952 learned_addr learned my addr 172.21.15.29:0/2752134952 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:04:16.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.901+0000 7f3d9effd700 1 --2- 172.21.15.29:0/2752134952 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3da81021f0 0x7f3da8196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:16.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.901+0000 7f3da77fe700 1 -- 172.21.15.29:0/2752134952 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3da81021f0 msgr2=0x7f3da8196070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:16.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.901+0000 7f3da77fe700 1 --2- 172.21.15.29:0/2752134952 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3da81021f0 0x7f3da8196070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:16.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.901+0000 7f3da77fe700 1 -- 172.21.15.29:0/2752134952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d980095f0 con 0x7f3da81018a0 2024-04-27T02:04:16.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.902+0000 7f3da77fe700 1 --2- 172.21.15.29:0/2752134952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3da81018a0 0x7f3da8195b30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3d9800bd70 tx=0x7f3d9800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:16.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.902+0000 7f3da57fa700 1 -- 172.21.15.29:0/2752134952 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d9801d070 con 0x7f3da81018a0 2024-04-27T02:04:16.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.902+0000 7f3da57fa700 1 -- 172.21.15.29:0/2752134952 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3d98005dc0 con 0x7f3da81018a0 2024-04-27T02:04:16.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.902+0000 7f3da57fa700 1 -- 172.21.15.29:0/2752134952 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d98021410 con 0x7f3da81018a0 2024-04-27T02:04:16.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.902+0000 7f3dadede700 1 -- 172.21.15.29:0/2752134952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3da80ff430 con 0x7f3da81018a0 2024-04-27T02:04:16.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.902+0000 7f3dadede700 1 -- 172.21.15.29:0/2752134952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da80ff8d0 con 0x7f3da81018a0 2024-04-27T02:04:16.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.903+0000 7f3dadede700 1 -- 172.21.15.29:0/2752134952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3da8108d30 con 0x7f3da81018a0 2024-04-27T02:04:16.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.904+0000 7f3da57fa700 1 -- 172.21.15.29:0/2752134952 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f3d9800f7d0 con 0x7f3da81018a0 2024-04-27T02:04:16.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.904+0000 7f3da57fa700 1 --2- 172.21.15.29:0/2752134952 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3d9406c680 0x7f3d9406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:16.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.905+0000 7f3da57fa700 1 -- 172.21.15.29:0/2752134952 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f3d9808cb80 con 0x7f3da81018a0 2024-04-27T02:04:16.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.905+0000 7f3d9effd700 1 --2- 172.21.15.29:0/2752134952 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3d9406c680 0x7f3d9406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:16.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.905+0000 7f3d9effd700 1 --2- 172.21.15.29:0/2752134952 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3d9406c680 0x7f3d9406eb30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3d90005950 tx=0x7f3d9000b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:16.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:16.913+0000 7f3da57fa700 1 -- 172.21.15.29:0/2752134952 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3d9805acf0 con 0x7f3da81018a0 2024-04-27T02:04:17.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:17.183+0000 7f3dadede700 1 -- 172.21.15.29:0/2752134952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f3da81002d0 con 0x7f3da81018a0 2024-04-27T02:04:17.700 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:17 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:17.700 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:17 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:17.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:17 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:17.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:17 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi029.njjezc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T02:04:17.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:17 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi029.njjezc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T02:04:17.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:17 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:04:17.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:17 smithi029 ceph-mon[26769]: Deploying daemon mds.cephfs.smithi029.njjezc on smithi029 2024-04-27T02:04:17.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:17 smithi029 ceph-mon[26769]: mds.? [v2:172.21.15.52:6824/2444559071,v1:172.21.15.52:6825/2444559071] up:boot 2024-04-27T02:04:17.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:17 smithi029 ceph-mon[26769]: fsmap cephfs:1 {0=cephfs.smithi029.dyrjrd=up:active} 1 up:standby 2024-04-27T02:04:17.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:17 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi052.yylmkb"}]: dispatch 2024-04-27T02:04:17.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:17 smithi029 ceph-mon[26769]: fsmap cephfs:1 {0=cephfs.smithi029.dyrjrd=up:active} 1 up:standby 2024-04-27T02:04:17.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:17 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/2752134952' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-04-27T02:04:17.899 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:17 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:17.899 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:17 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:17.899 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:17 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:17.899 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:17 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi029.njjezc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T02:04:17.899 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:17 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi029.njjezc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T02:04:17.899 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:17 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:04:17.899 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:17 smithi052 ceph-mon[32586]: Deploying daemon mds.cephfs.smithi029.njjezc on smithi029 2024-04-27T02:04:17.900 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:17 smithi052 ceph-mon[32586]: mds.? [v2:172.21.15.52:6824/2444559071,v1:172.21.15.52:6825/2444559071] up:boot 2024-04-27T02:04:17.900 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:17 smithi052 ceph-mon[32586]: fsmap cephfs:1 {0=cephfs.smithi029.dyrjrd=up:active} 1 up:standby 2024-04-27T02:04:17.900 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:17 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi052.yylmkb"}]: dispatch 2024-04-27T02:04:17.900 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:17 smithi052 ceph-mon[32586]: fsmap cephfs:1 {0=cephfs.smithi029.dyrjrd=up:active} 1 up:standby 2024-04-27T02:04:17.900 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:17 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/2752134952' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-04-27T02:04:18.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:18.082+0000 7f3da57fa700 1 -- 172.21.15.29:0/2752134952 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v8) v1 ==== 105+0+0 (secure 0 0 0) 0x7f3d9800f460 con 0x7f3da81018a0 2024-04-27T02:04:18.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:18.085+0000 7f3dadede700 1 -- 172.21.15.29:0/2752134952 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3d9406c680 msgr2=0x7f3d9406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:18.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:18.085+0000 7f3dadede700 1 --2- 172.21.15.29:0/2752134952 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3d9406c680 0x7f3d9406eb30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3d90005950 tx=0x7f3d9000b410 comp rx=0 tx=0).stop 2024-04-27T02:04:18.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:18.085+0000 7f3dadede700 1 -- 172.21.15.29:0/2752134952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3da81018a0 msgr2=0x7f3da8195b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:18.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:18.085+0000 7f3dadede700 1 --2- 172.21.15.29:0/2752134952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3da81018a0 0x7f3da8195b30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3d9800bd70 tx=0x7f3d9800bda0 comp rx=0 tx=0).stop 2024-04-27T02:04:18.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:18.085+0000 7f3dadede700 1 -- 172.21.15.29:0/2752134952 shutdown_connections 2024-04-27T02:04:18.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:18.085+0000 7f3dadede700 1 --2- 172.21.15.29:0/2752134952 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f3d9406c680 0x7f3d9406eb30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:18.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:18.085+0000 7f3dadede700 1 --2- 172.21.15.29:0/2752134952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3da81018a0 0x7f3da8195b30 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:18.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:18.085+0000 7f3dadede700 1 --2- 172.21.15.29:0/2752134952 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3da81021f0 0x7f3da8196070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:18.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:18.085+0000 7f3dadede700 1 -- 172.21.15.29:0/2752134952 >> 172.21.15.29:0/2752134952 conn(0x7f3da80f9f10 msgr2=0x7f3da8104870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:18.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:18.085+0000 7f3dadede700 1 -- 172.21.15.29:0/2752134952 shutdown_connections 2024-04-27T02:04:18.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:18.085+0000 7f3dadede700 1 -- 172.21.15.29:0/2752134952 wait complete. 2024-04-27T02:04:18.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:18 smithi029 ceph-mon[26769]: pgmap v188: 65 pgs: 65 active+clean; 578 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 789 B/s wr, 4 op/s 2024-04-27T02:04:18.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:18 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/2752134952' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-04-27T02:04:18.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:18 smithi029 ceph-mon[26769]: fsmap cephfs:1 {0=cephfs.smithi029.dyrjrd=up:active} 1 up:standby 2024-04-27T02:04:18.977 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T02:04:18.987 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi029.front.sepia.ceph.com 2024-04-27T02:04:18.987 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-04-27T02:04:19.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:18 smithi052 ceph-mon[32586]: pgmap v188: 65 pgs: 65 active+clean; 578 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 789 B/s wr, 4 op/s 2024-04-27T02:04:19.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:18 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/2752134952' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-04-27T02:04:19.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:18 smithi052 ceph-mon[32586]: fsmap cephfs:1 {0=cephfs.smithi029.dyrjrd=up:active} 1 up:standby 2024-04-27T02:04:19.875 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:04:20.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:20 smithi029 ceph-mon[26769]: mds.? [v2:172.21.15.29:6828/1353923457,v1:172.21.15.29:6829/1353923457] up:boot 2024-04-27T02:04:20.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:20 smithi029 ceph-mon[26769]: fsmap cephfs:1 {0=cephfs.smithi029.dyrjrd=up:active} 2 up:standby 2024-04-27T02:04:20.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:20 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi029.njjezc"}]: dispatch 2024-04-27T02:04:20.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:20 smithi029 ceph-mon[26769]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-04-27T02:04:20.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:20 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:20.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:20 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:20.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:20 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:20.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:20 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi052.nbjpjd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T02:04:20.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:20 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi052.nbjpjd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T02:04:20.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:20 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:04:20.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:20 smithi029 ceph-mon[26769]: Deploying daemon mds.cephfs.smithi052.nbjpjd on smithi052 2024-04-27T02:04:21.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:20 smithi052 ceph-mon[32586]: mds.? [v2:172.21.15.29:6828/1353923457,v1:172.21.15.29:6829/1353923457] up:boot 2024-04-27T02:04:21.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:20 smithi052 ceph-mon[32586]: fsmap cephfs:1 {0=cephfs.smithi029.dyrjrd=up:active} 2 up:standby 2024-04-27T02:04:21.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:20 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi029.njjezc"}]: dispatch 2024-04-27T02:04:21.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:20 smithi052 ceph-mon[32586]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-04-27T02:04:21.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:20 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:21.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:20 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:21.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:20 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:21.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:20 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi052.nbjpjd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T02:04:21.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:20 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi052.nbjpjd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T02:04:21.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:20 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:04:21.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:20 smithi052 ceph-mon[32586]: Deploying daemon mds.cephfs.smithi052.nbjpjd on smithi052 2024-04-27T02:04:21.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.523+0000 7f1a492e7700 1 -- 172.21.15.29:0/223414770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a440ff150 msgr2=0x7f1a440ff560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:21.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.523+0000 7f1a492e7700 1 --2- 172.21.15.29:0/223414770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a440ff150 0x7f1a440ff560 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1a3400b0a0 tx=0x7f1a3400b3b0 comp rx=0 tx=0).stop 2024-04-27T02:04:21.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.523+0000 7f1a492e7700 1 -- 172.21.15.29:0/223414770 shutdown_connections 2024-04-27T02:04:21.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.523+0000 7f1a492e7700 1 --2- 172.21.15.29:0/223414770 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a44100350 0x7f1a441007a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:21.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.523+0000 7f1a492e7700 1 --2- 172.21.15.29:0/223414770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a440ff150 0x7f1a440ff560 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:21.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.523+0000 7f1a492e7700 1 -- 172.21.15.29:0/223414770 >> 172.21.15.29:0/223414770 conn(0x7f1a440fa700 msgr2=0x7f1a440fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:21.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.523+0000 7f1a492e7700 1 -- 172.21.15.29:0/223414770 shutdown_connections 2024-04-27T02:04:21.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.523+0000 7f1a492e7700 1 -- 172.21.15.29:0/223414770 wait complete. 2024-04-27T02:04:21.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.524+0000 7f1a492e7700 1 Processor -- start 2024-04-27T02:04:21.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.524+0000 7f1a492e7700 1 -- start start 2024-04-27T02:04:21.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.524+0000 7f1a492e7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a440ff150 0x7f1a441030e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:21.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.524+0000 7f1a492e7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a44100350 0x7f1a44101730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:21.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.524+0000 7f1a492e7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a44101c70 con 0x7f1a440ff150 2024-04-27T02:04:21.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.524+0000 7f1a492e7700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a44101db0 con 0x7f1a44100350 2024-04-27T02:04:21.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.524+0000 7f1a42ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a440ff150 0x7f1a441030e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:21.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.524+0000 7f1a42ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a440ff150 0x7f1a441030e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37942/0 (socket says 172.21.15.29:37942) 2024-04-27T02:04:21.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.524+0000 7f1a42ffd700 1 -- 172.21.15.29:0/1482847633 learned_addr learned my addr 172.21.15.29:0/1482847633 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:04:21.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.524+0000 7f1a427fc700 1 --2- 172.21.15.29:0/1482847633 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a44100350 0x7f1a44101730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:21.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.524+0000 7f1a42ffd700 1 -- 172.21.15.29:0/1482847633 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a44100350 msgr2=0x7f1a44101730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:21.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.524+0000 7f1a42ffd700 1 --2- 172.21.15.29:0/1482847633 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a44100350 0x7f1a44101730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:21.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.524+0000 7f1a42ffd700 1 -- 172.21.15.29:0/1482847633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a34009d00 con 0x7f1a440ff150 2024-04-27T02:04:21.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.525+0000 7f1a42ffd700 1 --2- 172.21.15.29:0/1482847633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a440ff150 0x7f1a441030e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1a3400b070 tx=0x7f1a34003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:21.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.525+0000 7f1a3bfff700 1 -- 172.21.15.29:0/1482847633 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a3400e040 con 0x7f1a440ff150 2024-04-27T02:04:21.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.525+0000 7f1a492e7700 1 -- 172.21.15.29:0/1482847633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a44102030 con 0x7f1a440ff150 2024-04-27T02:04:21.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.525+0000 7f1a3bfff700 1 -- 172.21.15.29:0/1482847633 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1a3400ce60 con 0x7f1a440ff150 2024-04-27T02:04:21.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.525+0000 7f1a3bfff700 1 -- 172.21.15.29:0/1482847633 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a3401da20 con 0x7f1a440ff150 2024-04-27T02:04:21.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.525+0000 7f1a492e7700 1 -- 172.21.15.29:0/1482847633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a44102520 con 0x7f1a440ff150 2024-04-27T02:04:21.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.526+0000 7f1a492e7700 1 -- 172.21.15.29:0/1482847633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a44107d20 con 0x7f1a440ff150 2024-04-27T02:04:21.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.527+0000 7f1a3bfff700 1 -- 172.21.15.29:0/1482847633 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f1a34004250 con 0x7f1a440ff150 2024-04-27T02:04:21.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.527+0000 7f1a3bfff700 1 --2- 172.21.15.29:0/1482847633 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1a3006c680 0x7f1a3006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:21.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.527+0000 7f1a3bfff700 1 -- 172.21.15.29:0/1482847633 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f1a3408d370 con 0x7f1a440ff150 2024-04-27T02:04:21.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.527+0000 7f1a427fc700 1 --2- 172.21.15.29:0/1482847633 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1a3006c680 0x7f1a3006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:21.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.527+0000 7f1a427fc700 1 --2- 172.21.15.29:0/1482847633 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1a3006c680 0x7f1a3006eb30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c009a10 tx=0x7f1a2c009190 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:21.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.532+0000 7f1a3bfff700 1 -- 172.21.15.29:0/1482847633 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1a3405b4e0 con 0x7f1a440ff150 2024-04-27T02:04:21.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.777+0000 7f1a492e7700 1 -- 172.21.15.29:0/1482847633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f1a4419e090 con 0x7f1a440ff150 2024-04-27T02:04:21.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.866+0000 7f1a3bfff700 1 -- 172.21.15.29:0/1482847633 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v10) v1 ==== 121+0+0 (secure 0 0 0) 0x7f1a3405b070 con 0x7f1a440ff150 2024-04-27T02:04:21.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.868+0000 7f1a492e7700 1 -- 172.21.15.29:0/1482847633 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1a3006c680 msgr2=0x7f1a3006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:21.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.868+0000 7f1a492e7700 1 --2- 172.21.15.29:0/1482847633 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1a3006c680 0x7f1a3006eb30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c009a10 tx=0x7f1a2c009190 comp rx=0 tx=0).stop 2024-04-27T02:04:21.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.868+0000 7f1a492e7700 1 -- 172.21.15.29:0/1482847633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a440ff150 msgr2=0x7f1a441030e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:21.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.868+0000 7f1a492e7700 1 --2- 172.21.15.29:0/1482847633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a440ff150 0x7f1a441030e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1a3400b070 tx=0x7f1a34003c30 comp rx=0 tx=0).stop 2024-04-27T02:04:21.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.868+0000 7f1a492e7700 1 -- 172.21.15.29:0/1482847633 shutdown_connections 2024-04-27T02:04:21.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.868+0000 7f1a492e7700 1 --2- 172.21.15.29:0/1482847633 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1a3006c680 0x7f1a3006eb30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:21.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.868+0000 7f1a492e7700 1 --2- 172.21.15.29:0/1482847633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a440ff150 0x7f1a441030e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:21.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.868+0000 7f1a492e7700 1 --2- 172.21.15.29:0/1482847633 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a44100350 0x7f1a44101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:21.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.868+0000 7f1a492e7700 1 -- 172.21.15.29:0/1482847633 >> 172.21.15.29:0/1482847633 conn(0x7f1a440fa700 msgr2=0x7f1a44068dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:21.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.869+0000 7f1a492e7700 1 -- 172.21.15.29:0/1482847633 shutdown_connections 2024-04-27T02:04:21.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:21.869+0000 7f1a492e7700 1 -- 172.21.15.29:0/1482847633 wait complete. 2024-04-27T02:04:22.137 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:21 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/1482847633' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-27T02:04:22.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:21 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/1482847633' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-27T02:04:22.515 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T02:04:22.527 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi029.front.sepia.ceph.com 2024-04-27T02:04:22.528 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-04-27T02:04:22.759 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:04:23.172 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:22 smithi029 ceph-mon[26769]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-27T02:04:23.172 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:22 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/1482847633' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-27T02:04:23.172 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:22 smithi029 ceph-mon[26769]: fsmap cephfs:1 {0=cephfs.smithi029.dyrjrd=up:active} 2 up:standby 2024-04-27T02:04:23.172 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:22 smithi029 ceph-mon[26769]: fsmap cephfs:1 {0=cephfs.smithi029.dyrjrd=up:active} 1 up:standby-replay 1 up:standby 2024-04-27T02:04:23.173 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:22 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:23.173 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:22 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:23.173 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:22 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:23.173 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:22 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:23.173 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:22 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:23.173 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:22 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:04:23.173 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:22 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:04:23.173 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:22 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:23.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:22 smithi052 ceph-mon[32586]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-27T02:04:23.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:22 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/1482847633' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-27T02:04:23.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:22 smithi052 ceph-mon[32586]: fsmap cephfs:1 {0=cephfs.smithi029.dyrjrd=up:active} 2 up:standby 2024-04-27T02:04:23.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:22 smithi052 ceph-mon[32586]: fsmap cephfs:1 {0=cephfs.smithi029.dyrjrd=up:active} 1 up:standby-replay 1 up:standby 2024-04-27T02:04:23.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:22 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:23.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:22 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:23.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:22 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:23.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:22 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:23.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:22 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:23.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:22 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:04:23.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:22 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:04:23.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:22 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:24.112 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:23 smithi052 ceph-mon[32586]: mds.? [v2:172.21.15.52:6826/3513632158,v1:172.21.15.52:6827/3513632158] up:boot 2024-04-27T02:04:24.112 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:23 smithi052 ceph-mon[32586]: fsmap cephfs:1 {0=cephfs.smithi029.dyrjrd=up:active} 1 up:standby-replay 2 up:standby 2024-04-27T02:04:24.112 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:23 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi052.nbjpjd"}]: dispatch 2024-04-27T02:04:24.154 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:23 smithi029 ceph-mon[26769]: mds.? [v2:172.21.15.52:6826/3513632158,v1:172.21.15.52:6827/3513632158] up:boot 2024-04-27T02:04:24.154 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:23 smithi029 ceph-mon[26769]: fsmap cephfs:1 {0=cephfs.smithi029.dyrjrd=up:active} 1 up:standby-replay 2 up:standby 2024-04-27T02:04:24.154 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:23 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi052.nbjpjd"}]: dispatch 2024-04-27T02:04:24.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.551+0000 7f96ca2d4700 1 -- 172.21.15.29:0/670488878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96c4100250 msgr2=0x7f96c4100660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:24.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.551+0000 7f96ca2d4700 1 --2- 172.21.15.29:0/670488878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96c4100250 0x7f96c4100660 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f96b4009910 tx=0x7f96b4009c20 comp rx=0 tx=0).stop 2024-04-27T02:04:24.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.551+0000 7f96ca2d4700 1 -- 172.21.15.29:0/670488878 shutdown_connections 2024-04-27T02:04:24.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.551+0000 7f96ca2d4700 1 --2- 172.21.15.29:0/670488878 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96c4101450 0x7f96c41018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:24.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.551+0000 7f96ca2d4700 1 --2- 172.21.15.29:0/670488878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96c4100250 0x7f96c4100660 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:24.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.551+0000 7f96ca2d4700 1 -- 172.21.15.29:0/670488878 >> 172.21.15.29:0/670488878 conn(0x7f96c40fb7e0 msgr2=0x7f96c40fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:24.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.551+0000 7f96ca2d4700 1 -- 172.21.15.29:0/670488878 shutdown_connections 2024-04-27T02:04:24.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.551+0000 7f96ca2d4700 1 -- 172.21.15.29:0/670488878 wait complete. 2024-04-27T02:04:24.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.552+0000 7f96ca2d4700 1 Processor -- start 2024-04-27T02:04:24.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.552+0000 7f96ca2d4700 1 -- start start 2024-04-27T02:04:24.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.552+0000 7f96ca2d4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96c4100250 0x7f96c4197c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:24.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.552+0000 7f96ca2d4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96c4101450 0x7f96c41981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:24.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.552+0000 7f96ca2d4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c41987a0 con 0x7f96c4100250 2024-04-27T02:04:24.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.552+0000 7f96ca2d4700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c4198910 con 0x7f96c4101450 2024-04-27T02:04:24.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.552+0000 7f96c3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96c4100250 0x7f96c4197c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:24.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.552+0000 7f96c3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96c4100250 0x7f96c4197c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37982/0 (socket says 172.21.15.29:37982) 2024-04-27T02:04:24.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.552+0000 7f96c3fff700 1 -- 172.21.15.29:0/2958626409 learned_addr learned my addr 172.21.15.29:0/2958626409 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:04:24.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.552+0000 7f96c37fe700 1 --2- 172.21.15.29:0/2958626409 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96c4101450 0x7f96c41981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:24.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.552+0000 7f96c3fff700 1 -- 172.21.15.29:0/2958626409 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96c4101450 msgr2=0x7f96c41981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:24.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.552+0000 7f96c3fff700 1 --2- 172.21.15.29:0/2958626409 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96c4101450 0x7f96c41981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:24.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.552+0000 7f96c3fff700 1 -- 172.21.15.29:0/2958626409 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b40095f0 con 0x7f96c4100250 2024-04-27T02:04:24.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.553+0000 7f96c3fff700 1 --2- 172.21.15.29:0/2958626409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96c4100250 0x7f96c4197c60 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f96b400bda0 tx=0x7f96b400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:24.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.553+0000 7f96c17fa700 1 -- 172.21.15.29:0/2958626409 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96b401d070 con 0x7f96c4100250 2024-04-27T02:04:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.553+0000 7f96c17fa700 1 -- 172.21.15.29:0/2958626409 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f96b400f460 con 0x7f96c4100250 2024-04-27T02:04:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.553+0000 7f96ca2d4700 1 -- 172.21.15.29:0/2958626409 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c419d340 con 0x7f96c4100250 2024-04-27T02:04:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.553+0000 7f96c17fa700 1 -- 172.21.15.29:0/2958626409 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96b4021620 con 0x7f96c4100250 2024-04-27T02:04:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.553+0000 7f96ca2d4700 1 -- 172.21.15.29:0/2958626409 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c419d830 con 0x7f96c4100250 2024-04-27T02:04:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.554+0000 7f96ca2d4700 1 -- 172.21.15.29:0/2958626409 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c4105530 con 0x7f96c4100250 2024-04-27T02:04:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.554+0000 7f96c17fa700 1 -- 172.21.15.29:0/2958626409 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f96b400f5d0 con 0x7f96c4100250 2024-04-27T02:04:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.555+0000 7f96c17fa700 1 --2- 172.21.15.29:0/2958626409 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f96ac06c680 0x7f96ac06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.555+0000 7f96c17fa700 1 -- 172.21.15.29:0/2958626409 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f96b408cb80 con 0x7f96c4100250 2024-04-27T02:04:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.555+0000 7f96c37fe700 1 --2- 172.21.15.29:0/2958626409 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f96ac06c680 0x7f96ac06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.555+0000 7f96c37fe700 1 --2- 172.21.15.29:0/2958626409 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f96ac06c680 0x7f96ac06eb30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f96c4199170 tx=0x7f96b800b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:24.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.559+0000 7f96c17fa700 1 -- 172.21.15.29:0/2958626409 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f96b4057200 con 0x7f96c4100250 2024-04-27T02:04:24.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.813+0000 7f96ca2d4700 1 -- 172.21.15.29:0/2958626409 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f96c402cc30 con 0x7f96c4100250 2024-04-27T02:04:24.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.886+0000 7f96c17fa700 1 -- 172.21.15.29:0/2958626409 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v16) v1 ==== 168+0+0 (secure 0 0 0) 0x7f96b405a820 con 0x7f96c4100250 2024-04-27T02:04:24.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.888+0000 7f96ca2d4700 1 -- 172.21.15.29:0/2958626409 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f96ac06c680 msgr2=0x7f96ac06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:24.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.888+0000 7f96ca2d4700 1 --2- 172.21.15.29:0/2958626409 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f96ac06c680 0x7f96ac06eb30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f96c4199170 tx=0x7f96b800b410 comp rx=0 tx=0).stop 2024-04-27T02:04:24.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.888+0000 7f96ca2d4700 1 -- 172.21.15.29:0/2958626409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96c4100250 msgr2=0x7f96c4197c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:24.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.888+0000 7f96ca2d4700 1 --2- 172.21.15.29:0/2958626409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96c4100250 0x7f96c4197c60 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f96b400bda0 tx=0x7f96b400bdd0 comp rx=0 tx=0).stop 2024-04-27T02:04:24.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.889+0000 7f96ca2d4700 1 -- 172.21.15.29:0/2958626409 shutdown_connections 2024-04-27T02:04:24.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.889+0000 7f96ca2d4700 1 --2- 172.21.15.29:0/2958626409 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f96ac06c680 0x7f96ac06eb30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:24.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.889+0000 7f96ca2d4700 1 --2- 172.21.15.29:0/2958626409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96c4100250 0x7f96c4197c60 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:24.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.889+0000 7f96ca2d4700 1 --2- 172.21.15.29:0/2958626409 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96c4101450 0x7f96c41981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:24.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.889+0000 7f96ca2d4700 1 -- 172.21.15.29:0/2958626409 >> 172.21.15.29:0/2958626409 conn(0x7f96c40fb7e0 msgr2=0x7f96c4102670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:24.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.889+0000 7f96ca2d4700 1 -- 172.21.15.29:0/2958626409 shutdown_connections 2024-04-27T02:04:24.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:24.889+0000 7f96ca2d4700 1 -- 172.21.15.29:0/2958626409 wait complete. 2024-04-27T02:04:24.870 INFO:teuthology.orchestra.run.smithi029.stderr:inline data enabled 2024-04-27T02:04:25.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:24 smithi029 ceph-mon[26769]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 347 B/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-27T02:04:25.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:24 smithi029 ceph-mon[26769]: mds.? [v2:172.21.15.29:6828/1353923457,v1:172.21.15.29:6829/1353923457] up:standby 2024-04-27T02:04:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:24 smithi029 ceph-mon[26769]: Dropping low affinity standby-replay daemon mds.cephfs.smithi052.yylmkb in favor of higher affinity standby. 2024-04-27T02:04:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:24 smithi029 ceph-mon[26769]: fsmap cephfs:1 {0=cephfs.smithi029.dyrjrd=up:active} 1 up:standby-replay 2 up:standby 2024-04-27T02:04:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:24 smithi029 ceph-mon[26769]: Dropping low affinity active daemon mds.cephfs.smithi029.dyrjrd in favor of higher affinity standby. 2024-04-27T02:04:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:24 smithi029 ceph-mon[26769]: Replacing daemon mds.cephfs.smithi029.dyrjrd as rank 0 with standby daemon mds.cephfs.smithi029.njjezc 2024-04-27T02:04:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:24 smithi029 ceph-mon[26769]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-27T02:04:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:24 smithi029 ceph-mon[26769]: fsmap cephfs:1 {0=cephfs.smithi029.dyrjrd=up:active} 1 up:standby-replay 1 up:standby 2024-04-27T02:04:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:24 smithi029 ceph-mon[26769]: osdmap e46: 6 total, 6 up, 6 in 2024-04-27T02:04:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:24 smithi029 ceph-mon[26769]: fsmap cephfs:1/1 {0=cephfs.smithi029.njjezc=up:replay} 1 up:standby 2024-04-27T02:04:25.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:24 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/2958626409' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T02:04:25.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:24 smithi052 ceph-mon[32586]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 347 B/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-27T02:04:25.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:24 smithi052 ceph-mon[32586]: mds.? [v2:172.21.15.29:6828/1353923457,v1:172.21.15.29:6829/1353923457] up:standby 2024-04-27T02:04:25.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:24 smithi052 ceph-mon[32586]: Dropping low affinity standby-replay daemon mds.cephfs.smithi052.yylmkb in favor of higher affinity standby. 2024-04-27T02:04:25.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:24 smithi052 ceph-mon[32586]: fsmap cephfs:1 {0=cephfs.smithi029.dyrjrd=up:active} 1 up:standby-replay 2 up:standby 2024-04-27T02:04:25.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:24 smithi052 ceph-mon[32586]: Dropping low affinity active daemon mds.cephfs.smithi029.dyrjrd in favor of higher affinity standby. 2024-04-27T02:04:25.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:24 smithi052 ceph-mon[32586]: Replacing daemon mds.cephfs.smithi029.dyrjrd as rank 0 with standby daemon mds.cephfs.smithi029.njjezc 2024-04-27T02:04:25.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:24 smithi052 ceph-mon[32586]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-27T02:04:25.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:24 smithi052 ceph-mon[32586]: fsmap cephfs:1 {0=cephfs.smithi029.dyrjrd=up:active} 1 up:standby-replay 1 up:standby 2024-04-27T02:04:25.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:24 smithi052 ceph-mon[32586]: osdmap e46: 6 total, 6 up, 6 in 2024-04-27T02:04:25.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:24 smithi052 ceph-mon[32586]: fsmap cephfs:1/1 {0=cephfs.smithi029.njjezc=up:replay} 1 up:standby 2024-04-27T02:04:25.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:24 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/2958626409' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T02:04:26.124 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:25 smithi029 ceph-mon[26769]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-04-27T02:04:26.125 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:25 smithi029 ceph-mon[26769]: mds.? [v2:172.21.15.29:6828/1353923457,v1:172.21.15.29:6829/1353923457] up:reconnect 2024-04-27T02:04:26.125 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:25 smithi029 ceph-mon[26769]: mds.? [v2:172.21.15.52:6824/2352841895,v1:172.21.15.52:6825/2352841895] up:boot 2024-04-27T02:04:26.125 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:25 smithi029 ceph-mon[26769]: mds.? [v2:172.21.15.29:6826/3428808494,v1:172.21.15.29:6827/3428808494] up:boot 2024-04-27T02:04:26.125 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:25 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/2958626409' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T02:04:26.125 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:25 smithi029 ceph-mon[26769]: fsmap cephfs:1/1 {0=cephfs.smithi029.njjezc=up:reconnect} 3 up:standby 2024-04-27T02:04:26.125 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:25 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi052.yylmkb"}]: dispatch 2024-04-27T02:04:26.125 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:25 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi029.dyrjrd"}]: dispatch 2024-04-27T02:04:26.125 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:25 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:26.125 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:25 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:26.150 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T02:04:26.160 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi029.front.sepia.ceph.com 2024-04-27T02:04:26.160 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-27T02:04:26.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:25 smithi052 ceph-mon[32586]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-04-27T02:04:26.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:25 smithi052 ceph-mon[32586]: mds.? [v2:172.21.15.29:6828/1353923457,v1:172.21.15.29:6829/1353923457] up:reconnect 2024-04-27T02:04:26.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:25 smithi052 ceph-mon[32586]: mds.? [v2:172.21.15.52:6824/2352841895,v1:172.21.15.52:6825/2352841895] up:boot 2024-04-27T02:04:26.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:25 smithi052 ceph-mon[32586]: mds.? [v2:172.21.15.29:6826/3428808494,v1:172.21.15.29:6827/3428808494] up:boot 2024-04-27T02:04:26.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:25 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/2958626409' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T02:04:26.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:25 smithi052 ceph-mon[32586]: fsmap cephfs:1/1 {0=cephfs.smithi029.njjezc=up:reconnect} 3 up:standby 2024-04-27T02:04:26.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:25 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi052.yylmkb"}]: dispatch 2024-04-27T02:04:26.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:25 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi029.dyrjrd"}]: dispatch 2024-04-27T02:04:26.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:25 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:26.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:25 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:27.054 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:04:27.185 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:26 smithi029 ceph-mon[26769]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-27T02:04:27.185 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:26 smithi029 ceph-mon[26769]: mds.? [v2:172.21.15.29:6828/1353923457,v1:172.21.15.29:6829/1353923457] up:rejoin 2024-04-27T02:04:27.185 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:26 smithi029 ceph-mon[26769]: fsmap cephfs:1/1 {0=cephfs.smithi029.njjezc=up:rejoin} 3 up:standby 2024-04-27T02:04:27.185 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:26 smithi029 ceph-mon[26769]: daemon mds.cephfs.smithi029.njjezc is now active in filesystem cephfs as rank 0 2024-04-27T02:04:27.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:26 smithi052 ceph-mon[32586]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-27T02:04:27.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:26 smithi052 ceph-mon[32586]: mds.? [v2:172.21.15.29:6828/1353923457,v1:172.21.15.29:6829/1353923457] up:rejoin 2024-04-27T02:04:27.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:26 smithi052 ceph-mon[32586]: fsmap cephfs:1/1 {0=cephfs.smithi029.njjezc=up:rejoin} 3 up:standby 2024-04-27T02:04:27.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:26 smithi052 ceph-mon[32586]: daemon mds.cephfs.smithi029.njjezc is now active in filesystem cephfs as rank 0 2024-04-27T02:04:28.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:27 smithi029 ceph-mon[26769]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-27T02:04:28.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:27 smithi029 ceph-mon[26769]: mds.? [v2:172.21.15.29:6828/1353923457,v1:172.21.15.29:6829/1353923457] up:active 2024-04-27T02:04:28.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:27 smithi029 ceph-mon[26769]: mds.? [v2:172.21.15.52:6826/3513632158,v1:172.21.15.52:6827/3513632158] up:standby 2024-04-27T02:04:28.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:27 smithi029 ceph-mon[26769]: fsmap cephfs:1 {0=cephfs.smithi029.njjezc=up:active} 3 up:standby 2024-04-27T02:04:28.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:27 smithi029 ceph-mon[26769]: fsmap cephfs:1 {0=cephfs.smithi029.njjezc=up:active} 1 up:standby-replay 2 up:standby 2024-04-27T02:04:28.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:27 smithi052 ceph-mon[32586]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-27T02:04:28.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:27 smithi052 ceph-mon[32586]: mds.? [v2:172.21.15.29:6828/1353923457,v1:172.21.15.29:6829/1353923457] up:active 2024-04-27T02:04:28.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:27 smithi052 ceph-mon[32586]: mds.? [v2:172.21.15.52:6826/3513632158,v1:172.21.15.52:6827/3513632158] up:standby 2024-04-27T02:04:28.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:27 smithi052 ceph-mon[32586]: fsmap cephfs:1 {0=cephfs.smithi029.njjezc=up:active} 3 up:standby 2024-04-27T02:04:28.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:27 smithi052 ceph-mon[32586]: fsmap cephfs:1 {0=cephfs.smithi029.njjezc=up:active} 1 up:standby-replay 2 up:standby 2024-04-27T02:04:29.134 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:28 smithi029 ceph-mon[26769]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.9 KiB/s rd, 818 B/s wr, 4 op/s 2024-04-27T02:04:29.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:28 smithi052 ceph-mon[32586]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.9 KiB/s rd, 818 B/s wr, 4 op/s 2024-04-27T02:04:29.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.700+0000 7f0f1f890700 1 -- 172.21.15.29:0/1323775183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f18100240 msgr2=0x7f0f18100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:29.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.700+0000 7f0f1f890700 1 --2- 172.21.15.29:0/1323775183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f18100240 0x7f0f18100650 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0f14009910 tx=0x7f0f14009c20 comp rx=0 tx=0).stop 2024-04-27T02:04:29.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.700+0000 7f0f1f890700 1 -- 172.21.15.29:0/1323775183 shutdown_connections 2024-04-27T02:04:29.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.700+0000 7f0f1f890700 1 --2- 172.21.15.29:0/1323775183 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f18101440 0x7f0f18101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:29.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.700+0000 7f0f1f890700 1 --2- 172.21.15.29:0/1323775183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f18100240 0x7f0f18100650 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:29.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.700+0000 7f0f1f890700 1 -- 172.21.15.29:0/1323775183 >> 172.21.15.29:0/1323775183 conn(0x7f0f180fb7f0 msgr2=0x7f0f180fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:29.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.700+0000 7f0f1f890700 1 -- 172.21.15.29:0/1323775183 shutdown_connections 2024-04-27T02:04:29.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.700+0000 7f0f1f890700 1 -- 172.21.15.29:0/1323775183 wait complete. 2024-04-27T02:04:29.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.700+0000 7f0f1f890700 1 Processor -- start 2024-04-27T02:04:29.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.700+0000 7f0f1f890700 1 -- start start 2024-04-27T02:04:29.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.701+0000 7f0f1f890700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f18100240 0x7f0f18074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:29.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.701+0000 7f0f1f890700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f18101440 0x7f0f18073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:29.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.701+0000 7f0f1f890700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f18073680 con 0x7f0f18100240 2024-04-27T02:04:29.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.701+0000 7f0f1f890700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f180737c0 con 0x7f0f18101440 2024-04-27T02:04:29.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.701+0000 7f0f1d62c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f18100240 0x7f0f18074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:29.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.701+0000 7f0f1d62c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f18100240 0x7f0f18074af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40844/0 (socket says 172.21.15.29:40844) 2024-04-27T02:04:29.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.701+0000 7f0f1d62c700 1 -- 172.21.15.29:0/1595488939 learned_addr learned my addr 172.21.15.29:0/1595488939 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:04:29.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.701+0000 7f0f1ce2b700 1 --2- 172.21.15.29:0/1595488939 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f18101440 0x7f0f18073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:29.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.701+0000 7f0f1d62c700 1 -- 172.21.15.29:0/1595488939 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f18101440 msgr2=0x7f0f18073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:29.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.701+0000 7f0f1d62c700 1 --2- 172.21.15.29:0/1595488939 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f18101440 0x7f0f18073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:29.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.701+0000 7f0f1d62c700 1 -- 172.21.15.29:0/1595488939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f140095f0 con 0x7f0f18100240 2024-04-27T02:04:29.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.701+0000 7f0f1d62c700 1 --2- 172.21.15.29:0/1595488939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f18100240 0x7f0f18074af0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0f1400bda0 tx=0x7f0f1400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:29.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.701+0000 7f0f0e7fc700 1 -- 172.21.15.29:0/1595488939 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f1401d070 con 0x7f0f18100240 2024-04-27T02:04:29.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.701+0000 7f0f1f890700 1 -- 172.21.15.29:0/1595488939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f18073a40 con 0x7f0f18100240 2024-04-27T02:04:29.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.701+0000 7f0f0e7fc700 1 -- 172.21.15.29:0/1595488939 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0f14022470 con 0x7f0f18100240 2024-04-27T02:04:29.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.701+0000 7f0f0e7fc700 1 -- 172.21.15.29:0/1595488939 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f1400f650 con 0x7f0f18100240 2024-04-27T02:04:29.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.701+0000 7f0f1f890700 1 -- 172.21.15.29:0/1595488939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f18073f30 con 0x7f0f18100240 2024-04-27T02:04:29.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.702+0000 7f0f1f890700 1 -- 172.21.15.29:0/1595488939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f18108e10 con 0x7f0f18100240 2024-04-27T02:04:29.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.703+0000 7f0f0e7fc700 1 -- 172.21.15.29:0/1595488939 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f0f140225e0 con 0x7f0f18100240 2024-04-27T02:04:29.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.703+0000 7f0f0e7fc700 1 --2- 172.21.15.29:0/1595488939 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f0f0406c680 0x7f0f0406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:29.992 INFO:teuthology.orchestra.run.smithi029.stdout:e19 2024-04-27T02:04:29.992 INFO:teuthology.orchestra.run.smithi029.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T02:04:29.992 INFO:teuthology.orchestra.run.smithi029.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T02:04:29.992 INFO:teuthology.orchestra.run.smithi029.stdout:legacy client fscid: 1 2024-04-27T02:04:29.992 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:04:29.992 INFO:teuthology.orchestra.run.smithi029.stdout:Filesystem 'cephfs' (1) 2024-04-27T02:04:29.992 INFO:teuthology.orchestra.run.smithi029.stdout:fs_name cephfs 2024-04-27T02:04:29.992 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 19 2024-04-27T02:04:29.992 INFO:teuthology.orchestra.run.smithi029.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-27T02:04:29.992 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-04-27T02:04:10.035536+0000 2024-04-27T02:04:29.992 INFO:teuthology.orchestra.run.smithi029.stdout:modified 2024-04-27T02:04:26.892992+0000 2024-04-27T02:04:29.992 INFO:teuthology.orchestra.run.smithi029.stdout:tableserver 0 2024-04-27T02:04:29.992 INFO:teuthology.orchestra.run.smithi029.stdout:root 0 2024-04-27T02:04:29.992 INFO:teuthology.orchestra.run.smithi029.stdout:session_timeout 60 2024-04-27T02:04:29.993 INFO:teuthology.orchestra.run.smithi029.stdout:session_autoclose 300 2024-04-27T02:04:29.993 INFO:teuthology.orchestra.run.smithi029.stdout:max_file_size 1099511627776 2024-04-27T02:04:29.993 INFO:teuthology.orchestra.run.smithi029.stdout:required_client_features {} 2024-04-27T02:04:29.993 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure 0 2024-04-27T02:04:29.993 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure_osd_epoch 46 2024-04-27T02:04:29.993 INFO:teuthology.orchestra.run.smithi029.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T02:04:29.993 INFO:teuthology.orchestra.run.smithi029.stdout:max_mds 1 2024-04-27T02:04:29.993 INFO:teuthology.orchestra.run.smithi029.stdout:in 0 2024-04-27T02:04:29.993 INFO:teuthology.orchestra.run.smithi029.stdout:up {0=14484} 2024-04-27T02:04:29.993 INFO:teuthology.orchestra.run.smithi029.stdout:failed 2024-04-27T02:04:29.993 INFO:teuthology.orchestra.run.smithi029.stdout:damaged 2024-04-27T02:04:29.994 INFO:teuthology.orchestra.run.smithi029.stdout:stopped 2024-04-27T02:04:29.994 INFO:teuthology.orchestra.run.smithi029.stdout:data_pools [3] 2024-04-27T02:04:29.994 INFO:teuthology.orchestra.run.smithi029.stdout:metadata_pool 2 2024-04-27T02:04:29.994 INFO:teuthology.orchestra.run.smithi029.stdout:inline_data enabled 2024-04-27T02:04:29.994 INFO:teuthology.orchestra.run.smithi029.stdout:balancer 2024-04-27T02:04:29.994 INFO:teuthology.orchestra.run.smithi029.stdout:bal_rank_mask -1 2024-04-27T02:04:29.994 INFO:teuthology.orchestra.run.smithi029.stdout:standby_count_wanted 1 2024-04-27T02:04:29.994 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi029.njjezc{0:14484} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.29:6828/1353923457,v1:172.21.15.29:6829/1353923457] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T02:04:29.994 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi052.yylmkb{0:14494} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.52:6824/2352841895,v1:172.21.15.52:6825/2352841895] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T02:04:29.994 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:04:29.994 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:04:29.994 INFO:teuthology.orchestra.run.smithi029.stdout:Standby daemons: 2024-04-27T02:04:29.995 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:04:29.995 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi029.dyrjrd{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.29:6826/3428808494,v1:172.21.15.29:6827/3428808494] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T02:04:29.995 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi052.nbjpjd{-1:24325} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.52:6826/3513632158,v1:172.21.15.52:6827/3513632158] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T02:04:29.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.704+0000 7f0f1ce2b700 1 --2- 172.21.15.29:0/1595488939 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f0f0406c680 0x7f0f0406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:29.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.704+0000 7f0f0e7fc700 1 -- 172.21.15.29:0/1595488939 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f0f1408cf10 con 0x7f0f18100240 2024-04-27T02:04:29.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.704+0000 7f0f1ce2b700 1 --2- 172.21.15.29:0/1595488939 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f0f0406c680 0x7f0f0406eb30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0f08009a10 tx=0x7f0f08009190 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:29.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.708+0000 7f0f0e7fc700 1 -- 172.21.15.29:0/1595488939 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0f1405b0d0 con 0x7f0f18100240 2024-04-27T02:04:29.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.954+0000 7f0f1f890700 1 -- 172.21.15.29:0/1595488939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0f1819dfe0 con 0x7f0f18100240 2024-04-27T02:04:29.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.955+0000 7f0f0e7fc700 1 -- 172.21.15.29:0/1595488939 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 19 v19) v1 ==== 76+0+1851 (secure 0 0 0) 0x7f0f14027750 con 0x7f0f18100240 2024-04-27T02:04:29.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.957+0000 7f0f1f890700 1 -- 172.21.15.29:0/1595488939 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f0f0406c680 msgr2=0x7f0f0406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:29.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.957+0000 7f0f1f890700 1 --2- 172.21.15.29:0/1595488939 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f0f0406c680 0x7f0f0406eb30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0f08009a10 tx=0x7f0f08009190 comp rx=0 tx=0).stop 2024-04-27T02:04:29.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.957+0000 7f0f1f890700 1 -- 172.21.15.29:0/1595488939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f18100240 msgr2=0x7f0f18074af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:29.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.957+0000 7f0f1f890700 1 --2- 172.21.15.29:0/1595488939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f18100240 0x7f0f18074af0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0f1400bda0 tx=0x7f0f1400bdd0 comp rx=0 tx=0).stop 2024-04-27T02:04:29.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.957+0000 7f0f1f890700 1 -- 172.21.15.29:0/1595488939 shutdown_connections 2024-04-27T02:04:29.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.957+0000 7f0f1f890700 1 --2- 172.21.15.29:0/1595488939 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f0f0406c680 0x7f0f0406eb30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:29.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.957+0000 7f0f1f890700 1 --2- 172.21.15.29:0/1595488939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f18100240 0x7f0f18074af0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:29.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.957+0000 7f0f1f890700 1 --2- 172.21.15.29:0/1595488939 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f18101440 0x7f0f18073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:29.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.957+0000 7f0f1f890700 1 -- 172.21.15.29:0/1595488939 >> 172.21.15.29:0/1595488939 conn(0x7f0f180fb7f0 msgr2=0x7f0f18104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:29.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.957+0000 7f0f1f890700 1 -- 172.21.15.29:0/1595488939 shutdown_connections 2024-04-27T02:04:29.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:29.957+0000 7f0f1f890700 1 -- 172.21.15.29:0/1595488939 wait complete. 2024-04-27T02:04:29.996 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 19 2024-04-27T02:04:30.617 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-04-27T02:04:31.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:30 smithi029 ceph-mon[26769]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 4.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-27T02:04:31.170 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:30 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/1595488939' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T02:04:31.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:30 smithi052 ceph-mon[32586]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 4.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-27T02:04:31.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:30 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/1595488939' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T02:04:31.439 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:04:33.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.073+0000 7f518bf24700 1 -- 172.21.15.29:0/372471871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51841036a0 msgr2=0x7f5184105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:33.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.073+0000 7f518bf24700 1 --2- 172.21.15.29:0/372471871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51841036a0 0x7f5184105a80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5174009910 tx=0x7f5174009c20 comp rx=0 tx=0).stop 2024-04-27T02:04:33.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.073+0000 7f518bf24700 1 -- 172.21.15.29:0/372471871 shutdown_connections 2024-04-27T02:04:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.073+0000 7f518bf24700 1 --2- 172.21.15.29:0/372471871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51841036a0 0x7f5184105a80 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.073+0000 7f518bf24700 1 --2- 172.21.15.29:0/372471871 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5184100d80 0x7f5184103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.073+0000 7f518bf24700 1 -- 172.21.15.29:0/372471871 >> 172.21.15.29:0/372471871 conn(0x7f51840fa7a0 msgr2=0x7f51840fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.073+0000 7f518bf24700 1 -- 172.21.15.29:0/372471871 shutdown_connections 2024-04-27T02:04:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.073+0000 7f518bf24700 1 -- 172.21.15.29:0/372471871 wait complete. 2024-04-27T02:04:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.074+0000 7f518bf24700 1 Processor -- start 2024-04-27T02:04:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.074+0000 7f518bf24700 1 -- start start 2024-04-27T02:04:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.074+0000 7f518bf24700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5184100d80 0x7f51841006e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.074+0000 7f518bf24700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51841036a0 0x7f51840fed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.074+0000 7f518bf24700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51840ff270 con 0x7f51841036a0 2024-04-27T02:04:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.074+0000 7f518bf24700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51840ff3b0 con 0x7f5184100d80 2024-04-27T02:04:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.074+0000 7f51894bf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51841036a0 0x7f51840fed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.074+0000 7f51894bf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51841036a0 0x7f51840fed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40856/0 (socket says 172.21.15.29:40856) 2024-04-27T02:04:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.074+0000 7f51894bf700 1 -- 172.21.15.29:0/1206692379 learned_addr learned my addr 172.21.15.29:0/1206692379 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:04:33.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.074+0000 7f5189cc0700 1 --2- 172.21.15.29:0/1206692379 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5184100d80 0x7f51841006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:33.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.075+0000 7f51894bf700 1 -- 172.21.15.29:0/1206692379 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5184100d80 msgr2=0x7f51841006e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:33.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.075+0000 7f51894bf700 1 --2- 172.21.15.29:0/1206692379 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5184100d80 0x7f51841006e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:33.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.075+0000 7f51894bf700 1 -- 172.21.15.29:0/1206692379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51740095f0 con 0x7f51841036a0 2024-04-27T02:04:33.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.075+0000 7f51894bf700 1 --2- 172.21.15.29:0/1206692379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51841036a0 0x7f51840fed30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5174006010 tx=0x7f517400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:33.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.075+0000 7f517affd700 1 -- 172.21.15.29:0/1206692379 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f517401d070 con 0x7f51841036a0 2024-04-27T02:04:33.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.075+0000 7f518bf24700 1 -- 172.21.15.29:0/1206692379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51840ff630 con 0x7f51841036a0 2024-04-27T02:04:33.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.075+0000 7f517affd700 1 -- 172.21.15.29:0/1206692379 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f517400f460 con 0x7f51841036a0 2024-04-27T02:04:33.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.075+0000 7f517affd700 1 -- 172.21.15.29:0/1206692379 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5174021640 con 0x7f51841036a0 2024-04-27T02:04:33.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.075+0000 7f518bf24700 1 -- 172.21.15.29:0/1206692379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51840ffb20 con 0x7f51841036a0 2024-04-27T02:04:33.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.076+0000 7f518bf24700 1 -- 172.21.15.29:0/1206692379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f518418fc00 con 0x7f51841036a0 2024-04-27T02:04:33.059 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:32 smithi029 ceph-mon[26769]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 9.9 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-27T02:04:33.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.077+0000 7f517affd700 1 -- 172.21.15.29:0/1206692379 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f517400fab0 con 0x7f51841036a0 2024-04-27T02:04:33.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.077+0000 7f517affd700 1 --2- 172.21.15.29:0/1206692379 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f517006c680 0x7f517006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:33.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.077+0000 7f517affd700 1 -- 172.21.15.29:0/1206692379 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f517408cb30 con 0x7f51841036a0 2024-04-27T02:04:33.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.077+0000 7f5189cc0700 1 --2- 172.21.15.29:0/1206692379 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f517006c680 0x7f517006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:33.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.078+0000 7f5189cc0700 1 --2- 172.21.15.29:0/1206692379 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f517006c680 0x7f517006eb30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5180005950 tx=0x7f518000b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:33.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.082+0000 7f517affd700 1 -- 172.21.15.29:0/1206692379 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f517405ac40 con 0x7f51841036a0 2024-04-27T02:04:33.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:32 smithi052 ceph-mon[32586]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 9.9 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-27T02:04:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.335+0000 7f518bf24700 1 -- 172.21.15.29:0/1206692379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f518402cc30 con 0x7f51841036a0 2024-04-27T02:04:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.336+0000 7f517affd700 1 -- 172.21.15.29:0/1206692379 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4747 (secure 0 0 0) 0x7f517405a7d0 con 0x7f51841036a0 2024-04-27T02:04:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.338+0000 7f518bf24700 1 -- 172.21.15.29:0/1206692379 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f517006c680 msgr2=0x7f517006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.338+0000 7f518bf24700 1 --2- 172.21.15.29:0/1206692379 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f517006c680 0x7f517006eb30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5180005950 tx=0x7f518000b410 comp rx=0 tx=0).stop 2024-04-27T02:04:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.338+0000 7f518bf24700 1 -- 172.21.15.29:0/1206692379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51841036a0 msgr2=0x7f51840fed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.338+0000 7f518bf24700 1 --2- 172.21.15.29:0/1206692379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51841036a0 0x7f51840fed30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5174006010 tx=0x7f517400bd70 comp rx=0 tx=0).stop 2024-04-27T02:04:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.338+0000 7f518bf24700 1 -- 172.21.15.29:0/1206692379 shutdown_connections 2024-04-27T02:04:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.338+0000 7f518bf24700 1 --2- 172.21.15.29:0/1206692379 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f517006c680 0x7f517006eb30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.338+0000 7f518bf24700 1 --2- 172.21.15.29:0/1206692379 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5184100d80 0x7f51841006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.338+0000 7f518bf24700 1 --2- 172.21.15.29:0/1206692379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51841036a0 0x7f51840fed30 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.338+0000 7f518bf24700 1 -- 172.21.15.29:0/1206692379 >> 172.21.15.29:0/1206692379 conn(0x7f51840fa7a0 msgr2=0x7f51840fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.338+0000 7f518bf24700 1 -- 172.21.15.29:0/1206692379 shutdown_connections 2024-04-27T02:04:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:33.338+0000 7f518bf24700 1 -- 172.21.15.29:0/1206692379 wait complete. 2024-04-27T02:04:33.321 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 19 2024-04-27T02:04:33.330 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-04-27T02:04:34.036 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-04-27T02:04:34.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:34 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:34.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:34 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:34.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:34 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:04:34.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:34 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:04:34.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:34 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:34.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:34 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:04:34.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:34 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/1206692379' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T02:04:34.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:34 smithi052 ceph-mon[32586]: pgmap v197: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 10 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-27T02:04:34.407 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:34 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:34.408 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:34 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:34.408 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:34 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:04:34.408 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:34 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:04:34.408 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:34 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:34.408 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:34 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:04:34.408 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:34 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/1206692379' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T02:04:34.408 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:34 smithi029 ceph-mon[26769]: pgmap v197: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 10 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-27T02:04:34.843 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:04:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.446+0000 7f2644c29700 1 -- 172.21.15.29:0/3501879365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2640101440 msgr2=0x7f2640101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.446+0000 7f2644c29700 1 --2- 172.21.15.29:0/3501879365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2640101440 0x7f2640101890 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2628009910 tx=0x7f2628009c20 comp rx=0 tx=0).stop 2024-04-27T02:04:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.446+0000 7f2644c29700 1 -- 172.21.15.29:0/3501879365 shutdown_connections 2024-04-27T02:04:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.446+0000 7f2644c29700 1 --2- 172.21.15.29:0/3501879365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2640101440 0x7f2640101890 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.446+0000 7f2644c29700 1 --2- 172.21.15.29:0/3501879365 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2640100240 0x7f2640100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.446+0000 7f2644c29700 1 -- 172.21.15.29:0/3501879365 >> 172.21.15.29:0/3501879365 conn(0x7f26400fb7f0 msgr2=0x7f26400fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.446+0000 7f2644c29700 1 -- 172.21.15.29:0/3501879365 shutdown_connections 2024-04-27T02:04:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.446+0000 7f2644c29700 1 -- 172.21.15.29:0/3501879365 wait complete. 2024-04-27T02:04:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.447+0000 7f2644c29700 1 Processor -- start 2024-04-27T02:04:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.447+0000 7f2644c29700 1 -- start start 2024-04-27T02:04:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.447+0000 7f2644c29700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2640100240 0x7f2640195b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.447+0000 7f2644c29700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2640101440 0x7f2640196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.447+0000 7f2644c29700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2640196690 con 0x7f2640100240 2024-04-27T02:04:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.447+0000 7f2644c29700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26401967d0 con 0x7f2640101440 2024-04-27T02:04:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.447+0000 7f263e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2640100240 0x7f2640195b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.447+0000 7f263e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2640100240 0x7f2640195b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40874/0 (socket says 172.21.15.29:40874) 2024-04-27T02:04:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.447+0000 7f263e59c700 1 -- 172.21.15.29:0/3842445696 learned_addr learned my addr 172.21.15.29:0/3842445696 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:04:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.447+0000 7f263dd9b700 1 --2- 172.21.15.29:0/3842445696 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2640101440 0x7f2640196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.447+0000 7f263e59c700 1 -- 172.21.15.29:0/3842445696 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2640101440 msgr2=0x7f2640196070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.448+0000 7f263e59c700 1 --2- 172.21.15.29:0/3842445696 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2640101440 0x7f2640196070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.448+0000 7f263e59c700 1 -- 172.21.15.29:0/3842445696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26280095f0 con 0x7f2640100240 2024-04-27T02:04:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.448+0000 7f263e59c700 1 --2- 172.21.15.29:0/3842445696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2640100240 0x7f2640195b30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f263800b9c0 tx=0x7f263800bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.448+0000 7f26377fe700 1 -- 172.21.15.29:0/3842445696 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f263800c980 con 0x7f2640100240 2024-04-27T02:04:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.448+0000 7f26377fe700 1 -- 172.21.15.29:0/3842445696 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2638014440 con 0x7f2640100240 2024-04-27T02:04:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.448+0000 7f2644c29700 1 -- 172.21.15.29:0/3842445696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f264019b280 con 0x7f2640100240 2024-04-27T02:04:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.448+0000 7f26377fe700 1 -- 172.21.15.29:0/3842445696 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2638012550 con 0x7f2640100240 2024-04-27T02:04:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.448+0000 7f2644c29700 1 -- 172.21.15.29:0/3842445696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f264019b7d0 con 0x7f2640100240 2024-04-27T02:04:36.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.449+0000 7f2644c29700 1 -- 172.21.15.29:0/3842445696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2640108e30 con 0x7f2640100240 2024-04-27T02:04:36.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.449+0000 7f26377fe700 1 -- 172.21.15.29:0/3842445696 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f26380145b0 con 0x7f2640100240 2024-04-27T02:04:36.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.450+0000 7f26377fe700 1 --2- 172.21.15.29:0/3842445696 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f262c06c680 0x7f262c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:36.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.450+0000 7f26377fe700 1 -- 172.21.15.29:0/3842445696 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f2638006e40 con 0x7f2640100240 2024-04-27T02:04:36.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.450+0000 7f263dd9b700 1 --2- 172.21.15.29:0/3842445696 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f262c06c680 0x7f262c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:36.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.450+0000 7f263dd9b700 1 --2- 172.21.15.29:0/3842445696 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f262c06c680 0x7f262c06eb30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2628006010 tx=0x7f262800b580 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:36.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.454+0000 7f26377fe700 1 -- 172.21.15.29:0/3842445696 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f263804e720 con 0x7f2640100240 2024-04-27T02:04:36.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.764+0000 7f2644c29700 1 -- 172.21.15.29:0/3842445696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f2640108ff0 con 0x7f2640100240 2024-04-27T02:04:36.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.765+0000 7f26377fe700 1 -- 172.21.15.29:0/3842445696 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v19) v1 ==== 78+0+83 (secure 0 0 0) 0x7f2638019070 con 0x7f2640100240 2024-04-27T02:04:36.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.767+0000 7f2644c29700 1 -- 172.21.15.29:0/3842445696 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f262c06c680 msgr2=0x7f262c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:36.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.767+0000 7f2644c29700 1 --2- 172.21.15.29:0/3842445696 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f262c06c680 0x7f262c06eb30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2628006010 tx=0x7f262800b580 comp rx=0 tx=0).stop 2024-04-27T02:04:36.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.767+0000 7f2644c29700 1 -- 172.21.15.29:0/3842445696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2640100240 msgr2=0x7f2640195b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:36.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.767+0000 7f2644c29700 1 --2- 172.21.15.29:0/3842445696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2640100240 0x7f2640195b30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f263800b9c0 tx=0x7f263800bd80 comp rx=0 tx=0).stop 2024-04-27T02:04:36.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.767+0000 7f2644c29700 1 -- 172.21.15.29:0/3842445696 shutdown_connections 2024-04-27T02:04:36.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.767+0000 7f2644c29700 1 --2- 172.21.15.29:0/3842445696 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f262c06c680 0x7f262c06eb30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:36.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.767+0000 7f2644c29700 1 --2- 172.21.15.29:0/3842445696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2640100240 0x7f2640195b30 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:36.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.767+0000 7f2644c29700 1 --2- 172.21.15.29:0/3842445696 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2640101440 0x7f2640196070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:36.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.767+0000 7f2644c29700 1 -- 172.21.15.29:0/3842445696 >> 172.21.15.29:0/3842445696 conn(0x7f26400fb7f0 msgr2=0x7f2640104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:36.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.767+0000 7f2644c29700 1 -- 172.21.15.29:0/3842445696 shutdown_connections 2024-04-27T02:04:36.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:36.768+0000 7f2644c29700 1 -- 172.21.15.29:0/3842445696 wait complete. 2024-04-27T02:04:36.760 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-04-27T02:04:36.840 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:36 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:36.840 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:36 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:36.840 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:36 smithi029 ceph-mon[26769]: pgmap v198: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-27T02:04:37.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:36 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:37.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:36 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:37.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:36 smithi052 ceph-mon[32586]: pgmap v198: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-27T02:04:37.507 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-04-27T02:04:37.521 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T02:04:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.673+0000 7fe625300640 1 -- 172.21.15.29:0/3693736800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6200fe550 msgr2=0x7fe6200fe960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.673+0000 7fe625300640 1 --2- 172.21.15.29:0/3693736800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6200fe550 0x7fe6200fe960 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe610009880 tx=0x7fe61002bce0 comp rx=0 tx=0).stop 2024-04-27T02:04:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.673+0000 7fe625300640 1 -- 172.21.15.29:0/3693736800 shutdown_connections 2024-04-27T02:04:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.673+0000 7fe625300640 1 --2- 172.21.15.29:0/3693736800 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe6200ff740 0x7fe6200ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.673+0000 7fe625300640 1 --2- 172.21.15.29:0/3693736800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6200fe550 0x7fe6200fe960 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.673+0000 7fe625300640 1 -- 172.21.15.29:0/3693736800 >> 172.21.15.29:0/3693736800 conn(0x7fe6200f9ad0 msgr2=0x7fe6200fbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:37.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.673+0000 7fe625300640 1 -- 172.21.15.29:0/3693736800 shutdown_connections 2024-04-27T02:04:37.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.673+0000 7fe625300640 1 -- 172.21.15.29:0/3693736800 wait complete. 2024-04-27T02:04:37.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.673+0000 7fe625300640 1 Processor -- start 2024-04-27T02:04:37.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.673+0000 7fe625300640 1 -- start start 2024-04-27T02:04:37.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.673+0000 7fe625300640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe6200fe550 0x7fe6201024e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:37.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.673+0000 7fe625300640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6200ff740 0x7fe620100b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:37.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.673+0000 7fe625300640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe620101060 con 0x7fe6200ff740 2024-04-27T02:04:37.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.673+0000 7fe625300640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6201011d0 con 0x7fe6200fe550 2024-04-27T02:04:37.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.674+0000 7fe617fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6200ff740 0x7fe620100b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:37.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.674+0000 7fe617fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6200ff740 0x7fe620100b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35278/0 (socket says 172.21.15.29:35278) 2024-04-27T02:04:37.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.674+0000 7fe617fff640 1 -- 172.21.15.29:0/1443447582 learned_addr learned my addr 172.21.15.29:0/1443447582 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:04:37.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.674+0000 7fe61effd640 1 --2- 172.21.15.29:0/1443447582 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe6200fe550 0x7fe6201024e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:37.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.674+0000 7fe617fff640 1 -- 172.21.15.29:0/1443447582 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe6200fe550 msgr2=0x7fe6201024e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:37.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.674+0000 7fe617fff640 1 --2- 172.21.15.29:0/1443447582 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe6200fe550 0x7fe6201024e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:37.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.674+0000 7fe617fff640 1 -- 172.21.15.29:0/1443447582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe610009560 con 0x7fe6200ff740 2024-04-27T02:04:37.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.674+0000 7fe617fff640 1 --2- 172.21.15.29:0/1443447582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6200ff740 0x7fe620100b20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe60800d6f0 tx=0x7fe60800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:37.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.674+0000 7fe61cff9640 1 -- 172.21.15.29:0/1443447582 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe608009a30 con 0x7fe6200ff740 2024-04-27T02:04:37.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.674+0000 7fe61cff9640 1 -- 172.21.15.29:0/1443447582 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe6080054d0 con 0x7fe6200ff740 2024-04-27T02:04:37.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.674+0000 7fe625300640 1 -- 172.21.15.29:0/1443447582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6201014b0 con 0x7fe6200ff740 2024-04-27T02:04:37.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.674+0000 7fe61cff9640 1 -- 172.21.15.29:0/1443447582 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe608010460 con 0x7fe6200ff740 2024-04-27T02:04:37.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.674+0000 7fe625300640 1 -- 172.21.15.29:0/1443447582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe62019cfb0 con 0x7fe6200ff740 2024-04-27T02:04:37.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.675+0000 7fe625300640 1 -- 172.21.15.29:0/1443447582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe620107130 con 0x7fe6200ff740 2024-04-27T02:04:37.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.676+0000 7fe61cff9640 1 -- 172.21.15.29:0/1443447582 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fe608009b90 con 0x7fe6200ff740 2024-04-27T02:04:37.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.676+0000 7fe61cff9640 1 --2- 172.21.15.29:0/1443447582 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fe60c06c5f0 0x7fe60c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:37.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.676+0000 7fe61cff9640 1 -- 172.21.15.29:0/1443447582 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fe60808b3d0 con 0x7fe6200ff740 2024-04-27T02:04:37.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.676+0000 7fe61effd640 1 --2- 172.21.15.29:0/1443447582 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fe60c06c5f0 0x7fe60c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:37.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.677+0000 7fe61effd640 1 --2- 172.21.15.29:0/1443447582 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fe60c06c5f0 0x7fe60c06eaa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe610000c00 tx=0x7fe610005fd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:37.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.680+0000 7fe61cff9640 1 -- 172.21.15.29:0/1443447582 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe608055b00 con 0x7fe6200ff740 2024-04-27T02:04:37.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.868+0000 7fe625300640 1 -- 172.21.15.29:0/1443447582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe620107320 con 0x7fe6200ff740 2024-04-27T02:04:37.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.869+0000 7fe61cff9640 1 -- 172.21.15.29:0/1443447582 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+15092 (secure 0 0 0) 0x7fe608059120 con 0x7fe6200ff740 2024-04-27T02:04:37.850 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:04:37.851 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":46,"fsid":"4c587076-0439-11ef-bc93-c7b262605968","created":"2024-04-27T01:56:52.758582+0000","modified":"2024-04-27T02:04:23.878984+0000","last_up_change":"2024-04-27T02:02:48.845017+0000","last_in_change":"2024-04-27T02:02:29.359236+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-27T02:01:30.306687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-27T02:04:08.974343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-27T02:04:09.470413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3686f584-4bb4-4d6a-8daa-93b9184e4bc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":1138569897},{"type":"v1","addr":"172.21.15.29:6803","nonce":1138569897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":1138569897},{"type":"v1","addr":"172.21.15.29:6805","nonce":1138569897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1138569897},{"type":"v1","addr":"172.21.15.29:6809","nonce":1138569897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":1138569897},{"type":"v1","addr":"172.21.15.29:6807","nonce":1138569897}]},"public_addr":"172.21.15.29:6803/1138569897","cluster_addr":"172.21.15.29:6805/1138569897","heartbeat_back_addr":"172.21.15.29:6809/1138569897","heartbeat_front_addr":"172.21.15.29:6807/1138569897","state":["exists","up"]},{"osd":1,"uuid":"ad9bc7ef-74a2-46c2-a201-a2820c9c8923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3705295680},{"type":"v1","addr":"172.21.15.29:6811","nonce":3705295680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3705295680},{"type":"v1","addr":"172.21.15.29:6813","nonce":3705295680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3705295680},{"type":"v1","addr":"172.21.15.29:6817","nonce":3705295680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3705295680},{"type":"v1","addr":"172.21.15.29:6815","nonce":3705295680}]},"public_addr":"172.21.15.29:6811/3705295680","cluster_addr":"172.21.15.29:6813/3705295680","heartbeat_back_addr":"172.21.15.29:6817/3705295680","heartbeat_front_addr":"172.21.15.29:6815/3705295680","state":["exists","up"]},{"osd":2,"uuid":"315cd205-cf98-4743-8b62-9a158d99a4e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3013669823},{"type":"v1","addr":"172.21.15.29:6819","nonce":3013669823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":3013669823},{"type":"v1","addr":"172.21.15.29:6821","nonce":3013669823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3013669823},{"type":"v1","addr":"172.21.15.29:6825","nonce":3013669823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":3013669823},{"type":"v1","addr":"172.21.15.29:6823","nonce":3013669823}]},"public_addr":"172.21.15.29:6819/3013669823","cluster_addr":"172.21.15.29:6821/3013669823","heartbeat_back_addr":"172.21.15.29:6825/3013669823","heartbeat_front_addr":"172.21.15.29:6823/3013669823","state":["exists","up"]},{"osd":3,"uuid":"59548c0e-dc14-4d71-940c-58802631725b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":263370941},{"type":"v1","addr":"172.21.15.52:6801","nonce":263370941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":263370941},{"type":"v1","addr":"172.21.15.52:6803","nonce":263370941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":263370941},{"type":"v1","addr":"172.21.15.52:6807","nonce":263370941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":263370941},{"type":"v1","addr":"172.21.15.52:6805","nonce":263370941}]},"public_addr":"172.21.15.52:6801/263370941","cluster_addr":"172.21.15.52:6803/263370941","heartbeat_back_addr":"172.21.15.52:6807/263370941","heartbeat_front_addr":"172.21.15.52:6805/263370941","state":["exists","up"]},{"osd":4,"uuid":"c7a75f26-ffa7-447e-8205-478e69d303b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":2191623063},{"type":"v1","addr":"172.21.15.52:6809","nonce":2191623063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":2191623063},{"type":"v1","addr":"172.21.15.52:6811","nonce":2191623063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":2191623063},{"type":"v1","addr":"172.21.15.52:6815","nonce":2191623063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":2191623063},{"type":"v1","addr":"172.21.15.52:6813","nonce":2191623063}]},"public_addr":"172.21.15.52:6809/2191623063","cluster_addr":"172.21.15.52:6811/2191623063","heartbeat_back_addr":"172.21.15.52:6815/2191623063","heartbeat_front_addr":"172.21.15.52:6813/2191623063","state":["exists","up"]},{"osd":5,"uuid":"5cad5f7f-079e-476a-aab5-d5328eda9c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":2667246548},{"type":"v1","addr":"172.21.15.52:6817","nonce":2667246548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":2667246548},{"type":"v1","addr":"172.21.15.52:6819","nonce":2667246548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":2667246548},{"type":"v1","addr":"172.21.15.52:6823","nonce":2667246548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":2667246548},{"type":"v1","addr":"172.21.15.52:6821","nonce":2667246548}]},"public_addr":"172.21.15.52:6817/2667246548","cluster_addr":"172.21.15.52:6819/2667246548","heartbeat_back_addr":"172.21.15.52:6823/2667246548","heartbeat_front_addr":"172.21.15.52:6821/2667246548","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:00:30.289475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:00:59.350372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:01:27.504097+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:01:54.076675+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:02:20.069253+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:02:46.979018+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6827/2380229866":"2024-04-28T02:04:23.878963+0000","172.21.15.29:0/1232596821":"2024-04-28T01:57:23.638376+0000","172.21.15.29:6801/3122572856":"2024-04-28T01:57:23.638376+0000","172.21.15.29:0/3724837509":"2024-04-28T01:57:23.638376+0000","172.21.15.29:6800/3122572856":"2024-04-28T01:57:23.638376+0000","172.21.15.29:0/3346693028":"2024-04-28T01:59:07.772795+0000","172.21.15.29:6800/2469926221":"2024-04-28T01:59:07.772795+0000","172.21.15.29:0/297079675":"2024-04-28T01:59:07.772795+0000","172.21.15.29:0/611503923":"2024-04-28T01:57:57.788454+0000","172.21.15.29:0/909831728":"2024-04-28T01:57:23.638376+0000","172.21.15.29:0/1899826075":"2024-04-28T01:59:07.772795+0000","172.21.15.29:6826/2380229866":"2024-04-28T02:04:23.878963+0000","172.21.15.29:0/3675815378":"2024-04-28T01:57:57.788454+0000","172.21.15.29:0/202142181":"2024-04-28T01:57:57.788454+0000","172.21.15.29:6801/3873642219":"2024-04-28T01:57:57.788454+0000","172.21.15.29:6800/3873642219":"2024-04-28T01:57:57.788454+0000","172.21.15.29:6801/2469926221":"2024-04-28T01:59:07.772795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T02:04:37.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.871+0000 7fe625300640 1 -- 172.21.15.29:0/1443447582 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fe60c06c5f0 msgr2=0x7fe60c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:37.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.871+0000 7fe625300640 1 --2- 172.21.15.29:0/1443447582 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fe60c06c5f0 0x7fe60c06eaa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe610000c00 tx=0x7fe610005fd0 comp rx=0 tx=0).stop 2024-04-27T02:04:37.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.871+0000 7fe625300640 1 -- 172.21.15.29:0/1443447582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6200ff740 msgr2=0x7fe620100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:37.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.871+0000 7fe625300640 1 --2- 172.21.15.29:0/1443447582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6200ff740 0x7fe620100b20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe60800d6f0 tx=0x7fe60800dbb0 comp rx=0 tx=0).stop 2024-04-27T02:04:37.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.871+0000 7fe625300640 1 -- 172.21.15.29:0/1443447582 shutdown_connections 2024-04-27T02:04:37.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.871+0000 7fe625300640 1 --2- 172.21.15.29:0/1443447582 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fe60c06c5f0 0x7fe60c06eaa0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:37.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.871+0000 7fe625300640 1 --2- 172.21.15.29:0/1443447582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6200ff740 0x7fe620100b20 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:37.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.871+0000 7fe625300640 1 --2- 172.21.15.29:0/1443447582 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe6200fe550 0x7fe6201024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:37.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.871+0000 7fe625300640 1 -- 172.21.15.29:0/1443447582 >> 172.21.15.29:0/1443447582 conn(0x7fe6200f9ad0 msgr2=0x7fe620068380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:37.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.871+0000 7fe625300640 1 -- 172.21.15.29:0/1443447582 shutdown_connections 2024-04-27T02:04:37.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.871+0000 7fe625300640 1 -- 172.21.15.29:0/1443447582 wait complete. 2024-04-27T02:04:37.864 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:37 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/3842445696' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-27T02:04:37.865 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-27T02:01:30.306687+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-27T02:04:08.974343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-27T02:04:09.470413+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-27T02:04:37.866 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-27T02:04:37.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.992+0000 7fdb7a947640 1 -- 172.21.15.29:0/2529396405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb74102a60 msgr2=0x7fdb74104e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:37.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.992+0000 7fdb7a947640 1 --2- 172.21.15.29:0/2529396405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb74102a60 0x7fdb74104e40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdb64009880 tx=0x7fdb6402bce0 comp rx=0 tx=0).stop 2024-04-27T02:04:37.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.992+0000 7fdb7a947640 1 -- 172.21.15.29:0/2529396405 shutdown_connections 2024-04-27T02:04:37.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.992+0000 7fdb7a947640 1 --2- 172.21.15.29:0/2529396405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb74102a60 0x7fdb74104e40 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:37.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.992+0000 7fdb7a947640 1 --2- 172.21.15.29:0/2529396405 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb74100140 0x7fdb74102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:37.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.992+0000 7fdb7a947640 1 -- 172.21.15.29:0/2529396405 >> 172.21.15.29:0/2529396405 conn(0x7fdb740f9b20 msgr2=0x7fdb740fbf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:37.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.992+0000 7fdb7a947640 1 -- 172.21.15.29:0/2529396405 shutdown_connections 2024-04-27T02:04:37.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.992+0000 7fdb7a947640 1 -- 172.21.15.29:0/2529396405 wait complete. 2024-04-27T02:04:37.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.993+0000 7fdb7a947640 1 Processor -- start 2024-04-27T02:04:37.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.993+0000 7fdb7a947640 1 -- start start 2024-04-27T02:04:37.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.993+0000 7fdb7a947640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb74100140 0x7fdb74194fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:37.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.993+0000 7fdb7a947640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb74102a60 0x7fdb74195510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:37.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.993+0000 7fdb7a947640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb74195ae0 con 0x7fdb74100140 2024-04-27T02:04:37.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.993+0000 7fdb7a947640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb74195c50 con 0x7fdb74102a60 2024-04-27T02:04:37.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.993+0000 7fdb73fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb74100140 0x7fdb74194fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:37.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.993+0000 7fdb73fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb74100140 0x7fdb74194fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35296/0 (socket says 172.21.15.29:35296) 2024-04-27T02:04:37.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.993+0000 7fdb73fff640 1 -- 172.21.15.29:0/3155294716 learned_addr learned my addr 172.21.15.29:0/3155294716 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:04:37.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.994+0000 7fdb73fff640 1 -- 172.21.15.29:0/3155294716 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb74102a60 msgr2=0x7fdb74195510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:37.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.994+0000 7fdb73fff640 1 --2- 172.21.15.29:0/3155294716 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb74102a60 0x7fdb74195510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:37.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.994+0000 7fdb73fff640 1 -- 172.21.15.29:0/3155294716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb64009560 con 0x7fdb74100140 2024-04-27T02:04:37.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.994+0000 7fdb73fff640 1 --2- 172.21.15.29:0/3155294716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb74100140 0x7fdb74194fd0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c00b3d0 tx=0x7fdb5c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:37.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.994+0000 7fdb717fa640 1 -- 172.21.15.29:0/3155294716 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb5c011800 con 0x7fdb74100140 2024-04-27T02:04:37.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.994+0000 7fdb7a947640 1 -- 172.21.15.29:0/3155294716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb7419a6e0 con 0x7fdb74100140 2024-04-27T02:04:37.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.994+0000 7fdb717fa640 1 -- 172.21.15.29:0/3155294716 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdb5c011e10 con 0x7fdb74100140 2024-04-27T02:04:37.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.994+0000 7fdb717fa640 1 -- 172.21.15.29:0/3155294716 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb5c00f2e0 con 0x7fdb74100140 2024-04-27T02:04:37.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.994+0000 7fdb7a947640 1 -- 172.21.15.29:0/3155294716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb7419acc0 con 0x7fdb74100140 2024-04-27T02:04:37.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.995+0000 7fdb7a947640 1 -- 172.21.15.29:0/3155294716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb7418f030 con 0x7fdb74100140 2024-04-27T02:04:37.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.996+0000 7fdb717fa640 1 -- 172.21.15.29:0/3155294716 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fdb5c010380 con 0x7fdb74100140 2024-04-27T02:04:37.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.996+0000 7fdb717fa640 1 --2- 172.21.15.29:0/3155294716 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fdb6006c5f0 0x7fdb6006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:37.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.996+0000 7fdb717fa640 1 -- 172.21.15.29:0/3155294716 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fdb5c08b570 con 0x7fdb74100140 2024-04-27T02:04:37.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.996+0000 7fdb737fe640 1 --2- 172.21.15.29:0/3155294716 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fdb6006c5f0 0x7fdb6006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:37.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:37.997+0000 7fdb737fe640 1 --2- 172.21.15.29:0/3155294716 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fdb6006c5f0 0x7fdb6006eaa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdb6402c7c0 tx=0x7fdb64000bc0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:37.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.001+0000 7fdb717fa640 1 -- 172.21.15.29:0/3155294716 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fdb5c055c70 con 0x7fdb74100140 2024-04-27T02:04:38.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:37 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/3842445696' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-27T02:04:38.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.180+0000 7fdb7a947640 1 -- 172.21.15.29:0/3155294716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdb7402c420 con 0x7fdb74100140 2024-04-27T02:04:38.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.180+0000 7fdb717fa640 1 -- 172.21.15.29:0/3155294716 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdb5c059290 con 0x7fdb74100140 2024-04-27T02:04:38.162 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-04-27T02:04:38.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.182+0000 7fdb7a947640 1 -- 172.21.15.29:0/3155294716 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fdb6006c5f0 msgr2=0x7fdb6006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:38.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.182+0000 7fdb7a947640 1 --2- 172.21.15.29:0/3155294716 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fdb6006c5f0 0x7fdb6006eaa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdb6402c7c0 tx=0x7fdb64000bc0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.182+0000 7fdb7a947640 1 -- 172.21.15.29:0/3155294716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb74100140 msgr2=0x7fdb74194fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:38.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.182+0000 7fdb7a947640 1 --2- 172.21.15.29:0/3155294716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb74100140 0x7fdb74194fd0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c00b3d0 tx=0x7fdb5c00b890 comp rx=0 tx=0).stop 2024-04-27T02:04:38.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.182+0000 7fdb7a947640 1 -- 172.21.15.29:0/3155294716 shutdown_connections 2024-04-27T02:04:38.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.182+0000 7fdb7a947640 1 --2- 172.21.15.29:0/3155294716 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fdb6006c5f0 0x7fdb6006eaa0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.182+0000 7fdb7a947640 1 --2- 172.21.15.29:0/3155294716 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb74102a60 0x7fdb74195510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.182+0000 7fdb7a947640 1 --2- 172.21.15.29:0/3155294716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb74100140 0x7fdb74194fd0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.182+0000 7fdb7a947640 1 -- 172.21.15.29:0/3155294716 >> 172.21.15.29:0/3155294716 conn(0x7fdb740f9b20 msgr2=0x7fdb740fbf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:38.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.182+0000 7fdb7a947640 1 -- 172.21.15.29:0/3155294716 shutdown_connections 2024-04-27T02:04:38.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.182+0000 7fdb7a947640 1 -- 172.21.15.29:0/3155294716 wait complete. 2024-04-27T02:04:38.174 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-27T02:04:38.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.299+0000 7ffbb56a1640 1 -- 172.21.15.29:0/3992307561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbb00ff7c0 msgr2=0x7ffbb00ffc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:38.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.299+0000 7ffbb56a1640 1 --2- 172.21.15.29:0/3992307561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbb00ff7c0 0x7ffbb00ffc30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ffba4009880 tx=0x7ffba402bce0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.299+0000 7ffbb56a1640 1 -- 172.21.15.29:0/3992307561 shutdown_connections 2024-04-27T02:04:38.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.299+0000 7ffbb56a1640 1 --2- 172.21.15.29:0/3992307561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbb00ff7c0 0x7ffbb00ffc30 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.299+0000 7ffbb56a1640 1 --2- 172.21.15.29:0/3992307561 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffbb00fe530 0x7ffbb00fe940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.299+0000 7ffbb56a1640 1 -- 172.21.15.29:0/3992307561 >> 172.21.15.29:0/3992307561 conn(0x7ffbb00f9b30 msgr2=0x7ffbb00fbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:38.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.299+0000 7ffbb56a1640 1 -- 172.21.15.29:0/3992307561 shutdown_connections 2024-04-27T02:04:38.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.299+0000 7ffbb56a1640 1 -- 172.21.15.29:0/3992307561 wait complete. 2024-04-27T02:04:38.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.299+0000 7ffbb56a1640 1 Processor -- start 2024-04-27T02:04:38.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.299+0000 7ffbb56a1640 1 -- start start 2024-04-27T02:04:38.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.300+0000 7ffbb56a1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbb00fe530 0x7ffbb0068ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:38.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.300+0000 7ffbb56a1640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffbb00ff7c0 0x7ffbb0199cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:38.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.300+0000 7ffbb56a1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb019a2a0 con 0x7ffbb00fe530 2024-04-27T02:04:38.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.300+0000 7ffbb56a1640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb019a3e0 con 0x7ffbb00ff7c0 2024-04-27T02:04:38.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.300+0000 7ffbaeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbb00fe530 0x7ffbb0068ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:38.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.300+0000 7ffbaeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbb00fe530 0x7ffbb0068ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35324/0 (socket says 172.21.15.29:35324) 2024-04-27T02:04:38.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.300+0000 7ffbaeffd640 1 -- 172.21.15.29:0/1806865083 learned_addr learned my addr 172.21.15.29:0/1806865083 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:04:38.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.300+0000 7ffbae7fc640 1 --2- 172.21.15.29:0/1806865083 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffbb00ff7c0 0x7ffbb0199cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:38.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.300+0000 7ffbaeffd640 1 -- 172.21.15.29:0/1806865083 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffbb00ff7c0 msgr2=0x7ffbb0199cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:38.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.300+0000 7ffbaeffd640 1 --2- 172.21.15.29:0/1806865083 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffbb00ff7c0 0x7ffbb0199cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.300+0000 7ffbaeffd640 1 -- 172.21.15.29:0/1806865083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba4009560 con 0x7ffbb00fe530 2024-04-27T02:04:38.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.300+0000 7ffbaeffd640 1 --2- 172.21.15.29:0/1806865083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbb00fe530 0x7ffbb0068ec0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ffba000e970 tx=0x7ffba000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:38.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.300+0000 7ffb97fff640 1 -- 172.21.15.29:0/1806865083 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffba000cc90 con 0x7ffbb00fe530 2024-04-27T02:04:38.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.300+0000 7ffb97fff640 1 -- 172.21.15.29:0/1806865083 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ffba000cdf0 con 0x7ffbb00fe530 2024-04-27T02:04:38.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.300+0000 7ffbb56a1640 1 -- 172.21.15.29:0/1806865083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb019a690 con 0x7ffbb00fe530 2024-04-27T02:04:38.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.300+0000 7ffb97fff640 1 -- 172.21.15.29:0/1806865083 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffba0010430 con 0x7ffbb00fe530 2024-04-27T02:04:38.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.300+0000 7ffbb56a1640 1 -- 172.21.15.29:0/1806865083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb019abc0 con 0x7ffbb00fe530 2024-04-27T02:04:38.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.301+0000 7ffbb56a1640 1 -- 172.21.15.29:0/1806865083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb0107090 con 0x7ffbb00fe530 2024-04-27T02:04:38.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.302+0000 7ffb97fff640 1 -- 172.21.15.29:0/1806865083 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7ffba00046c0 con 0x7ffbb00fe530 2024-04-27T02:04:38.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.302+0000 7ffb97fff640 1 --2- 172.21.15.29:0/1806865083 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ffb9806c5f0 0x7ffb9806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:38.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.302+0000 7ffb97fff640 1 -- 172.21.15.29:0/1806865083 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ffba0014070 con 0x7ffbb00fe530 2024-04-27T02:04:38.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.302+0000 7ffbae7fc640 1 --2- 172.21.15.29:0/1806865083 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ffb9806c5f0 0x7ffb9806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:38.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.303+0000 7ffbae7fc640 1 --2- 172.21.15.29:0/1806865083 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ffb9806c5f0 0x7ffb9806eaa0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffba4000c00 tx=0x7ffba4005fd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:38.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.307+0000 7ffb97fff640 1 -- 172.21.15.29:0/1806865083 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ffba0056560 con 0x7ffbb00fe530 2024-04-27T02:04:38.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.481+0000 7ffbb56a1640 1 -- 172.21.15.29:0/1806865083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7ffbb0107300 con 0x7ffbb00fe530 2024-04-27T02:04:38.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.482+0000 7ffb97fff640 1 -- 172.21.15.29:0/1806865083 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7ffba0059b80 con 0x7ffbb00fe530 2024-04-27T02:04:38.463 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-04-27T02:04:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.484+0000 7ffbb56a1640 1 -- 172.21.15.29:0/1806865083 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ffb9806c5f0 msgr2=0x7ffb9806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.484+0000 7ffbb56a1640 1 --2- 172.21.15.29:0/1806865083 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ffb9806c5f0 0x7ffb9806eaa0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffba4000c00 tx=0x7ffba4005fd0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.484+0000 7ffbb56a1640 1 -- 172.21.15.29:0/1806865083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbb00fe530 msgr2=0x7ffbb0068ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.484+0000 7ffbb56a1640 1 --2- 172.21.15.29:0/1806865083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbb00fe530 0x7ffbb0068ec0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ffba000e970 tx=0x7ffba000ed70 comp rx=0 tx=0).stop 2024-04-27T02:04:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.484+0000 7ffbb56a1640 1 -- 172.21.15.29:0/1806865083 shutdown_connections 2024-04-27T02:04:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.484+0000 7ffbb56a1640 1 --2- 172.21.15.29:0/1806865083 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ffb9806c5f0 0x7ffb9806eaa0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.484+0000 7ffbb56a1640 1 --2- 172.21.15.29:0/1806865083 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffbb00ff7c0 0x7ffbb0199cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.484+0000 7ffbb56a1640 1 --2- 172.21.15.29:0/1806865083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbb00fe530 0x7ffbb0068ec0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.484+0000 7ffbb56a1640 1 -- 172.21.15.29:0/1806865083 >> 172.21.15.29:0/1806865083 conn(0x7ffbb00f9b30 msgr2=0x7ffbb01029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.484+0000 7ffbb56a1640 1 -- 172.21.15.29:0/1806865083 shutdown_connections 2024-04-27T02:04:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.484+0000 7ffbb56a1640 1 -- 172.21.15.29:0/1806865083 wait complete. 2024-04-27T02:04:38.476 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-27T02:04:38.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.609+0000 7fcab433c640 1 -- 172.21.15.29:0/3870145061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcaac0ffbc0 msgr2=0x7fcaac0fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:38.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.609+0000 7fcab433c640 1 --2- 172.21.15.29:0/3870145061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcaac0ffbc0 0x7fcaac0fffd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8009880 tx=0x7fcaa802bce0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.609+0000 7fcab433c640 1 -- 172.21.15.29:0/3870145061 shutdown_connections 2024-04-27T02:04:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.609+0000 7fcab433c640 1 --2- 172.21.15.29:0/3870145061 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcaac0fd6b0 0x7fcaac0fdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.609+0000 7fcab433c640 1 --2- 172.21.15.29:0/3870145061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcaac0ffbc0 0x7fcaac0fffd0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.609+0000 7fcab433c640 1 -- 172.21.15.29:0/3870145061 >> 172.21.15.29:0/3870145061 conn(0x7fcaac0f9220 msgr2=0x7fcaac0fb670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.609+0000 7fcab433c640 1 -- 172.21.15.29:0/3870145061 shutdown_connections 2024-04-27T02:04:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.609+0000 7fcab433c640 1 -- 172.21.15.29:0/3870145061 wait complete. 2024-04-27T02:04:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.610+0000 7fcab433c640 1 Processor -- start 2024-04-27T02:04:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.610+0000 7fcab433c640 1 -- start start 2024-04-27T02:04:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.610+0000 7fcab433c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcaac0fd6b0 0x7fcaac19b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.610+0000 7fcab433c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcaac0ffbc0 0x7fcaac19bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.610+0000 7fcab433c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaac19c090 con 0x7fcaac0fd6b0 2024-04-27T02:04:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.610+0000 7fcab433c640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaac19c200 con 0x7fcaac0ffbc0 2024-04-27T02:04:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.610+0000 7fcab20d8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcaac0fd6b0 0x7fcaac19b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.610+0000 7fcab20d8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcaac0fd6b0 0x7fcaac19b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35344/0 (socket says 172.21.15.29:35344) 2024-04-27T02:04:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.610+0000 7fcab20d8640 1 -- 172.21.15.29:0/3155049204 learned_addr learned my addr 172.21.15.29:0/3155049204 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:04:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.610+0000 7fcab18d7640 1 --2- 172.21.15.29:0/3155049204 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcaac0ffbc0 0x7fcaac19bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.610+0000 7fcab20d8640 1 -- 172.21.15.29:0/3155049204 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcaac0ffbc0 msgr2=0x7fcaac19bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.610+0000 7fcab20d8640 1 --2- 172.21.15.29:0/3155049204 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcaac0ffbc0 0x7fcaac19bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.610+0000 7fcab20d8640 1 -- 172.21.15.29:0/3155049204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa8009560 con 0x7fcaac0fd6b0 2024-04-27T02:04:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.610+0000 7fcab20d8640 1 --2- 172.21.15.29:0/3155049204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcaac0fd6b0 0x7fcaac19b580 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8009230 tx=0x7fcaa802cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.611+0000 7fcaa37fe640 1 -- 172.21.15.29:0/3155049204 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcaa8004420 con 0x7fcaac0fd6b0 2024-04-27T02:04:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.611+0000 7fcaa37fe640 1 -- 172.21.15.29:0/3155049204 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcaa80054b0 con 0x7fcaac0fd6b0 2024-04-27T02:04:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.611+0000 7fcaa37fe640 1 -- 172.21.15.29:0/3155049204 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcaa8034440 con 0x7fcaac0fd6b0 2024-04-27T02:04:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.611+0000 7fcab433c640 1 -- 172.21.15.29:0/3155049204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaac1a0c30 con 0x7fcaac0fd6b0 2024-04-27T02:04:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.611+0000 7fcab433c640 1 -- 172.21.15.29:0/3155049204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaac1a1130 con 0x7fcaac0fd6b0 2024-04-27T02:04:38.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.612+0000 7fcab433c640 1 -- 172.21.15.29:0/3155049204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaac1057d0 con 0x7fcaac0fd6b0 2024-04-27T02:04:38.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.612+0000 7fcaa37fe640 1 -- 172.21.15.29:0/3155049204 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fcaa80037d0 con 0x7fcaac0fd6b0 2024-04-27T02:04:38.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.613+0000 7fcaa37fe640 1 --2- 172.21.15.29:0/3155049204 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fca9806c5f0 0x7fca9806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:38.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.613+0000 7fcaa37fe640 1 -- 172.21.15.29:0/3155049204 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fcaa80ade10 con 0x7fcaac0fd6b0 2024-04-27T02:04:38.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.613+0000 7fcab18d7640 1 --2- 172.21.15.29:0/3155049204 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fca9806c5f0 0x7fca9806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:38.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.613+0000 7fcab18d7640 1 --2- 172.21.15.29:0/3155049204 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fca9806c5f0 0x7fca9806eaa0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcaac19ca90 tx=0x7fca9c005f90 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:38.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.618+0000 7fcaa37fe640 1 -- 172.21.15.29:0/3155049204 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fcaa8078490 con 0x7fcaac0fd6b0 2024-04-27T02:04:38.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.801+0000 7fcab433c640 1 -- 172.21.15.29:0/3155049204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fcaac10a590 con 0x7fcaac0fd6b0 2024-04-27T02:04:38.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.802+0000 7fcaa37fe640 1 -- 172.21.15.29:0/3155049204 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7fcaa804b020 con 0x7fcaac0fd6b0 2024-04-27T02:04:38.783 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-04-27T02:04:38.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.804+0000 7fcab433c640 1 -- 172.21.15.29:0/3155049204 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fca9806c5f0 msgr2=0x7fca9806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:38.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.804+0000 7fcab433c640 1 --2- 172.21.15.29:0/3155049204 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fca9806c5f0 0x7fca9806eaa0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcaac19ca90 tx=0x7fca9c005f90 comp rx=0 tx=0).stop 2024-04-27T02:04:38.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.804+0000 7fcab433c640 1 -- 172.21.15.29:0/3155049204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcaac0fd6b0 msgr2=0x7fcaac19b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:38.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.804+0000 7fcab433c640 1 --2- 172.21.15.29:0/3155049204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcaac0fd6b0 0x7fcaac19b580 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8009230 tx=0x7fcaa802cec0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.804+0000 7fcab433c640 1 -- 172.21.15.29:0/3155049204 shutdown_connections 2024-04-27T02:04:38.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.804+0000 7fcab433c640 1 --2- 172.21.15.29:0/3155049204 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fca9806c5f0 0x7fca9806eaa0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.804+0000 7fcab433c640 1 --2- 172.21.15.29:0/3155049204 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcaac0ffbc0 0x7fcaac19bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.804+0000 7fcab433c640 1 --2- 172.21.15.29:0/3155049204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcaac0fd6b0 0x7fcaac19b580 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.804+0000 7fcab433c640 1 -- 172.21.15.29:0/3155049204 >> 172.21.15.29:0/3155049204 conn(0x7fcaac0f9220 msgr2=0x7fcaac1029a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:38.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.804+0000 7fcab433c640 1 -- 172.21.15.29:0/3155049204 shutdown_connections 2024-04-27T02:04:38.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.804+0000 7fcab433c640 1 -- 172.21.15.29:0/3155049204 wait complete. 2024-04-27T02:04:38.796 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-27T02:04:38.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.921+0000 7fc16587a640 1 -- 172.21.15.29:0/3444560541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1600ff730 msgr2=0x7fc1600ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:38.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.921+0000 7fc16587a640 1 --2- 172.21.15.29:0/3444560541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1600ff730 0x7fc1600ffba0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc148009880 tx=0x7fc14802bce0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.921+0000 7fc16587a640 1 -- 172.21.15.29:0/3444560541 shutdown_connections 2024-04-27T02:04:38.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.921+0000 7fc16587a640 1 --2- 172.21.15.29:0/3444560541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1600ff730 0x7fc1600ffba0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.921+0000 7fc16587a640 1 --2- 172.21.15.29:0/3444560541 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1600fe540 0x7fc1600fe950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.921+0000 7fc16587a640 1 -- 172.21.15.29:0/3444560541 >> 172.21.15.29:0/3444560541 conn(0x7fc1600f9ac0 msgr2=0x7fc1600fbef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:38.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.921+0000 7fc16587a640 1 -- 172.21.15.29:0/3444560541 shutdown_connections 2024-04-27T02:04:38.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.921+0000 7fc16587a640 1 -- 172.21.15.29:0/3444560541 wait complete. 2024-04-27T02:04:38.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.922+0000 7fc16587a640 1 Processor -- start 2024-04-27T02:04:38.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.922+0000 7fc16587a640 1 -- start start 2024-04-27T02:04:38.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.922+0000 7fc16587a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1600fe540 0x7fc160068f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:38.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.922+0000 7fc16587a640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1600ff730 0x7fc160199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:38.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.922+0000 7fc16587a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16019a380 con 0x7fc1600fe540 2024-04-27T02:04:38.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.922+0000 7fc16587a640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16019a4f0 con 0x7fc1600ff730 2024-04-27T02:04:38.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.922+0000 7fc15effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1600fe540 0x7fc160068f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:38.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.922+0000 7fc15effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1600fe540 0x7fc160068f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35360/0 (socket says 172.21.15.29:35360) 2024-04-27T02:04:38.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.922+0000 7fc15effd640 1 -- 172.21.15.29:0/2963917580 learned_addr learned my addr 172.21.15.29:0/2963917580 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:04:38.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.923+0000 7fc15e7fc640 1 --2- 172.21.15.29:0/2963917580 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1600ff730 0x7fc160199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:38.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.923+0000 7fc15effd640 1 -- 172.21.15.29:0/2963917580 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1600ff730 msgr2=0x7fc160199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:38.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.923+0000 7fc15effd640 1 --2- 172.21.15.29:0/2963917580 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1600ff730 0x7fc160199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:38.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.923+0000 7fc15effd640 1 -- 172.21.15.29:0/2963917580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc148009560 con 0x7fc1600fe540 2024-04-27T02:04:38.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.923+0000 7fc15e7fc640 1 --2- 172.21.15.29:0/2963917580 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1600ff730 0x7fc160199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T02:04:38.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.923+0000 7fc15effd640 1 --2- 172.21.15.29:0/2963917580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1600fe540 0x7fc160068f50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc15000b3d0 tx=0x7fc15000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:38.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.923+0000 7fc164878640 1 -- 172.21.15.29:0/2963917580 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc150011800 con 0x7fc1600fe540 2024-04-27T02:04:38.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.923+0000 7fc16587a640 1 -- 172.21.15.29:0/2963917580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc16019a7d0 con 0x7fc1600fe540 2024-04-27T02:04:38.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.923+0000 7fc164878640 1 -- 172.21.15.29:0/2963917580 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc150011e10 con 0x7fc1600fe540 2024-04-27T02:04:38.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.923+0000 7fc164878640 1 -- 172.21.15.29:0/2963917580 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc15000f510 con 0x7fc1600fe540 2024-04-27T02:04:38.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.923+0000 7fc16587a640 1 -- 172.21.15.29:0/2963917580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc16019adb0 con 0x7fc1600fe540 2024-04-27T02:04:38.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.924+0000 7fc16587a640 1 -- 172.21.15.29:0/2963917580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc160107330 con 0x7fc1600fe540 2024-04-27T02:04:38.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.925+0000 7fc164878640 1 -- 172.21.15.29:0/2963917580 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fc150010380 con 0x7fc1600fe540 2024-04-27T02:04:38.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.925+0000 7fc164878640 1 --2- 172.21.15.29:0/2963917580 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fc14c06c5f0 0x7fc14c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:38.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.925+0000 7fc164878640 1 -- 172.21.15.29:0/2963917580 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fc15008b690 con 0x7fc1600fe540 2024-04-27T02:04:38.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.925+0000 7fc15e7fc640 1 --2- 172.21.15.29:0/2963917580 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fc14c06c5f0 0x7fc14c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:38.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.925+0000 7fc15e7fc640 1 --2- 172.21.15.29:0/2963917580 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fc14c06c5f0 0x7fc14c06eaa0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc148036040 tx=0x7fc14802cf90 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:38.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:38.929+0000 7fc164878640 1 -- 172.21.15.29:0/2963917580 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc150055d10 con 0x7fc1600fe540 2024-04-27T02:04:39.057 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:38 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:39.057 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:38 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:04:39.057 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:38 smithi029 ceph-mon[26769]: pgmap v199: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.9 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-27T02:04:39.057 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:38 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/1443447582' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T02:04:39.057 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:38 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/3155294716' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T02:04:39.057 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:38 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/1806865083' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-27T02:04:39.057 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:38 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/3155049204' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-27T02:04:39.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:39.151+0000 7fc16587a640 1 -- 172.21.15.29:0/2963917580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc16019b030 con 0x7fc1600fe540 2024-04-27T02:04:39.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:39.151+0000 7fc164878640 1 -- 172.21.15.29:0/2963917580 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4747 (secure 0 0 0) 0x7fc150059330 con 0x7fc1600fe540 2024-04-27T02:04:39.133 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:04:39.133 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14498,"name":"cephfs.smithi029.dyrjrd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6827/3428808494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3428808494},{"type":"v1","addr":"172.21.15.29:6827","nonce":3428808494}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":24325,"name":"cephfs.smithi052.nbjpjd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.52:6827/3513632158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":3513632158},{"type":"v1","addr":"172.21.15.52:6827","nonce":3513632158}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-27T02:04:10.035536+0000","modified":"2024-04-27T02:04:26.892992+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14484":{"gid":14484,"name":"cephfs.smithi029.njjezc","rank":0,"incarnation":15,"state":"up:active","state_seq":5,"addr":"172.21.15.29:6829/1353923457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1353923457},{"type":"v1","addr":"172.21.15.29:6829","nonce":1353923457}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14494":{"gid":14494,"name":"cephfs.smithi052.yylmkb","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6825/2352841895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":2352841895},{"type":"v1","addr":"172.21.15.52:6825","nonce":2352841895}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-27T02:04:39.133 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 19 2024-04-27T02:04:39.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:39.153+0000 7fc16587a640 1 -- 172.21.15.29:0/2963917580 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fc14c06c5f0 msgr2=0x7fc14c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:39.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:39.153+0000 7fc16587a640 1 --2- 172.21.15.29:0/2963917580 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fc14c06c5f0 0x7fc14c06eaa0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc148036040 tx=0x7fc14802cf90 comp rx=0 tx=0).stop 2024-04-27T02:04:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:39.153+0000 7fc16587a640 1 -- 172.21.15.29:0/2963917580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1600fe540 msgr2=0x7fc160068f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:39.153+0000 7fc16587a640 1 --2- 172.21.15.29:0/2963917580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1600fe540 0x7fc160068f50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc15000b3d0 tx=0x7fc15000b890 comp rx=0 tx=0).stop 2024-04-27T02:04:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:39.153+0000 7fc16587a640 1 -- 172.21.15.29:0/2963917580 shutdown_connections 2024-04-27T02:04:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:39.153+0000 7fc16587a640 1 --2- 172.21.15.29:0/2963917580 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fc14c06c5f0 0x7fc14c06eaa0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:39.153+0000 7fc16587a640 1 --2- 172.21.15.29:0/2963917580 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1600ff730 0x7fc160199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:39.153+0000 7fc16587a640 1 --2- 172.21.15.29:0/2963917580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1600fe540 0x7fc160068f50 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:39.153+0000 7fc16587a640 1 -- 172.21.15.29:0/2963917580 >> 172.21.15.29:0/2963917580 conn(0x7fc1600f9ac0 msgr2=0x7fc160102960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:39.153+0000 7fc16587a640 1 -- 172.21.15.29:0/2963917580 shutdown_connections 2024-04-27T02:04:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:39.153+0000 7fc16587a640 1 -- 172.21.15.29:0/2963917580 wait complete. 2024-04-27T02:04:39.145 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 19, 'max_mds': 1, 'flags': 50} 2024-04-27T02:04:39.146 INFO:teuthology.run_tasks:Running task kclient... 2024-04-27T02:04:39.170 INFO:tasks.kclient:Mounting kernel clients... 2024-04-27T02:04:39.170 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-04-27T02:04:39.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T02:04:39.170 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2024-04-27T02:04:39.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T02:04:39.186 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link delete ceph-brx 2024-04-27T02:04:39.253 INFO:teuthology.orchestra.run.smithi029.stderr:Cannot find device "ceph-brx" 2024-04-27T02:04:39.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:38 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:39.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:38 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:04:39.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:38 smithi052 ceph-mon[32586]: pgmap v199: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.9 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-27T02:04:39.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:38 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/1443447582' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T02:04:39.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:38 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/3155294716' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T02:04:39.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:38 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/1806865083' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-27T02:04:39.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:38 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/3155049204' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-27T02:04:39.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T02:04:39.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T02:04:39.255 DEBUG:teuthology.orchestra.run.smithi052:> ip netns list 2024-04-27T02:04:39.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T02:04:39.271 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link delete ceph-brx 2024-04-27T02:04:39.341 INFO:teuthology.orchestra.run.smithi052.stderr:Cannot find device "ceph-brx" 2024-04-27T02:04:39.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T02:04:39.342 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-04-27T02:04:39.342 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-27T02:04:39.342 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-27T02:04:39.343 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-27T02:04:39.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:39 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/2963917580' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T02:04:40.199 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:04:40.247 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:39 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/2963917580' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T02:04:40.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:40 smithi029 ceph-mon[26769]: pgmap v200: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-27T02:04:41.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:40 smithi052 ceph-mon[32586]: pgmap v200: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-27T02:04:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.902+0000 7fefa88d7700 1 -- 172.21.15.29:0/1264346254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefa40ffbd0 msgr2=0x7fefa4100040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.902+0000 7fefa88d7700 1 --2- 172.21.15.29:0/1264346254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefa40ffbd0 0x7fefa4100040 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fef98009910 tx=0x7fef98009c20 comp rx=0 tx=0).stop 2024-04-27T02:04:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.902+0000 7fefa88d7700 1 -- 172.21.15.29:0/1264346254 shutdown_connections 2024-04-27T02:04:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.902+0000 7fefa88d7700 1 --2- 172.21.15.29:0/1264346254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefa40ffbd0 0x7fefa4100040 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.902+0000 7fefa88d7700 1 --2- 172.21.15.29:0/1264346254 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fefa4104130 0x7fefa4104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.902+0000 7fefa88d7700 1 -- 172.21.15.29:0/1264346254 >> 172.21.15.29:0/1264346254 conn(0x7fefa4075130 msgr2=0x7fefa4075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.902+0000 7fefa88d7700 1 -- 172.21.15.29:0/1264346254 shutdown_connections 2024-04-27T02:04:41.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.902+0000 7fefa88d7700 1 -- 172.21.15.29:0/1264346254 wait complete. 2024-04-27T02:04:41.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.903+0000 7fefa88d7700 1 Processor -- start 2024-04-27T02:04:41.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.903+0000 7fefa88d7700 1 -- start start 2024-04-27T02:04:41.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.903+0000 7fefa88d7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fefa40ffbd0 0x7fefa4196ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:41.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.903+0000 7fefa88d7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefa4104130 0x7fefa41971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:41.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.903+0000 7fefa88d7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa4197800 con 0x7fefa4104130 2024-04-27T02:04:41.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.903+0000 7fefa88d7700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa4197940 con 0x7fefa40ffbd0 2024-04-27T02:04:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.903+0000 7fefa1d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefa4104130 0x7fefa41971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.903+0000 7fefa1d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefa4104130 0x7fefa41971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35366/0 (socket says 172.21.15.29:35366) 2024-04-27T02:04:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.903+0000 7fefa1d9b700 1 -- 172.21.15.29:0/2236129213 learned_addr learned my addr 172.21.15.29:0/2236129213 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:04:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.903+0000 7fefa259c700 1 --2- 172.21.15.29:0/2236129213 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fefa40ffbd0 0x7fefa4196ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.904+0000 7fefa1d9b700 1 -- 172.21.15.29:0/2236129213 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fefa40ffbd0 msgr2=0x7fefa4196ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.904+0000 7fefa1d9b700 1 --2- 172.21.15.29:0/2236129213 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fefa40ffbd0 0x7fefa4196ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.904+0000 7fefa1d9b700 1 -- 172.21.15.29:0/2236129213 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef980095f0 con 0x7fefa4104130 2024-04-27T02:04:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.904+0000 7fefa1d9b700 1 --2- 172.21.15.29:0/2236129213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefa4104130 0x7fefa41971e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fef98006010 tx=0x7fef9800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.904+0000 7fef937fe700 1 -- 172.21.15.29:0/2236129213 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef9801d070 con 0x7fefa4104130 2024-04-27T02:04:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.904+0000 7fefa88d7700 1 -- 172.21.15.29:0/2236129213 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefa419fd10 con 0x7fefa4104130 2024-04-27T02:04:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.904+0000 7fef937fe700 1 -- 172.21.15.29:0/2236129213 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fef9800f460 con 0x7fefa4104130 2024-04-27T02:04:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.904+0000 7fef937fe700 1 -- 172.21.15.29:0/2236129213 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef98017630 con 0x7fefa4104130 2024-04-27T02:04:41.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.904+0000 7fefa88d7700 1 -- 172.21.15.29:0/2236129213 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa41a0120 con 0x7fefa4104130 2024-04-27T02:04:41.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.905+0000 7fefa88d7700 1 -- 172.21.15.29:0/2236129213 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefa404f2a0 con 0x7fefa4104130 2024-04-27T02:04:41.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.906+0000 7fef937fe700 1 -- 172.21.15.29:0/2236129213 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fef98003a60 con 0x7fefa4104130 2024-04-27T02:04:41.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.906+0000 7fef937fe700 1 --2- 172.21.15.29:0/2236129213 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fef8c06c680 0x7fef8c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:41.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.906+0000 7fef937fe700 1 -- 172.21.15.29:0/2236129213 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fef9808cd70 con 0x7fefa4104130 2024-04-27T02:04:41.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.906+0000 7fefa259c700 1 --2- 172.21.15.29:0/2236129213 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fef8c06c680 0x7fef8c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:41.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.906+0000 7fefa259c700 1 --2- 172.21.15.29:0/2236129213 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fef8c06c680 0x7fef8c06eb30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fef94005950 tx=0x7fef9400b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:41.911+0000 7fef937fe700 1 -- 172.21.15.29:0/2236129213 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fef980574b0 con 0x7fefa4104130 2024-04-27T02:04:42.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:42.160+0000 7fefa88d7700 1 -- 172.21.15.29:0/2236129213 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fefa41a03d0 con 0x7fefa4104130 2024-04-27T02:04:42.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:42.160+0000 7fef937fe700 1 -- 172.21.15.29:0/2236129213 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) v1 ==== 53+0+83 (secure 0 0 0) 0x7fef9805aad0 con 0x7fefa4104130 2024-04-27T02:04:42.142 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-27T02:04:42.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:42.162+0000 7fefa88d7700 1 -- 172.21.15.29:0/2236129213 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fef8c06c680 msgr2=0x7fef8c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:42.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:42.162+0000 7fefa88d7700 1 --2- 172.21.15.29:0/2236129213 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fef8c06c680 0x7fef8c06eb30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fef94005950 tx=0x7fef9400b410 comp rx=0 tx=0).stop 2024-04-27T02:04:42.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:42.162+0000 7fefa88d7700 1 -- 172.21.15.29:0/2236129213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefa4104130 msgr2=0x7fefa41971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:42.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:42.162+0000 7fefa88d7700 1 --2- 172.21.15.29:0/2236129213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefa4104130 0x7fefa41971e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fef98006010 tx=0x7fef9800bd70 comp rx=0 tx=0).stop 2024-04-27T02:04:42.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:42.162+0000 7fefa88d7700 1 -- 172.21.15.29:0/2236129213 shutdown_connections 2024-04-27T02:04:42.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:42.162+0000 7fefa88d7700 1 --2- 172.21.15.29:0/2236129213 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fef8c06c680 0x7fef8c06eb30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:42.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:42.162+0000 7fefa88d7700 1 --2- 172.21.15.29:0/2236129213 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fefa40ffbd0 0x7fefa4196ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:42.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:42.162+0000 7fefa88d7700 1 --2- 172.21.15.29:0/2236129213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefa4104130 0x7fefa41971e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:42.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:42.162+0000 7fefa88d7700 1 -- 172.21.15.29:0/2236129213 >> 172.21.15.29:0/2236129213 conn(0x7fefa4075130 msgr2=0x7fefa40feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:42.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:42.163+0000 7fefa88d7700 1 -- 172.21.15.29:0/2236129213 shutdown_connections 2024-04-27T02:04:42.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:42.163+0000 7fefa88d7700 1 -- 172.21.15.29:0/2236129213 wait complete. 2024-04-27T02:04:42.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:42 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:42.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:42 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:42.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:42 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:04:42.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:42 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:04:42.251 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:42 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:42.252 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:42 smithi052 ceph-mon[32586]: pgmap v201: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-27T02:04:42.450 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:42 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:42.450 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:42 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:42.450 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:42 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:04:42.450 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:42 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:04:42.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:42 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:42.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:42 smithi029 ceph-mon[26769]: pgmap v201: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-27T02:04:42.767 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-27T02:04:42.767 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-27T02:04:42.767 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2024-04-27T02:04:42.767 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-27T02:04:42.767 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-27T02:04:42.767 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-27T02:04:42.767 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-27T02:04:42.767 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-27T02:04:42.767 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-27T02:04:42.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T02:04:42.768 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2024-04-27T02:04:42.784 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-27T02:04:42.785 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-27T02:04:42.785 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2024-04-27T02:04:42.785 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-04-27T02:04:42.785 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2024-04-27T02:04:42.785 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-04-27T02:04:42.785 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-27T02:04:42.785 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2024-04-27T02:04:42.785 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2024-04-27T02:04:42.785 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-27T02:04:42.785 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2024-04-27T02:04:42.786 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2024-04-27T02:04:42.786 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-27T02:04:42.786 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2024-04-27T02:04:42.786 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2024-04-27T02:04:42.786 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-27T02:04:42.786 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 42219sec preferred_lft 42219sec 2024-04-27T02:04:42.786 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2024-04-27T02:04:42.786 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-04-27T02:04:42.786 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-27T02:04:42.786 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2024-04-27T02:04:42.786 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2024-04-27T02:04:42.787 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-27T02:04:42.787 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T02:04:42.787 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-04-27T02:04:42.787 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add name ceph-brx type bridge 2024-04-27T02:04:42.787 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr flush dev ceph-brx 2024-04-27T02:04:42.787 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set ceph-brx up 2024-04-27T02:04:42.788 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-27T02:04:42.788 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-04-27T02:04:42.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T02:04:42.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T02:04:42.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T02:04:42.969 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-27T02:04:42.993 INFO:teuthology.orchestra.run.smithi029.stdout:1 2024-04-27T02:04:42.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T02:04:42.995 DEBUG:teuthology.orchestra.run.smithi029:> route 2024-04-27T02:04:43.102 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2024-04-27T02:04:43.103 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-27T02:04:43.103 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-27T02:04:43.103 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-27T02:04:43.103 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-27T02:04:43.103 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T02:04:43.103 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-04-27T02:04:43.103 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-27T02:04:43.104 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-27T02:04:43.104 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-27T02:04:43.104 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-04-27T02:04:43.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T02:04:43.266 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:43 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/2236129213' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-27T02:04:43.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T02:04:43.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T02:04:43.347 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2024-04-27T02:04:43.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T02:04:43.363 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list-id 2024-04-27T02:04:43.419 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T02:04:43.419 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-04-27T02:04:43.419 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-27T02:04:43.419 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-27T02:04:43.419 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-04-27T02:04:43.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:43 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/2236129213' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-27T02:04:43.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T02:04:43.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T02:04:43.536 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-27T02:04:43.536 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T02:04:43.536 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-04-27T02:04:43.536 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-27T02:04:43.536 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-27T02:04:43.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-27T02:04:43.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-27T02:04:43.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-27T02:04:43.537 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-04-27T02:04:43.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T02:04:43.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T02:04:43.676 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T02:04:43.676 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-04-27T02:04:43.676 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2024-04-27T02:04:43.676 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set dev brx.0 master ceph-brx 2024-04-27T02:04:43.676 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-04-27T02:04:43.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T02:04:43.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T02:04:43.755 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-27T02:04:43.755 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-27T02:04:43.770 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-27T02:04:43.771 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-27T02:04:43.771 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-27T02:04:43.826 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-27T02:04:43.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T02:04:43.826 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-27T02:04:43.936 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-27T02:04:43.936 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-27T02:04:43.936 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2024-04-27T02:04:44.051 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-27T02:04:44.051 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-27T02:04:44.051 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2024-04-27T02:04:44.051 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.52:3300" 2024-04-27T02:04:44.051 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-27T02:04:44.057 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-27T02:04:44.057 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-27T02:04:44.057 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2024-04-27T02:04:44.057 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.52:3300" 2024-04-27T02:04:44.057 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-27T02:04:44.058 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.52:6789" 2024-04-27T02:04:44.058 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.52:6789:/ 2024-04-27T02:04:44.058 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=4c587076-0439-11ef-bc93-c7b262605968" will pass to kernel 2024-04-27T02:04:44.059 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-27T02:04:44.059 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2024-04-27T02:04:44.059 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-27T02:04:44.059 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2024-04-27T02:04:44.059 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-27T02:04:44.061 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-27T02:04:44.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T02:04:44.061 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-27T02:04:44.087 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-27T02:04:44.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:44 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-27T02:04:44.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T02:04:44.188 INFO:teuthology.orchestra.run.smithi029.stdout:client24363 2024-04-27T02:04:44.189 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-27T02:04:44.189 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-27T02:04:44.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:44 smithi029 ceph-mon[26769]: pgmap v202: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-27T02:04:44.205 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-27T02:04:44.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:44 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-27T02:04:44.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T02:04:44.308 INFO:teuthology.orchestra.run.smithi029.stdout:client24363 2024-04-27T02:04:44.308 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-27T02:04:44.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:44 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-27T02:04:44.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T02:04:44.369 INFO:teuthology.orchestra.run.smithi029.stdout:client24363 2024-04-27T02:04:44.369 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/4c587076-0439-11ef-bc93-c7b262605968.client24363/status') 2024-04-27T02:04:44.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T02:04:44.432 INFO:teuthology.orchestra.run.smithi029.stdout:instance: client.24363 (3)172.21.15.29:0/2637728762 2024-04-27T02:04:44.432 INFO:teuthology.orchestra.run.smithi029.stdout:blocklisted: false 2024-04-27T02:04:44.432 INFO:teuthology.orchestra.run.smithi029.stderr:0+1 records in 2024-04-27T02:04:44.432 INFO:teuthology.orchestra.run.smithi029.stderr:0+1 records out 2024-04-27T02:04:44.432 INFO:teuthology.orchestra.run.smithi029.stderr:71 bytes copied, 2.5064e-05 s, 2.8 MB/s 2024-04-27T02:04:44.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T02:04:44.439 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-04-27T02:04:44.440 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-27T02:04:44.440 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-27T02:04:44.440 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 4c587076-0439-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-27T02:04:44.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:44 smithi052 ceph-mon[32586]: pgmap v202: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-27T02:04:44.653 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:04:46.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.293+0000 7fdaa5deb700 1 -- 172.21.15.29:0/2333491819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaa01004e0 msgr2=0x7fdaa01008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:46.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.293+0000 7fdaa5deb700 1 --2- 172.21.15.29:0/2333491819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaa01004e0 0x7fdaa01008b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fda90009910 tx=0x7fda90009c20 comp rx=0 tx=0).stop 2024-04-27T02:04:46.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.293+0000 7fdaa5deb700 1 -- 172.21.15.29:0/2333491819 shutdown_connections 2024-04-27T02:04:46.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.293+0000 7fdaa5deb700 1 --2- 172.21.15.29:0/2333491819 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdaa0100e80 0x7fdaa0104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:46.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.293+0000 7fdaa5deb700 1 --2- 172.21.15.29:0/2333491819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaa01004e0 0x7fdaa01008b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:46.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.293+0000 7fdaa5deb700 1 -- 172.21.15.29:0/2333491819 >> 172.21.15.29:0/2333491819 conn(0x7fdaa00fbda0 msgr2=0x7fdaa00fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:46.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.293+0000 7fdaa5deb700 1 -- 172.21.15.29:0/2333491819 shutdown_connections 2024-04-27T02:04:46.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.293+0000 7fdaa5deb700 1 -- 172.21.15.29:0/2333491819 wait complete. 2024-04-27T02:04:46.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.293+0000 7fdaa5deb700 1 Processor -- start 2024-04-27T02:04:46.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.294+0000 7fdaa5deb700 1 -- start start 2024-04-27T02:04:46.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.294+0000 7fdaa5deb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaa01004e0 0x7fdaa0195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:46.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.294+0000 7fdaa5deb700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdaa0100e80 0x7fdaa0196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:46.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.294+0000 7fdaa5deb700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa0196b10 con 0x7fdaa01004e0 2024-04-27T02:04:46.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.294+0000 7fdaa5deb700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa019a8a0 con 0x7fdaa0100e80 2024-04-27T02:04:46.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.294+0000 7fda9f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaa01004e0 0x7fdaa0195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:46.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.294+0000 7fda9f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaa01004e0 0x7fdaa0195ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35386/0 (socket says 172.21.15.29:35386) 2024-04-27T02:04:46.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.294+0000 7fda9f7fe700 1 -- 172.21.15.29:0/336069706 learned_addr learned my addr 172.21.15.29:0/336069706 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:04:46.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.294+0000 7fda9effd700 1 --2- 172.21.15.29:0/336069706 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdaa0100e80 0x7fdaa0196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:46.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.294+0000 7fda9f7fe700 1 -- 172.21.15.29:0/336069706 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdaa0100e80 msgr2=0x7fdaa0196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:46.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.294+0000 7fda9f7fe700 1 --2- 172.21.15.29:0/336069706 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdaa0100e80 0x7fdaa0196430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:46.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.294+0000 7fda9f7fe700 1 -- 172.21.15.29:0/336069706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda900095f0 con 0x7fdaa01004e0 2024-04-27T02:04:46.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.294+0000 7fda9effd700 1 --2- 172.21.15.29:0/336069706 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdaa0100e80 0x7fdaa0196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T02:04:46.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.294+0000 7fda9f7fe700 1 --2- 172.21.15.29:0/336069706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaa01004e0 0x7fdaa0195ef0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fda90006010 tx=0x7fda9000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:46.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.295+0000 7fda9cff9700 1 -- 172.21.15.29:0/336069706 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda9001d070 con 0x7fdaa01004e0 2024-04-27T02:04:46.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.295+0000 7fdaa5deb700 1 -- 172.21.15.29:0/336069706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa019ab20 con 0x7fdaa01004e0 2024-04-27T02:04:46.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.295+0000 7fda9cff9700 1 -- 172.21.15.29:0/336069706 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fda9000f460 con 0x7fdaa01004e0 2024-04-27T02:04:46.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.295+0000 7fda9cff9700 1 -- 172.21.15.29:0/336069706 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda90021640 con 0x7fdaa01004e0 2024-04-27T02:04:46.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.295+0000 7fdaa5deb700 1 -- 172.21.15.29:0/336069706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa019b010 con 0x7fdaa01004e0 2024-04-27T02:04:46.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.296+0000 7fdaa5deb700 1 -- 172.21.15.29:0/336069706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa01087f0 con 0x7fdaa01004e0 2024-04-27T02:04:46.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.296+0000 7fda9cff9700 1 -- 172.21.15.29:0/336069706 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fda9000faf0 con 0x7fdaa01004e0 2024-04-27T02:04:46.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.297+0000 7fda9cff9700 1 --2- 172.21.15.29:0/336069706 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fda8806c680 0x7fda8806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:46.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.297+0000 7fda9cff9700 1 -- 172.21.15.29:0/336069706 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fda9008cc50 con 0x7fdaa01004e0 2024-04-27T02:04:46.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.297+0000 7fda9effd700 1 --2- 172.21.15.29:0/336069706 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fda8806c680 0x7fda8806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:46.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.297+0000 7fda9effd700 1 --2- 172.21.15.29:0/336069706 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fda8806c680 0x7fda8806eb30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fda94005fd0 tx=0x7fda94005dc0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:46.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.301+0000 7fda9cff9700 1 -- 172.21.15.29:0/336069706 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fda900572d0 con 0x7fdaa01004e0 2024-04-27T02:04:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.551+0000 7fdaa5deb700 1 -- 172.21.15.29:0/336069706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdaa01081d0 con 0x7fdaa01004e0 2024-04-27T02:04:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.551+0000 7fda9cff9700 1 -- 172.21.15.29:0/336069706 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) v1 ==== 53+0+83 (secure 0 0 0) 0x7fda90026070 con 0x7fdaa01004e0 2024-04-27T02:04:46.531 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-27T02:04:46.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.553+0000 7fdaa5deb700 1 -- 172.21.15.29:0/336069706 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fda8806c680 msgr2=0x7fda8806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:46.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.553+0000 7fdaa5deb700 1 --2- 172.21.15.29:0/336069706 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fda8806c680 0x7fda8806eb30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fda94005fd0 tx=0x7fda94005dc0 comp rx=0 tx=0).stop 2024-04-27T02:04:46.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.553+0000 7fdaa5deb700 1 -- 172.21.15.29:0/336069706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaa01004e0 msgr2=0x7fdaa0195ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:46.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.553+0000 7fdaa5deb700 1 --2- 172.21.15.29:0/336069706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaa01004e0 0x7fdaa0195ef0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fda90006010 tx=0x7fda9000be20 comp rx=0 tx=0).stop 2024-04-27T02:04:46.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.553+0000 7fdaa5deb700 1 -- 172.21.15.29:0/336069706 shutdown_connections 2024-04-27T02:04:46.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.553+0000 7fdaa5deb700 1 --2- 172.21.15.29:0/336069706 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fda8806c680 0x7fda8806eb30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:46.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.553+0000 7fdaa5deb700 1 --2- 172.21.15.29:0/336069706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaa01004e0 0x7fdaa0195ef0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:46.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.553+0000 7fdaa5deb700 1 --2- 172.21.15.29:0/336069706 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdaa0100e80 0x7fdaa0196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:46.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.553+0000 7fdaa5deb700 1 -- 172.21.15.29:0/336069706 >> 172.21.15.29:0/336069706 conn(0x7fdaa00fbda0 msgr2=0x7fdaa00fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:46.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.553+0000 7fdaa5deb700 1 -- 172.21.15.29:0/336069706 shutdown_connections 2024-04-27T02:04:46.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:46.553+0000 7fdaa5deb700 1 -- 172.21.15.29:0/336069706 wait complete. 2024-04-27T02:04:47.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:46 smithi029 ceph-mon[26769]: pgmap v203: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:04:47.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:46 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/336069706' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-27T02:04:47.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:46 smithi052 ceph-mon[32586]: pgmap v203: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:04:47.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:46 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/336069706' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-27T02:04:47.356 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-27T02:04:47.356 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-27T02:04:47.357 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi052.front.sepia.ceph.com 2024-04-27T02:04:47.357 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-27T02:04:47.357 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-27T02:04:47.357 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-27T02:04:47.357 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-27T02:04:47.357 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-27T02:04:47.357 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-27T02:04:47.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T02:04:47.357 DEBUG:teuthology.orchestra.run.smithi052:> ip addr 2024-04-27T02:04:47.375 INFO:teuthology.orchestra.run.smithi052.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-27T02:04:47.375 INFO:teuthology.orchestra.run.smithi052.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-27T02:04:47.375 INFO:teuthology.orchestra.run.smithi052.stdout: inet 127.0.0.1/8 scope host lo 2024-04-27T02:04:47.375 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft forever preferred_lft forever 2024-04-27T02:04:47.376 INFO:teuthology.orchestra.run.smithi052.stdout: inet6 ::1/128 scope host 2024-04-27T02:04:47.376 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft forever preferred_lft forever 2024-04-27T02:04:47.376 INFO:teuthology.orchestra.run.smithi052.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-27T02:04:47.376 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:6b:fd:d2 brd ff:ff:ff:ff:ff:ff 2024-04-27T02:04:47.376 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp7s0f0 2024-04-27T02:04:47.376 INFO:teuthology.orchestra.run.smithi052.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-27T02:04:47.376 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:6b:fd:d3 brd ff:ff:ff:ff:ff:ff 2024-04-27T02:04:47.376 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp7s0f1 2024-04-27T02:04:47.376 INFO:teuthology.orchestra.run.smithi052.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-27T02:04:47.376 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:bd:15:de brd ff:ff:ff:ff:ff:ff 2024-04-27T02:04:47.376 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp1s0f0 2024-04-27T02:04:47.376 INFO:teuthology.orchestra.run.smithi052.stdout: inet 172.21.15.52/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-27T02:04:47.376 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft 42167sec preferred_lft 42167sec 2024-04-27T02:04:47.376 INFO:teuthology.orchestra.run.smithi052.stdout: inet6 fe80::ec4:7aff:febd:15de/64 scope link noprefixroute 2024-04-27T02:04:47.377 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft forever preferred_lft forever 2024-04-27T02:04:47.377 INFO:teuthology.orchestra.run.smithi052.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-27T02:04:47.377 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:bd:15:df brd ff:ff:ff:ff:ff:ff 2024-04-27T02:04:47.377 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp1s0f1 2024-04-27T02:04:47.377 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-27T02:04:47.377 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T02:04:47.377 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2024-04-27T02:04:47.377 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link add name ceph-brx type bridge 2024-04-27T02:04:47.377 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip addr flush dev ceph-brx 2024-04-27T02:04:47.377 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link set ceph-brx up 2024-04-27T02:04:47.377 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-27T02:04:47.377 DEBUG:teuthology.orchestra.run.smithi052:> ') 2024-04-27T02:04:47.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:04:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T02:04:47.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:04:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T02:04:47.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T02:04:47.571 DEBUG:teuthology.orchestra.run.smithi052:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-27T02:04:47.594 INFO:teuthology.orchestra.run.smithi052.stdout:1 2024-04-27T02:04:47.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T02:04:47.596 DEBUG:teuthology.orchestra.run.smithi052:> route 2024-04-27T02:04:47.665 INFO:teuthology.orchestra.run.smithi052.stdout:Kernel IP routing table 2024-04-27T02:04:47.665 INFO:teuthology.orchestra.run.smithi052.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-27T02:04:47.665 INFO:teuthology.orchestra.run.smithi052.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-27T02:04:47.665 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-27T02:04:47.666 INFO:teuthology.orchestra.run.smithi052.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-27T02:04:47.666 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T02:04:47.666 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2024-04-27T02:04:47.666 DEBUG:teuthology.orchestra.run.smithi052:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-27T02:04:47.666 DEBUG:teuthology.orchestra.run.smithi052:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-27T02:04:47.666 DEBUG:teuthology.orchestra.run.smithi052:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-27T02:04:47.666 DEBUG:teuthology.orchestra.run.smithi052:> ') 2024-04-27T02:04:47.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:04:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T02:04:47.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:04:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T02:04:47.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T02:04:47.945 DEBUG:teuthology.orchestra.run.smithi052:> ip netns list 2024-04-27T02:04:47.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T02:04:47.962 DEBUG:teuthology.orchestra.run.smithi052:> ip netns list-id 2024-04-27T02:04:48.017 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T02:04:48.017 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2024-04-27T02:04:48.017 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-27T02:04:48.017 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-27T02:04:48.017 DEBUG:teuthology.orchestra.run.smithi052:> ') 2024-04-27T02:04:48.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:04:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T02:04:48.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:04:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T02:04:48.134 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-27T02:04:48.134 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T02:04:48.135 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2024-04-27T02:04:48.135 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-27T02:04:48.135 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-27T02:04:48.135 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-27T02:04:48.135 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-27T02:04:48.135 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-27T02:04:48.135 DEBUG:teuthology.orchestra.run.smithi052:> ') 2024-04-27T02:04:48.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:04:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T02:04:48.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:04:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T02:04:48.255 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T02:04:48.255 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2024-04-27T02:04:48.255 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link set brx.0 up 2024-04-27T02:04:48.255 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link set dev brx.0 master ceph-brx 2024-04-27T02:04:48.255 DEBUG:teuthology.orchestra.run.smithi052:> ') 2024-04-27T02:04:48.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:04:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T02:04:48.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:04:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T02:04:48.332 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-27T02:04:48.332 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-27T02:04:48.347 INFO:teuthology.orchestra.run.smithi052.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-27T02:04:48.347 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-27T02:04:48.347 DEBUG:teuthology.orchestra.run.smithi052:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-27T02:04:48.401 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-27T02:04:48.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T02:04:48.401 DEBUG:teuthology.orchestra.run.smithi052:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-27T02:04:48.499 INFO:teuthology.orchestra.run.smithi052.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-27T02:04:48.499 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-27T02:04:48.499 INFO:teuthology.orchestra.run.smithi052.stdout:invalid new device string format 2024-04-27T02:04:48.614 INFO:teuthology.orchestra.run.smithi052.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-27T02:04:48.614 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-27T02:04:48.614 INFO:teuthology.orchestra.run.smithi052.stdout:invalid new device string format 2024-04-27T02:04:48.614 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.52:3300" 2024-04-27T02:04:48.614 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-27T02:04:48.633 INFO:teuthology.orchestra.run.smithi052.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-27T02:04:48.633 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-27T02:04:48.633 INFO:teuthology.orchestra.run.smithi052.stdout:invalid new device string format 2024-04-27T02:04:48.633 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.52:3300" 2024-04-27T02:04:48.633 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-27T02:04:48.633 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.52:6789" 2024-04-27T02:04:48.633 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.52:6789:/ 2024-04-27T02:04:48.633 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=4c587076-0439-11ef-bc93-c7b262605968" will pass to kernel 2024-04-27T02:04:48.634 INFO:teuthology.orchestra.run.smithi052.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-27T02:04:48.634 INFO:teuthology.orchestra.run.smithi052.stdout: You just mounted a file system that supports labels which does not 2024-04-27T02:04:48.634 INFO:teuthology.orchestra.run.smithi052.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-27T02:04:48.634 INFO:teuthology.orchestra.run.smithi052.stdout: applications will generate AVC messages and not be allowed access to 2024-04-27T02:04:48.634 INFO:teuthology.orchestra.run.smithi052.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-27T02:04:48.637 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-27T02:04:48.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T02:04:48.637 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-27T02:04:48.661 DEBUG:teuthology.orchestra.run.smithi052:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-27T02:04:48.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:04:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-27T02:04:48.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:04:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T02:04:48.759 INFO:teuthology.orchestra.run.smithi052.stdout:client24365 2024-04-27T02:04:48.759 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-27T02:04:48.759 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-27T02:04:48.776 DEBUG:teuthology.orchestra.run.smithi052:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-27T02:04:48.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:04:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-27T02:04:48.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:04:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T02:04:48.872 INFO:teuthology.orchestra.run.smithi052.stdout:client24365 2024-04-27T02:04:48.873 DEBUG:teuthology.orchestra.run.smithi052:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-27T02:04:48.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:04:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-27T02:04:48.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:04:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T02:04:48.930 INFO:teuthology.orchestra.run.smithi052.stdout:client24365 2024-04-27T02:04:48.930 DEBUG:teuthology.orchestra.run.smithi052:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/4c587076-0439-11ef-bc93-c7b262605968.client24365/status') 2024-04-27T02:04:48.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:04:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T02:04:48.988 INFO:teuthology.orchestra.run.smithi052.stdout:instance: client.24365 (3)192.168.0.1:0/4191043329 2024-04-27T02:04:48.989 INFO:teuthology.orchestra.run.smithi052.stdout:blocklisted: false 2024-04-27T02:04:48.989 INFO:teuthology.orchestra.run.smithi052.stderr:0+1 records in 2024-04-27T02:04:48.989 INFO:teuthology.orchestra.run.smithi052.stderr:0+1 records out 2024-04-27T02:04:48.989 INFO:teuthology.orchestra.run.smithi052.stderr:70 bytes copied, 2.4214e-05 s, 2.9 MB/s 2024-04-27T02:04:48.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-27T02:04:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T02:04:48.996 INFO:teuthology.run_tasks:Running task print... 2024-04-27T02:04:49.006 INFO:teuthology.task.print:**** done client 2024-04-27T02:04:49.006 INFO:teuthology.run_tasks:Running task parallel... 2024-04-27T02:04:49.018 INFO:teuthology.task.parallel:starting parallel... 2024-04-27T02:04:49.019 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-27T02:04:49.019 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-27T02:04:49.020 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi029.front.sepia.ceph.com 2024-04-27T02:04:49.020 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs false || true' 2024-04-27T02:04:49.020 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-27T02:04:49.020 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-04-27T02:04:49.024 INFO:tasks.workunit:Pulling workunits from ref b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T02:04:49.024 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-27T02:04:49.024 INFO:tasks.workunit:timeout=3h 2024-04-27T02:04:49.024 INFO:tasks.workunit:cleanup=True 2024-04-27T02:04:49.024 DEBUG:teuthology.orchestra.run.smithi029:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-27T02:04:49.045 INFO:teuthology.orchestra.run.smithi029.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-27T02:04:49.045 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-27T02:04:49.046 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 9ch/156d Inode: 1 Links: 2 2024-04-27T02:04:49.046 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T02:04:49.046 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-27T02:04:49.046 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-27T02:04:49.046 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-04-27 02:04:14.060665410 +0000 2024-04-27T02:04:49.046 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-04-27 02:04:44.104804407 +0000 2024-04-27T02:04:49.046 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-04-27 02:04:14.060665410 +0000 2024-04-27T02:04:49.046 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-27T02:04:49.047 DEBUG:teuthology.orchestra.run.smithi029:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-27T02:04:49.117 DEBUG:teuthology.orchestra.run.smithi052:> stat -- /home/ubuntu/cephtest/mnt.1 2024-04-27T02:04:49.133 INFO:teuthology.orchestra.run.smithi052.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-04-27T02:04:49.134 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-04-27T02:04:49.134 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 7bh/123d Inode: 1 Links: 3 2024-04-27T02:04:49.134 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T02:04:49.134 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-27T02:04:49.134 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-27T02:04:49.134 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-04-27 02:04:49.132720008 +0000 2024-04-27T02:04:49.134 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-04-27 02:04:49.132720008 +0000 2024-04-27T02:04:49.134 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-04-27 02:04:14.060665410 +0000 2024-04-27T02:04:49.134 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-04-27T02:04:49.134 DEBUG:teuthology.orchestra.run.smithi052:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-04-27T02:04:49.195 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:48 smithi052 ceph-mon[32586]: pgmap v204: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T02:04:49.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:48 smithi029 ceph-mon[26769]: pgmap v204: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T02:04:49.203 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T02:04:49.203 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T02:04:49.221 INFO:tasks.workunit.client.0.smithi029.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-27T02:04:49.259 INFO:tasks.workunit.client.1.smithi052.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-04-27T02:04:49.261 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:04:51.056 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:50 smithi029 ceph-mon[26769]: pgmap v205: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-27T02:04:51.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:50 smithi052 ceph-mon[32586]: pgmap v205: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-27T02:04:51.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.306+0000 7f1b8f2dd700 1 -- 172.21.15.29:0/3317184781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b88100240 msgr2=0x7f1b88100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:51.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.306+0000 7f1b8f2dd700 1 --2- 172.21.15.29:0/3317184781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b88100240 0x7f1b88100650 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1b84009910 tx=0x7f1b84009c20 comp rx=0 tx=0).stop 2024-04-27T02:04:51.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.306+0000 7f1b8f2dd700 1 -- 172.21.15.29:0/3317184781 shutdown_connections 2024-04-27T02:04:51.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.306+0000 7f1b8f2dd700 1 --2- 172.21.15.29:0/3317184781 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b88101440 0x7f1b88101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:51.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.306+0000 7f1b8f2dd700 1 --2- 172.21.15.29:0/3317184781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b88100240 0x7f1b88100650 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:51.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.306+0000 7f1b8f2dd700 1 -- 172.21.15.29:0/3317184781 >> 172.21.15.29:0/3317184781 conn(0x7f1b880fb7f0 msgr2=0x7f1b880fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:51.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.307+0000 7f1b8f2dd700 1 -- 172.21.15.29:0/3317184781 shutdown_connections 2024-04-27T02:04:51.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.307+0000 7f1b8f2dd700 1 -- 172.21.15.29:0/3317184781 wait complete. 2024-04-27T02:04:51.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.307+0000 7f1b8f2dd700 1 Processor -- start 2024-04-27T02:04:51.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.307+0000 7f1b8f2dd700 1 -- start start 2024-04-27T02:04:51.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.307+0000 7f1b8f2dd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b88100240 0x7f1b88195ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:51.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.307+0000 7f1b8f2dd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b88101440 0x7f1b88196010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:51.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.307+0000 7f1b8f2dd700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b88196630 con 0x7f1b88101440 2024-04-27T02:04:51.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.307+0000 7f1b8f2dd700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b88196770 con 0x7f1b88100240 2024-04-27T02:04:51.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.308+0000 7f1b8c878700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b88101440 0x7f1b88196010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:51.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.308+0000 7f1b8c878700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b88101440 0x7f1b88196010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53140/0 (socket says 172.21.15.29:53140) 2024-04-27T02:04:51.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.308+0000 7f1b8c878700 1 -- 172.21.15.29:0/1491070492 learned_addr learned my addr 172.21.15.29:0/1491070492 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:04:51.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.308+0000 7f1b8d079700 1 --2- 172.21.15.29:0/1491070492 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b88100240 0x7f1b88195ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:51.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.308+0000 7f1b8c878700 1 -- 172.21.15.29:0/1491070492 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b88100240 msgr2=0x7f1b88195ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:51.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.308+0000 7f1b8c878700 1 --2- 172.21.15.29:0/1491070492 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b88100240 0x7f1b88195ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:51.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.308+0000 7f1b8c878700 1 -- 172.21.15.29:0/1491070492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b840095f0 con 0x7f1b88101440 2024-04-27T02:04:51.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.308+0000 7f1b8c878700 1 --2- 172.21.15.29:0/1491070492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b88101440 0x7f1b88196010 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800d8d0 tx=0x7f1b7800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:51.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.308+0000 7f1b7e7fc700 1 -- 172.21.15.29:0/1491070492 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b78009940 con 0x7f1b88101440 2024-04-27T02:04:51.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.308+0000 7f1b7e7fc700 1 -- 172.21.15.29:0/1491070492 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1b78010460 con 0x7f1b88101440 2024-04-27T02:04:51.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.308+0000 7f1b8f2dd700 1 -- 172.21.15.29:0/1491070492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b8819b220 con 0x7f1b88101440 2024-04-27T02:04:51.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.308+0000 7f1b7e7fc700 1 -- 172.21.15.29:0/1491070492 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b7800f5f0 con 0x7f1b88101440 2024-04-27T02:04:51.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.308+0000 7f1b8f2dd700 1 -- 172.21.15.29:0/1491070492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8819b770 con 0x7f1b88101440 2024-04-27T02:04:51.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.309+0000 7f1b8f2dd700 1 -- 172.21.15.29:0/1491070492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b88108d90 con 0x7f1b88101440 2024-04-27T02:04:51.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.310+0000 7f1b7e7fc700 1 -- 172.21.15.29:0/1491070492 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f1b780105d0 con 0x7f1b88101440 2024-04-27T02:04:51.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.310+0000 7f1b7e7fc700 1 --2- 172.21.15.29:0/1491070492 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1b7406c680 0x7f1b7406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:51.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.310+0000 7f1b7e7fc700 1 -- 172.21.15.29:0/1491070492 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f1b7808b8e0 con 0x7f1b88101440 2024-04-27T02:04:51.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.310+0000 7f1b8d079700 1 --2- 172.21.15.29:0/1491070492 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1b7406c680 0x7f1b7406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:51.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.311+0000 7f1b8d079700 1 --2- 172.21.15.29:0/1491070492 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1b7406c680 0x7f1b7406eb30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1b84006010 tx=0x7f1b84005a90 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:51.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.315+0000 7f1b7e7fc700 1 -- 172.21.15.29:0/1491070492 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1b780599f0 con 0x7f1b88101440 2024-04-27T02:04:51.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.517+0000 7f1b8f2dd700 1 -- 172.21.15.29:0/1491070492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7f1b88108f80 con 0x7f1b88101440 2024-04-27T02:04:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.521+0000 7f1b7e7fc700 1 -- 172.21.15.29:0/1491070492 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v22)=0 v22) v1 ==== 126+0+0 (secure 0 0 0) 0x7f1b78059580 con 0x7f1b88101440 2024-04-27T02:04:51.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.524+0000 7f1b8f2dd700 1 -- 172.21.15.29:0/1491070492 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1b7406c680 msgr2=0x7f1b7406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:51.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.524+0000 7f1b8f2dd700 1 --2- 172.21.15.29:0/1491070492 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1b7406c680 0x7f1b7406eb30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1b84006010 tx=0x7f1b84005a90 comp rx=0 tx=0).stop 2024-04-27T02:04:51.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.524+0000 7f1b8f2dd700 1 -- 172.21.15.29:0/1491070492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b88101440 msgr2=0x7f1b88196010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.524+0000 7f1b8f2dd700 1 --2- 172.21.15.29:0/1491070492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b88101440 0x7f1b88196010 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800d8d0 tx=0x7f1b7800dc90 comp rx=0 tx=0).stop 2024-04-27T02:04:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.524+0000 7f1b8f2dd700 1 -- 172.21.15.29:0/1491070492 shutdown_connections 2024-04-27T02:04:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.524+0000 7f1b8f2dd700 1 --2- 172.21.15.29:0/1491070492 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1b7406c680 0x7f1b7406eb30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.524+0000 7f1b8f2dd700 1 --2- 172.21.15.29:0/1491070492 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b88100240 0x7f1b88195ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.524+0000 7f1b8f2dd700 1 --2- 172.21.15.29:0/1491070492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b88101440 0x7f1b88196010 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.524+0000 7f1b8f2dd700 1 -- 172.21.15.29:0/1491070492 >> 172.21.15.29:0/1491070492 conn(0x7f1b880fb7f0 msgr2=0x7f1b88104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.524+0000 7f1b8f2dd700 1 -- 172.21.15.29:0/1491070492 shutdown_connections 2024-04-27T02:04:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:51.524+0000 7f1b8f2dd700 1 -- 172.21.15.29:0/1491070492 wait complete. 2024-04-27T02:04:52.316 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-27T02:04:52.316 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi029.front.sepia.ceph.com 2024-04-27T02:04:52.316 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-27T02:04:52.589 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:04:52.645 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:52 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/1491070492' entity='client.admin' 2024-04-27T02:04:52.645 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:52 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:04:52.645 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:52 smithi029 ceph-mon[26769]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 853 B/s wr, 3 op/s 2024-04-27T02:04:52.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:52 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/1491070492' entity='client.admin' 2024-04-27T02:04:52.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:52 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:04:52.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:52 smithi052 ceph-mon[32586]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 853 B/s wr, 3 op/s 2024-04-27T02:04:53.806 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:53 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:04:53.807 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:53 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:04:53.807 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:53 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:53.807 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:53 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:04:54.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:53 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:04:54.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:53 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:04:54.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:53 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:04:54.004 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:53 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:04:54.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.363+0000 7fb24b6ca700 1 -- 172.21.15.29:0/3325246115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb244100760 msgr2=0x7fb244100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:54.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.363+0000 7fb24b6ca700 1 --2- 172.21.15.29:0/3325246115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb244100760 0x7fb244100b70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb240009910 tx=0x7fb240009c20 comp rx=0 tx=0).stop 2024-04-27T02:04:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.363+0000 7fb24b6ca700 1 -- 172.21.15.29:0/3325246115 shutdown_connections 2024-04-27T02:04:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.363+0000 7fb24b6ca700 1 --2- 172.21.15.29:0/3325246115 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2440ff410 0x7fb2440ff880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.363+0000 7fb24b6ca700 1 --2- 172.21.15.29:0/3325246115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb244100760 0x7fb244100b70 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.363+0000 7fb24b6ca700 1 -- 172.21.15.29:0/3325246115 >> 172.21.15.29:0/3325246115 conn(0x7fb2440f9e70 msgr2=0x7fb2440fc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.363+0000 7fb24b6ca700 1 -- 172.21.15.29:0/3325246115 shutdown_connections 2024-04-27T02:04:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.363+0000 7fb24b6ca700 1 -- 172.21.15.29:0/3325246115 wait complete. 2024-04-27T02:04:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.363+0000 7fb24b6ca700 1 Processor -- start 2024-04-27T02:04:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.363+0000 7fb24b6ca700 1 -- start start 2024-04-27T02:04:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.363+0000 7fb24b6ca700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2440ff410 0x7fb244195b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.364+0000 7fb24b6ca700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb244100760 0x7fb244196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.364+0000 7fb24b6ca700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2441966b0 con 0x7fb244100760 2024-04-27T02:04:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.364+0000 7fb24b6ca700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2441967f0 con 0x7fb2440ff410 2024-04-27T02:04:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.364+0000 7fb248c65700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb244100760 0x7fb244196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.364+0000 7fb248c65700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb244100760 0x7fb244196090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53152/0 (socket says 172.21.15.29:53152) 2024-04-27T02:04:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.364+0000 7fb248c65700 1 -- 172.21.15.29:0/886029836 learned_addr learned my addr 172.21.15.29:0/886029836 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:04:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.364+0000 7fb249466700 1 --2- 172.21.15.29:0/886029836 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2440ff410 0x7fb244195b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.364+0000 7fb248c65700 1 -- 172.21.15.29:0/886029836 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2440ff410 msgr2=0x7fb244195b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.364+0000 7fb248c65700 1 --2- 172.21.15.29:0/886029836 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2440ff410 0x7fb244195b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.364+0000 7fb248c65700 1 -- 172.21.15.29:0/886029836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2400095f0 con 0x7fb244100760 2024-04-27T02:04:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.364+0000 7fb248c65700 1 --2- 172.21.15.29:0/886029836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb244100760 0x7fb244196090 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb23400c8f0 tx=0x7fb23400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.364+0000 7fb23a7fc700 1 -- 172.21.15.29:0/886029836 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2340078e0 con 0x7fb244100760 2024-04-27T02:04:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.364+0000 7fb24b6ca700 1 -- 172.21.15.29:0/886029836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb24419b2a0 con 0x7fb244100760 2024-04-27T02:04:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.364+0000 7fb23a7fc700 1 -- 172.21.15.29:0/886029836 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb23400f450 con 0x7fb244100760 2024-04-27T02:04:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.364+0000 7fb23a7fc700 1 -- 172.21.15.29:0/886029836 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb234018610 con 0x7fb244100760 2024-04-27T02:04:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.364+0000 7fb24b6ca700 1 -- 172.21.15.29:0/886029836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb24419b7f0 con 0x7fb244100760 2024-04-27T02:04:54.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.365+0000 7fb24b6ca700 1 -- 172.21.15.29:0/886029836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb244107cd0 con 0x7fb244100760 2024-04-27T02:04:54.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.366+0000 7fb23a7fc700 1 -- 172.21.15.29:0/886029836 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fb23400f5c0 con 0x7fb244100760 2024-04-27T02:04:54.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.366+0000 7fb23a7fc700 1 --2- 172.21.15.29:0/886029836 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb23006c680 0x7fb23006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:54.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.366+0000 7fb23a7fc700 1 -- 172.21.15.29:0/886029836 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fb23408b7b0 con 0x7fb244100760 2024-04-27T02:04:54.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.366+0000 7fb249466700 1 --2- 172.21.15.29:0/886029836 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb23006c680 0x7fb23006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:54.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.367+0000 7fb249466700 1 --2- 172.21.15.29:0/886029836 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb23006c680 0x7fb23006eb30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb240005090 tx=0x7fb24001a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:54.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.371+0000 7fb23a7fc700 1 -- 172.21.15.29:0/886029836 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb2340598c0 con 0x7fb244100760 2024-04-27T02:04:54.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.570+0000 7fb24b6ca700 1 -- 172.21.15.29:0/886029836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7fb244107e90 con 0x7fb244100760 2024-04-27T02:04:54.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.570+0000 7fb23a7fc700 1 -- 172.21.15.29:0/886029836 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v22)=0 v22) v1 ==== 155+0+0 (secure 0 0 0) 0x7fb234059450 con 0x7fb244100760 2024-04-27T02:04:54.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.572+0000 7fb24b6ca700 1 -- 172.21.15.29:0/886029836 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb23006c680 msgr2=0x7fb23006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:54.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.572+0000 7fb24b6ca700 1 --2- 172.21.15.29:0/886029836 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb23006c680 0x7fb23006eb30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb240005090 tx=0x7fb24001a040 comp rx=0 tx=0).stop 2024-04-27T02:04:54.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.572+0000 7fb24b6ca700 1 -- 172.21.15.29:0/886029836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb244100760 msgr2=0x7fb244196090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:54.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.572+0000 7fb24b6ca700 1 --2- 172.21.15.29:0/886029836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb244100760 0x7fb244196090 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb23400c8f0 tx=0x7fb23400cc00 comp rx=0 tx=0).stop 2024-04-27T02:04:54.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.572+0000 7fb24b6ca700 1 -- 172.21.15.29:0/886029836 shutdown_connections 2024-04-27T02:04:54.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.572+0000 7fb24b6ca700 1 --2- 172.21.15.29:0/886029836 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb23006c680 0x7fb23006eb30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:54.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.572+0000 7fb24b6ca700 1 --2- 172.21.15.29:0/886029836 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2440ff410 0x7fb244195b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:54.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.572+0000 7fb24b6ca700 1 --2- 172.21.15.29:0/886029836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb244100760 0x7fb244196090 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:54.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.572+0000 7fb24b6ca700 1 -- 172.21.15.29:0/886029836 >> 172.21.15.29:0/886029836 conn(0x7fb2440f9e70 msgr2=0x7fb2440fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:54.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.572+0000 7fb24b6ca700 1 -- 172.21.15.29:0/886029836 shutdown_connections 2024-04-27T02:04:54.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:54.572+0000 7fb24b6ca700 1 -- 172.21.15.29:0/886029836 wait complete. 2024-04-27T02:04:54.938 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:54 smithi029 ceph-mon[26769]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 938 B/s wr, 2 op/s 2024-04-27T02:04:55.003 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:54 smithi052 ceph-mon[32586]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 938 B/s wr, 2 op/s 2024-04-27T02:04:55.352 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-27T02:04:55.573 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:04:57.139 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:56 smithi029 ceph-mon[26769]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 938 B/s wr, 2 op/s 2024-04-27T02:04:57.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:56 smithi052 ceph-mon[32586]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 938 B/s wr, 2 op/s 2024-04-27T02:04:57.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.430+0000 7f4cf48d7700 1 -- 172.21.15.29:0/2705423833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf00ff130 msgr2=0x7f4cf00ff540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:57.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.430+0000 7f4cf48d7700 1 --2- 172.21.15.29:0/2705423833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf00ff130 0x7f4cf00ff540 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0009910 tx=0x7f4ce0009c20 comp rx=0 tx=0).stop 2024-04-27T02:04:57.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.430+0000 7f4cf48d7700 1 -- 172.21.15.29:0/2705423833 shutdown_connections 2024-04-27T02:04:57.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.430+0000 7f4cf48d7700 1 --2- 172.21.15.29:0/2705423833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cf01003d0 0x7f4cf0100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:57.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.430+0000 7f4cf48d7700 1 --2- 172.21.15.29:0/2705423833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf00ff130 0x7f4cf00ff540 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:57.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.430+0000 7f4cf48d7700 1 -- 172.21.15.29:0/2705423833 >> 172.21.15.29:0/2705423833 conn(0x7f4cf00fa780 msgr2=0x7f4cf00fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:57.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.430+0000 7f4cf48d7700 1 -- 172.21.15.29:0/2705423833 shutdown_connections 2024-04-27T02:04:57.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.430+0000 7f4cf48d7700 1 -- 172.21.15.29:0/2705423833 wait complete. 2024-04-27T02:04:57.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.431+0000 7f4cf48d7700 1 Processor -- start 2024-04-27T02:04:57.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.431+0000 7f4cf48d7700 1 -- start start 2024-04-27T02:04:57.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.431+0000 7f4cf48d7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cf00ff130 0x7f4cf0195aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:57.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.431+0000 7f4cf48d7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf01003d0 0x7f4cf0195fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:57.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.431+0000 7f4cf48d7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf0196600 con 0x7f4cf01003d0 2024-04-27T02:04:57.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.431+0000 7f4cf48d7700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf0196740 con 0x7f4cf00ff130 2024-04-27T02:04:57.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.431+0000 7f4cedd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf01003d0 0x7f4cf0195fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:57.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.431+0000 7f4cedd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf01003d0 0x7f4cf0195fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42934/0 (socket says 172.21.15.29:42934) 2024-04-27T02:04:57.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.431+0000 7f4cedd9b700 1 -- 172.21.15.29:0/3599208886 learned_addr learned my addr 172.21.15.29:0/3599208886 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:04:57.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.432+0000 7f4cee59c700 1 --2- 172.21.15.29:0/3599208886 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cf00ff130 0x7f4cf0195aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:57.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.432+0000 7f4cedd9b700 1 -- 172.21.15.29:0/3599208886 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cf00ff130 msgr2=0x7f4cf0195aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:57.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.432+0000 7f4cedd9b700 1 --2- 172.21.15.29:0/3599208886 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cf00ff130 0x7f4cf0195aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:57.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.432+0000 7f4cedd9b700 1 -- 172.21.15.29:0/3599208886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce00095f0 con 0x7f4cf01003d0 2024-04-27T02:04:57.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.432+0000 7f4cedd9b700 1 --2- 172.21.15.29:0/3599208886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf01003d0 0x7f4cf0195fe0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4ce400d8d0 tx=0x7f4ce400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:57.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.432+0000 7f4cdf7fe700 1 -- 172.21.15.29:0/3599208886 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ce40098a0 con 0x7f4cf01003d0 2024-04-27T02:04:57.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.432+0000 7f4cf48d7700 1 -- 172.21.15.29:0/3599208886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf019b1f0 con 0x7f4cf01003d0 2024-04-27T02:04:57.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.432+0000 7f4cdf7fe700 1 -- 172.21.15.29:0/3599208886 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4ce4010460 con 0x7f4cf01003d0 2024-04-27T02:04:57.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.432+0000 7f4cdf7fe700 1 -- 172.21.15.29:0/3599208886 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ce400f5f0 con 0x7f4cf01003d0 2024-04-27T02:04:57.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.432+0000 7f4cf48d7700 1 -- 172.21.15.29:0/3599208886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf019b740 con 0x7f4cf01003d0 2024-04-27T02:04:57.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.433+0000 7f4cf48d7700 1 -- 172.21.15.29:0/3599208886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf0107c10 con 0x7f4cf01003d0 2024-04-27T02:04:57.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.434+0000 7f4cdf7fe700 1 -- 172.21.15.29:0/3599208886 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f4ce4009a00 con 0x7f4cf01003d0 2024-04-27T02:04:57.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.434+0000 7f4cdf7fe700 1 --2- 172.21.15.29:0/3599208886 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f4cd806c680 0x7f4cd806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:04:57.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.434+0000 7f4cdf7fe700 1 -- 172.21.15.29:0/3599208886 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f4ce408b7e0 con 0x7f4cf01003d0 2024-04-27T02:04:57.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.434+0000 7f4cee59c700 1 --2- 172.21.15.29:0/3599208886 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f4cd806c680 0x7f4cd806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:04:57.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.435+0000 7f4cee59c700 1 --2- 172.21.15.29:0/3599208886 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f4cd806c680 0x7f4cd806eb30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4ce000b5c0 tx=0x7f4ce0005ee0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:04:57.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.439+0000 7f4cdf7fe700 1 -- 172.21.15.29:0/3599208886 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4ce40598f0 con 0x7f4cf01003d0 2024-04-27T02:04:57.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.640+0000 7f4cf48d7700 1 -- 172.21.15.29:0/3599208886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f4cf0107dd0 con 0x7f4cf01003d0 2024-04-27T02:04:57.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.640+0000 7f4cdf7fe700 1 -- 172.21.15.29:0/3599208886 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v22)=0 v22) v1 ==== 163+0+0 (secure 0 0 0) 0x7f4ce4059480 con 0x7f4cf01003d0 2024-04-27T02:04:57.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.642+0000 7f4cf48d7700 1 -- 172.21.15.29:0/3599208886 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f4cd806c680 msgr2=0x7f4cd806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:57.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.642+0000 7f4cf48d7700 1 --2- 172.21.15.29:0/3599208886 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f4cd806c680 0x7f4cd806eb30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4ce000b5c0 tx=0x7f4ce0005ee0 comp rx=0 tx=0).stop 2024-04-27T02:04:57.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.642+0000 7f4cf48d7700 1 -- 172.21.15.29:0/3599208886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf01003d0 msgr2=0x7f4cf0195fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:04:57.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.642+0000 7f4cf48d7700 1 --2- 172.21.15.29:0/3599208886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf01003d0 0x7f4cf0195fe0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4ce400d8d0 tx=0x7f4ce400dbe0 comp rx=0 tx=0).stop 2024-04-27T02:04:57.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.642+0000 7f4cf48d7700 1 -- 172.21.15.29:0/3599208886 shutdown_connections 2024-04-27T02:04:57.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.642+0000 7f4cf48d7700 1 --2- 172.21.15.29:0/3599208886 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f4cd806c680 0x7f4cd806eb30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:57.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.642+0000 7f4cf48d7700 1 --2- 172.21.15.29:0/3599208886 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cf00ff130 0x7f4cf0195aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:57.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.642+0000 7f4cf48d7700 1 --2- 172.21.15.29:0/3599208886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf01003d0 0x7f4cf0195fe0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:04:57.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.642+0000 7f4cf48d7700 1 -- 172.21.15.29:0/3599208886 >> 172.21.15.29:0/3599208886 conn(0x7f4cf00fa780 msgr2=0x7f4cf0068540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:04:57.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.642+0000 7f4cf48d7700 1 -- 172.21.15.29:0/3599208886 shutdown_connections 2024-04-27T02:04:57.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:04:57.642+0000 7f4cf48d7700 1 -- 172.21.15.29:0/3599208886 wait complete. 2024-04-27T02:04:58.458 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-27T02:04:58.681 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:04:59.122 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:04:58 smithi029 ceph-mon[26769]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 938 B/s wr, 3 op/s 2024-04-27T02:04:59.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:04:58 smithi052 ceph-mon[32586]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 938 B/s wr, 3 op/s 2024-04-27T02:05:00.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.435+0000 7f5b3e679700 1 -- 172.21.15.29:0/2150788250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b380ff710 msgr2=0x7f5b380ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:00.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.435+0000 7f5b3e679700 1 --2- 172.21.15.29:0/2150788250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b380ff710 0x7f5b380ffb20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5b28009910 tx=0x7f5b28009c20 comp rx=0 tx=0).stop 2024-04-27T02:05:00.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.436+0000 7f5b3e679700 1 -- 172.21.15.29:0/2150788250 shutdown_connections 2024-04-27T02:05:00.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.436+0000 7f5b3e679700 1 --2- 172.21.15.29:0/2150788250 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b38100060 0x7f5b381004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:00.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.436+0000 7f5b3e679700 1 --2- 172.21.15.29:0/2150788250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b380ff710 0x7f5b380ffb20 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:00.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.436+0000 7f5b3e679700 1 -- 172.21.15.29:0/2150788250 >> 172.21.15.29:0/2150788250 conn(0x7f5b380f9f10 msgr2=0x7f5b380fc340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:00.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.436+0000 7f5b3e679700 1 -- 172.21.15.29:0/2150788250 shutdown_connections 2024-04-27T02:05:00.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.436+0000 7f5b3e679700 1 -- 172.21.15.29:0/2150788250 wait complete. 2024-04-27T02:05:00.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.436+0000 7f5b3e679700 1 Processor -- start 2024-04-27T02:05:00.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.436+0000 7f5b3e679700 1 -- start start 2024-04-27T02:05:00.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.436+0000 7f5b3e679700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b380ff710 0x7f5b38103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:00.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.436+0000 7f5b3e679700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b38100060 0x7f5b381018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:00.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.437+0000 7f5b3e679700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b38101de0 con 0x7f5b38100060 2024-04-27T02:05:00.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.437+0000 7f5b3e679700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b38101f20 con 0x7f5b380ff710 2024-04-27T02:05:00.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.437+0000 7f5b2ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b38100060 0x7f5b381018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:00.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.437+0000 7f5b2ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b38100060 0x7f5b381018a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42950/0 (socket says 172.21.15.29:42950) 2024-04-27T02:05:00.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.437+0000 7f5b2ffff700 1 -- 172.21.15.29:0/746735275 learned_addr learned my addr 172.21.15.29:0/746735275 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:05:00.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.437+0000 7f5b37fff700 1 --2- 172.21.15.29:0/746735275 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b380ff710 0x7f5b38103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:00.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.437+0000 7f5b2ffff700 1 -- 172.21.15.29:0/746735275 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b380ff710 msgr2=0x7f5b38103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:00.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.437+0000 7f5b2ffff700 1 --2- 172.21.15.29:0/746735275 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b380ff710 0x7f5b38103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:00.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.437+0000 7f5b2ffff700 1 -- 172.21.15.29:0/746735275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b280095f0 con 0x7f5b38100060 2024-04-27T02:05:00.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.437+0000 7f5b2ffff700 1 --2- 172.21.15.29:0/746735275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b38100060 0x7f5b381018a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5b2000c930 tx=0x7f5b2000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:00.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.437+0000 7f5b35ffb700 1 -- 172.21.15.29:0/746735275 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b20007ad0 con 0x7f5b38100060 2024-04-27T02:05:00.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.437+0000 7f5b35ffb700 1 -- 172.21.15.29:0/746735275 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5b2000ce80 con 0x7f5b38100060 2024-04-27T02:05:00.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.437+0000 7f5b3e679700 1 -- 172.21.15.29:0/746735275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b38102200 con 0x7f5b38100060 2024-04-27T02:05:00.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.437+0000 7f5b35ffb700 1 -- 172.21.15.29:0/746735275 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b20018630 con 0x7f5b38100060 2024-04-27T02:05:00.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.437+0000 7f5b3e679700 1 -- 172.21.15.29:0/746735275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b3819db10 con 0x7f5b38100060 2024-04-27T02:05:00.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.438+0000 7f5b3e679700 1 -- 172.21.15.29:0/746735275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b38107c10 con 0x7f5b38100060 2024-04-27T02:05:00.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.439+0000 7f5b35ffb700 1 -- 172.21.15.29:0/746735275 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f5b20007c30 con 0x7f5b38100060 2024-04-27T02:05:00.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.439+0000 7f5b35ffb700 1 --2- 172.21.15.29:0/746735275 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f5b2406c680 0x7f5b2406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:00.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.439+0000 7f5b35ffb700 1 -- 172.21.15.29:0/746735275 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f5b2008b8a0 con 0x7f5b38100060 2024-04-27T02:05:00.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.439+0000 7f5b37fff700 1 --2- 172.21.15.29:0/746735275 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f5b2406c680 0x7f5b2406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:00.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.440+0000 7f5b37fff700 1 --2- 172.21.15.29:0/746735275 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f5b2406c680 0x7f5b2406eb30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5b2800b5c0 tx=0x7f5b280052a0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:00.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.444+0000 7f5b35ffb700 1 -- 172.21.15.29:0/746735275 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f5b20059990 con 0x7f5b38100060 2024-04-27T02:05:00.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.642+0000 7f5b3e679700 1 -- 172.21.15.29:0/746735275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f5b38107dd0 con 0x7f5b38100060 2024-04-27T02:05:00.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.642+0000 7f5b35ffb700 1 -- 172.21.15.29:0/746735275 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v22)=0 v22) v1 ==== 135+0+0 (secure 0 0 0) 0x7f5b20059520 con 0x7f5b38100060 2024-04-27T02:05:00.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.644+0000 7f5b3e679700 1 -- 172.21.15.29:0/746735275 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f5b2406c680 msgr2=0x7f5b2406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:00.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.644+0000 7f5b3e679700 1 --2- 172.21.15.29:0/746735275 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f5b2406c680 0x7f5b2406eb30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5b2800b5c0 tx=0x7f5b280052a0 comp rx=0 tx=0).stop 2024-04-27T02:05:00.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.644+0000 7f5b3e679700 1 -- 172.21.15.29:0/746735275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b38100060 msgr2=0x7f5b381018a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:00.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.644+0000 7f5b3e679700 1 --2- 172.21.15.29:0/746735275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b38100060 0x7f5b381018a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5b2000c930 tx=0x7f5b2000cc40 comp rx=0 tx=0).stop 2024-04-27T02:05:00.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.644+0000 7f5b3e679700 1 -- 172.21.15.29:0/746735275 shutdown_connections 2024-04-27T02:05:00.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.644+0000 7f5b3e679700 1 --2- 172.21.15.29:0/746735275 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f5b2406c680 0x7f5b2406eb30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:00.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.644+0000 7f5b3e679700 1 --2- 172.21.15.29:0/746735275 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b380ff710 0x7f5b38103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:00.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.644+0000 7f5b3e679700 1 --2- 172.21.15.29:0/746735275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b38100060 0x7f5b381018a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:00.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.644+0000 7f5b3e679700 1 -- 172.21.15.29:0/746735275 >> 172.21.15.29:0/746735275 conn(0x7f5b380f9f10 msgr2=0x7f5b380fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:00.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.645+0000 7f5b3e679700 1 -- 172.21.15.29:0/746735275 shutdown_connections 2024-04-27T02:05:00.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:00.645+0000 7f5b3e679700 1 -- 172.21.15.29:0/746735275 wait complete. 2024-04-27T02:05:01.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:00 smithi029 ceph-mon[26769]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-27T02:05:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:00 smithi052 ceph-mon[32586]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-27T02:05:01.378 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-27T02:05:01.600 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:05:02.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:02 smithi029 ceph-mon[26769]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 682 B/s wr, 3 op/s 2024-04-27T02:05:03.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.249+0000 7f1bb1fae700 1 -- 172.21.15.29:0/2972445848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bac0ffae0 msgr2=0x7f1bac0fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:03.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.249+0000 7f1bb1fae700 1 --2- 172.21.15.29:0/2972445848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bac0ffae0 0x7f1bac0fff30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0009910 tx=0x7f1ba0009c20 comp rx=0 tx=0).stop 2024-04-27T02:05:03.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.249+0000 7f1bb1fae700 1 -- 172.21.15.29:0/2972445848 shutdown_connections 2024-04-27T02:05:03.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.249+0000 7f1bb1fae700 1 --2- 172.21.15.29:0/2972445848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bac0ffae0 0x7f1bac0fff30 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:03.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.249+0000 7f1bb1fae700 1 --2- 172.21.15.29:0/2972445848 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1bac0fe8e0 0x7f1bac0fecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:03.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.249+0000 7f1bb1fae700 1 -- 172.21.15.29:0/2972445848 >> 172.21.15.29:0/2972445848 conn(0x7f1bac0f9e70 msgr2=0x7f1bac0fc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:03.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.249+0000 7f1bb1fae700 1 -- 172.21.15.29:0/2972445848 shutdown_connections 2024-04-27T02:05:03.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.249+0000 7f1bb1fae700 1 -- 172.21.15.29:0/2972445848 wait complete. 2024-04-27T02:05:03.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.249+0000 7f1bb1fae700 1 Processor -- start 2024-04-27T02:05:03.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.250+0000 7f1bb1fae700 1 -- start start 2024-04-27T02:05:03.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.250+0000 7f1bb1fae700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bac0fe8e0 0x7f1bac195c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:03.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.250+0000 7f1bb1fae700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1bac0ffae0 0x7f1bac196150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:03.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.250+0000 7f1bb1fae700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac196770 con 0x7f1bac0fe8e0 2024-04-27T02:05:03.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.250+0000 7f1bb1fae700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac19b180 con 0x7f1bac0ffae0 2024-04-27T02:05:03.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.250+0000 7f1bab7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bac0fe8e0 0x7f1bac195c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:03.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.250+0000 7f1bab7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bac0fe8e0 0x7f1bac195c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42960/0 (socket says 172.21.15.29:42960) 2024-04-27T02:05:03.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.250+0000 7f1bab7fe700 1 -- 172.21.15.29:0/373915817 learned_addr learned my addr 172.21.15.29:0/373915817 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:05:03.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.250+0000 7f1baaffd700 1 --2- 172.21.15.29:0/373915817 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1bac0ffae0 0x7f1bac196150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:03.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.250+0000 7f1bab7fe700 1 -- 172.21.15.29:0/373915817 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1bac0ffae0 msgr2=0x7f1bac196150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:03.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.250+0000 7f1bab7fe700 1 --2- 172.21.15.29:0/373915817 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1bac0ffae0 0x7f1bac196150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:03.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.250+0000 7f1bab7fe700 1 -- 172.21.15.29:0/373915817 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba00095f0 con 0x7f1bac0fe8e0 2024-04-27T02:05:03.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.250+0000 7f1bab7fe700 1 --2- 172.21.15.29:0/373915817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bac0fe8e0 0x7f1bac195c10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c00b480 tx=0x7f1b9c00b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:03.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.251+0000 7f1ba8ff9700 1 -- 172.21.15.29:0/373915817 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b9c011800 con 0x7f1bac0fe8e0 2024-04-27T02:05:03.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.251+0000 7f1bb1fae700 1 -- 172.21.15.29:0/373915817 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bac19b380 con 0x7f1bac0fe8e0 2024-04-27T02:05:03.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.251+0000 7f1ba8ff9700 1 -- 172.21.15.29:0/373915817 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1b9c011e10 con 0x7f1bac0fe8e0 2024-04-27T02:05:03.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.251+0000 7f1ba8ff9700 1 -- 172.21.15.29:0/373915817 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b9c010ea0 con 0x7f1bac0fe8e0 2024-04-27T02:05:03.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.251+0000 7f1bb1fae700 1 -- 172.21.15.29:0/373915817 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bac19b8a0 con 0x7f1bac0fe8e0 2024-04-27T02:05:03.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.252+0000 7f1bb1fae700 1 -- 172.21.15.29:0/373915817 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bac107d50 con 0x7f1bac0fe8e0 2024-04-27T02:05:03.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.252+0000 7f1ba8ff9700 1 -- 172.21.15.29:0/373915817 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f1b9c010380 con 0x7f1bac0fe8e0 2024-04-27T02:05:03.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.253+0000 7f1ba8ff9700 1 --2- 172.21.15.29:0/373915817 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1b9406c680 0x7f1b9406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:03.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.253+0000 7f1ba8ff9700 1 -- 172.21.15.29:0/373915817 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f1b9c08b350 con 0x7f1bac0fe8e0 2024-04-27T02:05:03.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.253+0000 7f1baaffd700 1 --2- 172.21.15.29:0/373915817 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1b9406c680 0x7f1b9406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:03.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.253+0000 7f1baaffd700 1 --2- 172.21.15.29:0/373915817 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1b9406c680 0x7f1b9406eb30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0005090 tx=0x7f1ba001a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:03.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.257+0000 7f1ba8ff9700 1 -- 172.21.15.29:0/373915817 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1b9c055a50 con 0x7f1bac0fe8e0 2024-04-27T02:05:03.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:02 smithi052 ceph-mon[32586]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 682 B/s wr, 3 op/s 2024-04-27T02:05:03.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.474+0000 7f1bb1fae700 1 -- 172.21.15.29:0/373915817 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a", "target": ["mon-mgr", ""]}) v1 -- 0x7f1bac105630 con 0x7f1b9406c680 2024-04-27T02:05:03.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.480+0000 7f1ba8ff9700 1 -- 172.21.15.29:0/373915817 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f1bac105630 con 0x7f1b9406c680 2024-04-27T02:05:03.460 INFO:teuthology.orchestra.run.smithi029.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T02:05:03.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.482+0000 7f1bb1fae700 1 -- 172.21.15.29:0/373915817 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1b9406c680 msgr2=0x7f1b9406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:03.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.482+0000 7f1bb1fae700 1 --2- 172.21.15.29:0/373915817 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1b9406c680 0x7f1b9406eb30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0005090 tx=0x7f1ba001a040 comp rx=0 tx=0).stop 2024-04-27T02:05:03.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.482+0000 7f1bb1fae700 1 -- 172.21.15.29:0/373915817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bac0fe8e0 msgr2=0x7f1bac195c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:03.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.482+0000 7f1bb1fae700 1 --2- 172.21.15.29:0/373915817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bac0fe8e0 0x7f1bac195c10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c00b480 tx=0x7f1b9c00b790 comp rx=0 tx=0).stop 2024-04-27T02:05:03.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.482+0000 7f1bb1fae700 1 -- 172.21.15.29:0/373915817 shutdown_connections 2024-04-27T02:05:03.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.482+0000 7f1bb1fae700 1 --2- 172.21.15.29:0/373915817 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1b9406c680 0x7f1b9406eb30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:03.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.482+0000 7f1bb1fae700 1 --2- 172.21.15.29:0/373915817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bac0fe8e0 0x7f1bac195c10 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:03.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.482+0000 7f1bb1fae700 1 --2- 172.21.15.29:0/373915817 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1bac0ffae0 0x7f1bac196150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:03.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.482+0000 7f1bb1fae700 1 -- 172.21.15.29:0/373915817 >> 172.21.15.29:0/373915817 conn(0x7f1bac0f9e70 msgr2=0x7f1bac104f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:03.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.482+0000 7f1bb1fae700 1 -- 172.21.15.29:0/373915817 shutdown_connections 2024-04-27T02:05:03.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:03.482+0000 7f1bb1fae700 1 -- 172.21.15.29:0/373915817 wait complete. 2024-04-27T02:05:04.251 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-27T02:05:04.252 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi029.front.sepia.ceph.com 2024-04-27T02:05:04.252 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-27T02:05:04.484 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:05:04.607 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:04 smithi029 ceph-mon[26769]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:05:04.608 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:04 smithi029 ceph-mon[26769]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T02:05:04.608 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:04 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:05:04.608 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:04 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:05:04.608 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:04 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:05:04.608 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:04 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:05:04.609 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:04 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:05:04.609 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:04 smithi029 ceph-mon[26769]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T02:05:04.609 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:04 smithi029 ceph-mon[26769]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-04-27T02:05:04.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:04 smithi052 ceph-mon[32586]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:05:04.753 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:04 smithi052 ceph-mon[32586]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T02:05:04.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:04 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:05:04.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:04 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:05:04.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:04 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:05:04.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:04 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:05:04.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:04 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:05:04.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:04 smithi052 ceph-mon[32586]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T02:05:04.754 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:04 smithi052 ceph-mon[32586]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-04-27T02:05:06.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.387+0000 7efd0e3aa700 1 -- 172.21.15.29:0/741988357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd08101440 msgr2=0x7efd08101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:06.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.387+0000 7efd0e3aa700 1 --2- 172.21.15.29:0/741988357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd08101440 0x7efd08101890 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009910 tx=0x7efcfc009c20 comp rx=0 tx=0).stop 2024-04-27T02:05:06.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.387+0000 7efd0e3aa700 1 -- 172.21.15.29:0/741988357 shutdown_connections 2024-04-27T02:05:06.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.387+0000 7efd0e3aa700 1 --2- 172.21.15.29:0/741988357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd08101440 0x7efd08101890 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:06.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.387+0000 7efd0e3aa700 1 --2- 172.21.15.29:0/741988357 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efd08100240 0x7efd08100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:06.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.387+0000 7efd0e3aa700 1 -- 172.21.15.29:0/741988357 >> 172.21.15.29:0/741988357 conn(0x7efd080fb7f0 msgr2=0x7efd080fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:06.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.387+0000 7efd0e3aa700 1 -- 172.21.15.29:0/741988357 shutdown_connections 2024-04-27T02:05:06.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.387+0000 7efd0e3aa700 1 -- 172.21.15.29:0/741988357 wait complete. 2024-04-27T02:05:06.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.388+0000 7efd0e3aa700 1 Processor -- start 2024-04-27T02:05:06.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.388+0000 7efd0e3aa700 1 -- start start 2024-04-27T02:05:06.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.389+0000 7efd0e3aa700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efd08100240 0x7efd08195ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:06.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.389+0000 7efd0e3aa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd08101440 0x7efd08195ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:06.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.389+0000 7efd0e3aa700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd08196610 con 0x7efd08101440 2024-04-27T02:05:06.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.389+0000 7efd0e3aa700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd08196750 con 0x7efd08100240 2024-04-27T02:05:06.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.389+0000 7efd077fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd08101440 0x7efd08195ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:06.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.389+0000 7efd077fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd08101440 0x7efd08195ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42978/0 (socket says 172.21.15.29:42978) 2024-04-27T02:05:06.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.389+0000 7efd077fe700 1 -- 172.21.15.29:0/96073474 learned_addr learned my addr 172.21.15.29:0/96073474 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:05:06.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.389+0000 7efd077fe700 1 -- 172.21.15.29:0/96073474 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efd08100240 msgr2=0x7efd08195ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:06.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.389+0000 7efd07fff700 1 --2- 172.21.15.29:0/96073474 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efd08100240 0x7efd08195ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:06.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.389+0000 7efd077fe700 1 --2- 172.21.15.29:0/96073474 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efd08100240 0x7efd08195ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:06.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.389+0000 7efd077fe700 1 -- 172.21.15.29:0/96073474 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcfc0095f0 con 0x7efd08101440 2024-04-27T02:05:06.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.390+0000 7efd077fe700 1 --2- 172.21.15.29:0/96073474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd08101440 0x7efd08195ff0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00bd70 tx=0x7efcfc00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:06.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.390+0000 7efd057fa700 1 -- 172.21.15.29:0/96073474 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efcfc01d070 con 0x7efd08101440 2024-04-27T02:05:06.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.390+0000 7efd057fa700 1 -- 172.21.15.29:0/96073474 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7efcfc022470 con 0x7efd08101440 2024-04-27T02:05:06.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.390+0000 7efd0e3aa700 1 -- 172.21.15.29:0/96073474 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd0819b1a0 con 0x7efd08101440 2024-04-27T02:05:06.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.390+0000 7efd057fa700 1 -- 172.21.15.29:0/96073474 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efcfc00f670 con 0x7efd08101440 2024-04-27T02:05:06.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.390+0000 7efd0e3aa700 1 -- 172.21.15.29:0/96073474 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd0819b690 con 0x7efd08101440 2024-04-27T02:05:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.392+0000 7efd0e3aa700 1 -- 172.21.15.29:0/96073474 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd08108d90 con 0x7efd08101440 2024-04-27T02:05:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.392+0000 7efd057fa700 1 -- 172.21.15.29:0/96073474 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7efcfc022a50 con 0x7efd08101440 2024-04-27T02:05:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.393+0000 7efd057fa700 1 --2- 172.21.15.29:0/96073474 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7efcf006c680 0x7efcf006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.393+0000 7efd057fa700 1 -- 172.21.15.29:0/96073474 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7efcfc08cd40 con 0x7efd08101440 2024-04-27T02:05:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.393+0000 7efd07fff700 1 --2- 172.21.15.29:0/96073474 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7efcf006c680 0x7efcf006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.393+0000 7efd07fff700 1 --2- 172.21.15.29:0/96073474 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7efcf006c680 0x7efcf006eb30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efd081012a0 tx=0x7efcf8009eb0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:06.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.400+0000 7efd057fa700 1 -- 172.21.15.29:0/96073474 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7efcfc05af80 con 0x7efd08101440 2024-04-27T02:05:06.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.664+0000 7efd0e3aa700 1 -- 172.21.15.29:0/96073474 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efd08105d90 con 0x7efcf006c680 2024-04-27T02:05:06.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.665+0000 7efd057fa700 1 -- 172.21.15.29:0/96073474 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7efd08105d90 con 0x7efcf006c680 2024-04-27T02:05:06.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.667+0000 7efd0e3aa700 1 -- 172.21.15.29:0/96073474 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7efcf006c680 msgr2=0x7efcf006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:06.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.667+0000 7efd0e3aa700 1 --2- 172.21.15.29:0/96073474 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7efcf006c680 0x7efcf006eb30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efd081012a0 tx=0x7efcf8009eb0 comp rx=0 tx=0).stop 2024-04-27T02:05:06.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.667+0000 7efd0e3aa700 1 -- 172.21.15.29:0/96073474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd08101440 msgr2=0x7efd08195ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:06.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.667+0000 7efd0e3aa700 1 --2- 172.21.15.29:0/96073474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd08101440 0x7efd08195ff0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00bd70 tx=0x7efcfc00be50 comp rx=0 tx=0).stop 2024-04-27T02:05:06.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.667+0000 7efd0e3aa700 1 -- 172.21.15.29:0/96073474 shutdown_connections 2024-04-27T02:05:06.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.667+0000 7efd0e3aa700 1 --2- 172.21.15.29:0/96073474 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7efcf006c680 0x7efcf006eb30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:06.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.667+0000 7efd0e3aa700 1 --2- 172.21.15.29:0/96073474 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efd08100240 0x7efd08195ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:06.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.667+0000 7efd0e3aa700 1 --2- 172.21.15.29:0/96073474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd08101440 0x7efd08195ff0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:06.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.667+0000 7efd0e3aa700 1 -- 172.21.15.29:0/96073474 >> 172.21.15.29:0/96073474 conn(0x7efd080fb7f0 msgr2=0x7efd08104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:06.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.667+0000 7efd0e3aa700 1 -- 172.21.15.29:0/96073474 shutdown_connections 2024-04-27T02:05:06.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.667+0000 7efd0e3aa700 1 -- 172.21.15.29:0/96073474 wait complete. 2024-04-27T02:05:06.659 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-04-27T02:05:06.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.820+0000 7f939cf44700 1 -- 172.21.15.29:0/3509162677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93980ff130 msgr2=0x7f93980ff540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:06.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.820+0000 7f939cf44700 1 --2- 172.21.15.29:0/3509162677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93980ff130 0x7f93980ff540 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9388009910 tx=0x7f9388009c20 comp rx=0 tx=0).stop 2024-04-27T02:05:06.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.820+0000 7f939cf44700 1 -- 172.21.15.29:0/3509162677 shutdown_connections 2024-04-27T02:05:06.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.820+0000 7f939cf44700 1 --2- 172.21.15.29:0/3509162677 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93981003d0 0x7f9398100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:06.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.820+0000 7f939cf44700 1 --2- 172.21.15.29:0/3509162677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93980ff130 0x7f93980ff540 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:06.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.820+0000 7f939cf44700 1 -- 172.21.15.29:0/3509162677 >> 172.21.15.29:0/3509162677 conn(0x7f93980fa780 msgr2=0x7f93980fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:06.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.821+0000 7f939cf44700 1 -- 172.21.15.29:0/3509162677 shutdown_connections 2024-04-27T02:05:06.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.821+0000 7f939cf44700 1 -- 172.21.15.29:0/3509162677 wait complete. 2024-04-27T02:05:06.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.821+0000 7f939cf44700 1 Processor -- start 2024-04-27T02:05:06.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.822+0000 7f939cf44700 1 -- start start 2024-04-27T02:05:06.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.822+0000 7f939cf44700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93980ff130 0x7f93980690a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:06.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.822+0000 7f939cf44700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93981003d0 0x7f939819cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:06.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.822+0000 7f939cf44700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93981965f0 con 0x7f93980ff130 2024-04-27T02:05:06.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.822+0000 7f939cf44700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939819d150 con 0x7f93981003d0 2024-04-27T02:05:06.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.822+0000 7f939659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93980ff130 0x7f93980690a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:06.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.822+0000 7f939659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93980ff130 0x7f93980690a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35186/0 (socket says 172.21.15.29:35186) 2024-04-27T02:05:06.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.822+0000 7f939659c700 1 -- 172.21.15.29:0/1562315659 learned_addr learned my addr 172.21.15.29:0/1562315659 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:05:06.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.822+0000 7f939659c700 1 -- 172.21.15.29:0/1562315659 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93981003d0 msgr2=0x7f939819cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:06.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.822+0000 7f939659c700 1 --2- 172.21.15.29:0/1562315659 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93981003d0 0x7f939819cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:06.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.822+0000 7f939659c700 1 -- 172.21.15.29:0/1562315659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93880095f0 con 0x7f93980ff130 2024-04-27T02:05:06.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.822+0000 7f939659c700 1 --2- 172.21.15.29:0/1562315659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93980ff130 0x7f93980690a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f938800bda0 tx=0x7f938800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:06.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.823+0000 7f93877fe700 1 -- 172.21.15.29:0/1562315659 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f938801d070 con 0x7f93980ff130 2024-04-27T02:05:06.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.823+0000 7f939cf44700 1 -- 172.21.15.29:0/1562315659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f939819d350 con 0x7f93980ff130 2024-04-27T02:05:06.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.823+0000 7f93877fe700 1 -- 172.21.15.29:0/1562315659 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9388004c50 con 0x7f93980ff130 2024-04-27T02:05:06.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.823+0000 7f939cf44700 1 -- 172.21.15.29:0/1562315659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f939819d810 con 0x7f93980ff130 2024-04-27T02:05:06.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.823+0000 7f93877fe700 1 -- 172.21.15.29:0/1562315659 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9388021640 con 0x7f93980ff130 2024-04-27T02:05:06.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.824+0000 7f939cf44700 1 -- 172.21.15.29:0/1562315659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9378005320 con 0x7f93980ff130 2024-04-27T02:05:06.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.824+0000 7f93877fe700 1 -- 172.21.15.29:0/1562315659 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f938802b440 con 0x7f93980ff130 2024-04-27T02:05:06.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.825+0000 7f93877fe700 1 --2- 172.21.15.29:0/1562315659 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f938006c680 0x7f938006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:06.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.825+0000 7f93877fe700 1 -- 172.21.15.29:0/1562315659 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f938808cc80 con 0x7f93980ff130 2024-04-27T02:05:06.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.825+0000 7f9395d9b700 1 --2- 172.21.15.29:0/1562315659 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f938006c680 0x7f938006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:06.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.825+0000 7f9395d9b700 1 --2- 172.21.15.29:0/1562315659 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f938006c680 0x7f938006eb30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f938c005fd0 tx=0x7f938c005d00 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:06.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:06.829+0000 7f93877fe700 1 -- 172.21.15.29:0/1562315659 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f938805ad90 con 0x7f93980ff130 2024-04-27T02:05:06.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:06 smithi029 ceph-mon[26769]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:05:07.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.137+0000 7f939cf44700 1 -- 172.21.15.29:0/1562315659 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9378000bf0 con 0x7f938006c680 2024-04-27T02:05:07.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.138+0000 7f93877fe700 1 -- 172.21.15.29:0/1562315659 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f9378000bf0 con 0x7f938006c680 2024-04-27T02:05:07.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.140+0000 7f939cf44700 1 -- 172.21.15.29:0/1562315659 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f938006c680 msgr2=0x7f938006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:07.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.140+0000 7f939cf44700 1 --2- 172.21.15.29:0/1562315659 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f938006c680 0x7f938006eb30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f938c005fd0 tx=0x7f938c005d00 comp rx=0 tx=0).stop 2024-04-27T02:05:07.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.140+0000 7f939cf44700 1 -- 172.21.15.29:0/1562315659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93980ff130 msgr2=0x7f93980690a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:07.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.140+0000 7f939cf44700 1 --2- 172.21.15.29:0/1562315659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93980ff130 0x7f93980690a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f938800bda0 tx=0x7f938800be80 comp rx=0 tx=0).stop 2024-04-27T02:05:07.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.140+0000 7f939cf44700 1 -- 172.21.15.29:0/1562315659 shutdown_connections 2024-04-27T02:05:07.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.141+0000 7f939cf44700 1 --2- 172.21.15.29:0/1562315659 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f938006c680 0x7f938006eb30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:07.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.141+0000 7f939cf44700 1 --2- 172.21.15.29:0/1562315659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93980ff130 0x7f93980690a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:07.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.141+0000 7f939cf44700 1 --2- 172.21.15.29:0/1562315659 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93981003d0 0x7f939819cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:07.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.141+0000 7f939cf44700 1 -- 172.21.15.29:0/1562315659 >> 172.21.15.29:0/1562315659 conn(0x7f93980fa780 msgr2=0x7f9398103600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:07.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.141+0000 7f939cf44700 1 -- 172.21.15.29:0/1562315659 shutdown_connections 2024-04-27T02:05:07.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.141+0000 7f939cf44700 1 -- 172.21.15.29:0/1562315659 wait complete. 2024-04-27T02:05:07.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.265+0000 7f1b173c9700 1 -- 172.21.15.29:0/1516281190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b101003d0 msgr2=0x7f1b10100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:07.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.265+0000 7f1b173c9700 1 --2- 172.21.15.29:0/1516281190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b101003d0 0x7f1b10100840 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1b00009910 tx=0x7f1b00009c20 comp rx=0 tx=0).stop 2024-04-27T02:05:07.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.265+0000 7f1b173c9700 1 -- 172.21.15.29:0/1516281190 shutdown_connections 2024-04-27T02:05:07.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.265+0000 7f1b173c9700 1 --2- 172.21.15.29:0/1516281190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b101003d0 0x7f1b10100840 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:07.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.265+0000 7f1b173c9700 1 --2- 172.21.15.29:0/1516281190 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b100ff130 0x7f1b100ff540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:07.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.265+0000 7f1b173c9700 1 -- 172.21.15.29:0/1516281190 >> 172.21.15.29:0/1516281190 conn(0x7f1b100fa780 msgr2=0x7f1b100fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:07.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.265+0000 7f1b173c9700 1 -- 172.21.15.29:0/1516281190 shutdown_connections 2024-04-27T02:05:07.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.265+0000 7f1b173c9700 1 -- 172.21.15.29:0/1516281190 wait complete. 2024-04-27T02:05:07.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.266+0000 7f1b173c9700 1 Processor -- start 2024-04-27T02:05:07.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.266+0000 7f1b173c9700 1 -- start start 2024-04-27T02:05:07.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.266+0000 7f1b173c9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b100ff130 0x7f1b10103160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:07.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.266+0000 7f1b173c9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b101003d0 0x7f1b101017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:07.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.266+0000 7f1b173c9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b10101cf0 con 0x7f1b101003d0 2024-04-27T02:05:07.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.266+0000 7f1b173c9700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b10101e30 con 0x7f1b100ff130 2024-04-27T02:05:07.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.266+0000 7f1b14964700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b101003d0 0x7f1b101017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:07.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.266+0000 7f1b15165700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b100ff130 0x7f1b10103160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:07.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.266+0000 7f1b15165700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b100ff130 0x7f1b10103160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:59614/0 (socket says 172.21.15.29:59614) 2024-04-27T02:05:07.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.266+0000 7f1b15165700 1 -- 172.21.15.29:0/1233915038 learned_addr learned my addr 172.21.15.29:0/1233915038 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:05:07.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.267+0000 7f1b15165700 1 -- 172.21.15.29:0/1233915038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b101003d0 msgr2=0x7f1b101017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:07.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.267+0000 7f1b15165700 1 --2- 172.21.15.29:0/1233915038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b101003d0 0x7f1b101017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:07.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.267+0000 7f1b15165700 1 -- 172.21.15.29:0/1233915038 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b000095f0 con 0x7f1b100ff130 2024-04-27T02:05:07.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.267+0000 7f1b14964700 1 --2- 172.21.15.29:0/1233915038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b101003d0 0x7f1b101017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T02:05:07.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.267+0000 7f1b15165700 1 --2- 172.21.15.29:0/1233915038 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b100ff130 0x7f1b10103160 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c00d8d0 tx=0x7f1b0c00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:07.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.267+0000 7f1b067fc700 1 -- 172.21.15.29:0/1233915038 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b0c009940 con 0x7f1b100ff130 2024-04-27T02:05:07.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.267+0000 7f1b173c9700 1 -- 172.21.15.29:0/1233915038 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b10102110 con 0x7f1b100ff130 2024-04-27T02:05:07.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.267+0000 7f1b067fc700 1 -- 172.21.15.29:0/1233915038 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1b0c010460 con 0x7f1b100ff130 2024-04-27T02:05:07.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.267+0000 7f1b067fc700 1 -- 172.21.15.29:0/1233915038 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b0c00f5f0 con 0x7f1b100ff130 2024-04-27T02:05:07.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.267+0000 7f1b173c9700 1 -- 172.21.15.29:0/1233915038 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b101025a0 con 0x7f1b100ff130 2024-04-27T02:05:07.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.269+0000 7f1b173c9700 1 -- 172.21.15.29:0/1233915038 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b10107c10 con 0x7f1b100ff130 2024-04-27T02:05:07.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.269+0000 7f1b067fc700 1 -- 172.21.15.29:0/1233915038 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f1b0c0105d0 con 0x7f1b100ff130 2024-04-27T02:05:07.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.270+0000 7f1b067fc700 1 --2- 172.21.15.29:0/1233915038 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1afc06c680 0x7f1afc06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:07.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.270+0000 7f1b14964700 1 --2- 172.21.15.29:0/1233915038 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1afc06c680 0x7f1afc06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:07.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.270+0000 7f1b067fc700 1 -- 172.21.15.29:0/1233915038 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f1b0c08b8e0 con 0x7f1b100ff130 2024-04-27T02:05:07.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.270+0000 7f1b14964700 1 --2- 172.21.15.29:0/1233915038 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1afc06c680 0x7f1afc06eb30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1b00005b40 tx=0x7f1b00005a90 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:07.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:06 smithi052 ceph-mon[32586]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:05:07.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.275+0000 7f1b067fc700 1 -- 172.21.15.29:0/1233915038 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1b0c059a10 con 0x7f1b100ff130 2024-04-27T02:05:07.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.622+0000 7f1b173c9700 1 -- 172.21.15.29:0/1233915038 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b10068c50 con 0x7f1afc06c680 2024-04-27T02:05:07.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.636+0000 7f1b067fc700 1 -- 172.21.15.29:0/1233915038 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3480 (secure 0 0 0) 0x7f1b10068c50 con 0x7f1afc06c680 2024-04-27T02:05:07.616 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:05:07.616 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.smithi029 smithi029 *:9093,9094 running (5m) 26s ago 6m 25.9M - 0.25.0 c8568f914cd2 f713bca72a8f 2024-04-27T02:05:07.616 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-exporter.smithi029 smithi029 running (6m) 26s ago 6m 7428k - 18.2.0 dc2bc1663786 d585d22341b7 2024-04-27T02:05:07.616 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-exporter.smithi052 smithi052 running (5m) 32s ago 5m 7335k - 18.2.0 dc2bc1663786 3555a1bdbadc 2024-04-27T02:05:07.616 INFO:teuthology.orchestra.run.smithi029.stdout:crash.smithi029 smithi029 running (6m) 26s ago 6m 6665k - 18.2.0 dc2bc1663786 aa848ffaabfc 2024-04-27T02:05:07.616 INFO:teuthology.orchestra.run.smithi029.stdout:crash.smithi052 smithi052 running (5m) 32s ago 5m 6660k - 18.2.0 dc2bc1663786 460da511ea40 2024-04-27T02:05:07.616 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.smithi029 smithi029 *:3000 running (4m) 26s ago 6m 87.9M - 9.4.7 954c08fa6188 2e3e5d432724 2024-04-27T02:05:07.617 INFO:teuthology.orchestra.run.smithi029.stdout:mds.cephfs.smithi029.dyrjrd smithi029 running (54s) 26s ago 53s 11.5M - 18.2.0 dc2bc1663786 5e33fd05ca5f 2024-04-27T02:05:07.617 INFO:teuthology.orchestra.run.smithi029.stdout:mds.cephfs.smithi029.njjezc smithi029 running (48s) 26s ago 47s 17.1M - 18.2.0 dc2bc1663786 d278cd21f1d4 2024-04-27T02:05:07.617 INFO:teuthology.orchestra.run.smithi029.stdout:mds.cephfs.smithi052.nbjpjd smithi052 running (45s) 32s ago 45s 14.5M - 18.2.0 dc2bc1663786 b32817e8b1f9 2024-04-27T02:05:07.617 INFO:teuthology.orchestra.run.smithi029.stdout:mds.cephfs.smithi052.yylmkb smithi052 running (51s) 32s ago 51s 13.0M - 18.2.0 dc2bc1663786 9e395db428ee 2024-04-27T02:05:07.617 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.smithi029.ewvdzs smithi029 *:9283,8765,8443 running (8m) 26s ago 8m 486M - 18.2.0 dc2bc1663786 76689e16ce1a 2024-04-27T02:05:07.617 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.smithi052.gbsftx smithi052 *:8443,9283,8765 running (5m) 32s ago 5m 439M - 18.2.0 dc2bc1663786 72bb682980a1 2024-04-27T02:05:07.617 INFO:teuthology.orchestra.run.smithi029.stdout:mon.smithi029 smithi029 running (8m) 26s ago 8m 45.7M 2048M 18.2.0 dc2bc1663786 33130493e80d 2024-04-27T02:05:07.617 INFO:teuthology.orchestra.run.smithi029.stdout:mon.smithi052 smithi052 running (5m) 32s ago 5m 35.7M 2048M 18.2.0 dc2bc1663786 5af8b02bce2f 2024-04-27T02:05:07.617 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.smithi029 smithi029 *:9100 running (6m) 26s ago 6m 15.7M - 1.5.0 0da6a335fe13 f65d14d021a2 2024-04-27T02:05:07.617 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.smithi052 smithi052 *:9100 running (5m) 32s ago 5m 16.6M - 1.5.0 0da6a335fe13 71aadb2b8e2e 2024-04-27T02:05:07.617 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (4m) 26s ago 4m 47.5M 3948M 18.2.0 dc2bc1663786 b4688a5b18ec 2024-04-27T02:05:07.617 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (4m) 26s ago 4m 48.4M 3948M 18.2.0 dc2bc1663786 7123e1ad8219 2024-04-27T02:05:07.617 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (3m) 26s ago 3m 45.1M 3948M 18.2.0 dc2bc1663786 f8ecf2655a40 2024-04-27T02:05:07.617 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi052 running (3m) 32s ago 3m 46.3M 4972M 18.2.0 dc2bc1663786 584fb2e408cd 2024-04-27T02:05:07.618 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi052 running (2m) 32s ago 2m 43.1M 4972M 18.2.0 dc2bc1663786 26fdd5746d2a 2024-04-27T02:05:07.618 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi052 running (2m) 32s ago 2m 43.7M 4972M 18.2.0 dc2bc1663786 6fa7665653a4 2024-04-27T02:05:07.618 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.smithi029 smithi029 *:9095 running (4m) 26s ago 6m 64.6M - 2.43.0 a07b618ecd1d a9ebc9badf18 2024-04-27T02:05:07.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.638+0000 7f1b173c9700 1 -- 172.21.15.29:0/1233915038 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1afc06c680 msgr2=0x7f1afc06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:07.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.638+0000 7f1b173c9700 1 --2- 172.21.15.29:0/1233915038 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1afc06c680 0x7f1afc06eb30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1b00005b40 tx=0x7f1b00005a90 comp rx=0 tx=0).stop 2024-04-27T02:05:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.638+0000 7f1b173c9700 1 -- 172.21.15.29:0/1233915038 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b100ff130 msgr2=0x7f1b10103160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.638+0000 7f1b173c9700 1 --2- 172.21.15.29:0/1233915038 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b100ff130 0x7f1b10103160 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c00d8d0 tx=0x7f1b0c00dc90 comp rx=0 tx=0).stop 2024-04-27T02:05:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.638+0000 7f1b173c9700 1 -- 172.21.15.29:0/1233915038 shutdown_connections 2024-04-27T02:05:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.638+0000 7f1b173c9700 1 --2- 172.21.15.29:0/1233915038 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f1afc06c680 0x7f1afc06eb30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.638+0000 7f1b173c9700 1 --2- 172.21.15.29:0/1233915038 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b100ff130 0x7f1b10103160 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.638+0000 7f1b173c9700 1 --2- 172.21.15.29:0/1233915038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b101003d0 0x7f1b101017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.638+0000 7f1b173c9700 1 -- 172.21.15.29:0/1233915038 >> 172.21.15.29:0/1233915038 conn(0x7f1b100fa780 msgr2=0x7f1b10068540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.638+0000 7f1b173c9700 1 -- 172.21.15.29:0/1233915038 shutdown_connections 2024-04-27T02:05:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.638+0000 7f1b173c9700 1 -- 172.21.15.29:0/1233915038 wait complete. 2024-04-27T02:05:07.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.755+0000 7f199bfe9700 1 -- 172.21.15.29:0/3285631440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1994069270 msgr2=0x7f1994102dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:07.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.755+0000 7f199bfe9700 1 --2- 172.21.15.29:0/3285631440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1994069270 0x7f1994102dd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1990009910 tx=0x7f1990009c20 comp rx=0 tx=0).stop 2024-04-27T02:05:07.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.755+0000 7f199bfe9700 1 -- 172.21.15.29:0/3285631440 shutdown_connections 2024-04-27T02:05:07.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.755+0000 7f199bfe9700 1 --2- 172.21.15.29:0/3285631440 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1994103310 0x7f1994105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:07.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.755+0000 7f199bfe9700 1 --2- 172.21.15.29:0/3285631440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1994069270 0x7f1994102dd0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:07.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.755+0000 7f199bfe9700 1 -- 172.21.15.29:0/3285631440 >> 172.21.15.29:0/3285631440 conn(0x7f19940fa780 msgr2=0x7f19940fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:07.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.755+0000 7f199bfe9700 1 -- 172.21.15.29:0/3285631440 shutdown_connections 2024-04-27T02:05:07.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.755+0000 7f199bfe9700 1 -- 172.21.15.29:0/3285631440 wait complete. 2024-04-27T02:05:07.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.756+0000 7f199bfe9700 1 Processor -- start 2024-04-27T02:05:07.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.756+0000 7f199bfe9700 1 -- start start 2024-04-27T02:05:07.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.756+0000 7f199bfe9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1994103310 0x7f1994197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:07.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.756+0000 7f199bfe9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1994198220 0x7f199419d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:07.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.756+0000 7f199bfe9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1994198690 con 0x7f1994198220 2024-04-27T02:05:07.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.756+0000 7f199bfe9700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1994198800 con 0x7f1994103310 2024-04-27T02:05:07.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.756+0000 7f1999584700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1994198220 0x7f199419d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:07.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.756+0000 7f1999584700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1994198220 0x7f199419d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35212/0 (socket says 172.21.15.29:35212) 2024-04-27T02:05:07.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.756+0000 7f1999584700 1 -- 172.21.15.29:0/811390094 learned_addr learned my addr 172.21.15.29:0/811390094 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:05:07.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.757+0000 7f1999584700 1 -- 172.21.15.29:0/811390094 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1994103310 msgr2=0x7f1994197ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:07.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.757+0000 7f1999584700 1 --2- 172.21.15.29:0/811390094 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1994103310 0x7f1994197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:07.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.757+0000 7f1999584700 1 -- 172.21.15.29:0/811390094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19900095f0 con 0x7f1994198220 2024-04-27T02:05:07.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.757+0000 7f1999584700 1 --2- 172.21.15.29:0/811390094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1994198220 0x7f199419d250 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f198400c930 tx=0x7f198400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:07.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.757+0000 7f198affd700 1 -- 172.21.15.29:0/811390094 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1984007ad0 con 0x7f1994198220 2024-04-27T02:05:07.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.757+0000 7f198affd700 1 -- 172.21.15.29:0/811390094 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1984007c30 con 0x7f1994198220 2024-04-27T02:05:07.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.757+0000 7f198affd700 1 -- 172.21.15.29:0/811390094 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f198400e5e0 con 0x7f1994198220 2024-04-27T02:05:07.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.757+0000 7f199bfe9700 1 -- 172.21.15.29:0/811390094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f199419d790 con 0x7f1994198220 2024-04-27T02:05:07.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.757+0000 7f199bfe9700 1 -- 172.21.15.29:0/811390094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f199419dc60 con 0x7f1994198220 2024-04-27T02:05:07.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.759+0000 7f198affd700 1 -- 172.21.15.29:0/811390094 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f198400f450 con 0x7f1994198220 2024-04-27T02:05:07.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.759+0000 7f199bfe9700 1 -- 172.21.15.29:0/811390094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19940fc340 con 0x7f1994198220 2024-04-27T02:05:07.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.759+0000 7f198affd700 1 --2- 172.21.15.29:0/811390094 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f198006c680 0x7f198006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:07.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.760+0000 7f198affd700 1 -- 172.21.15.29:0/811390094 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f198408bb90 con 0x7f1994198220 2024-04-27T02:05:07.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.760+0000 7f1999d85700 1 --2- 172.21.15.29:0/811390094 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f198006c680 0x7f198006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:07.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.761+0000 7f1999d85700 1 --2- 172.21.15.29:0/811390094 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f198006c680 0x7f198006eb30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f199000b5c0 tx=0x7f1990005fd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:07.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:07.765+0000 7f198affd700 1 -- 172.21.15.29:0/811390094 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f19840561f0 con 0x7f1994198220 2024-04-27T02:05:08.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.165+0000 7f199bfe9700 1 -- 172.21.15.29:0/811390094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f199402cec0 con 0x7f1994198220 2024-04-27T02:05:08.146 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:07 smithi029 ceph-mon[26769]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:05:08.146 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:07 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:05:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.165+0000 7f198affd700 1 -- 172.21.15.29:0/811390094 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7f1984059810 con 0x7f1994198220 2024-04-27T02:05:08.146 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-04-27T02:05:08.147 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-04-27T02:05:08.147 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-04-27T02:05:08.147 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-27T02:05:08.147 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-04-27T02:05:08.147 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-04-27T02:05:08.147 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-27T02:05:08.148 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-04-27T02:05:08.148 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-04-27T02:05:08.148 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-27T02:05:08.148 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": { 2024-04-27T02:05:08.148 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-27T02:05:08.148 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-27T02:05:08.148 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-04-27T02:05:08.148 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 14 2024-04-27T02:05:08.148 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-27T02:05:08.148 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-04-27T02:05:08.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.168+0000 7f199bfe9700 1 -- 172.21.15.29:0/811390094 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f198006c680 msgr2=0x7f198006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:08.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.168+0000 7f199bfe9700 1 --2- 172.21.15.29:0/811390094 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f198006c680 0x7f198006eb30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f199000b5c0 tx=0x7f1990005fd0 comp rx=0 tx=0).stop 2024-04-27T02:05:08.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.168+0000 7f199bfe9700 1 -- 172.21.15.29:0/811390094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1994198220 msgr2=0x7f199419d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:08.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.168+0000 7f199bfe9700 1 --2- 172.21.15.29:0/811390094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1994198220 0x7f199419d250 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f198400c930 tx=0x7f198400ccf0 comp rx=0 tx=0).stop 2024-04-27T02:05:08.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.168+0000 7f199bfe9700 1 -- 172.21.15.29:0/811390094 shutdown_connections 2024-04-27T02:05:08.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.168+0000 7f199bfe9700 1 --2- 172.21.15.29:0/811390094 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f198006c680 0x7f198006eb30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:08.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.168+0000 7f199bfe9700 1 --2- 172.21.15.29:0/811390094 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1994103310 0x7f1994197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:08.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.168+0000 7f199bfe9700 1 --2- 172.21.15.29:0/811390094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1994198220 0x7f199419d250 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:08.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.168+0000 7f199bfe9700 1 -- 172.21.15.29:0/811390094 >> 172.21.15.29:0/811390094 conn(0x7f19940fa780 msgr2=0x7f1994100660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:08.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.168+0000 7f199bfe9700 1 -- 172.21.15.29:0/811390094 shutdown_connections 2024-04-27T02:05:08.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.168+0000 7f199bfe9700 1 -- 172.21.15.29:0/811390094 wait complete. 2024-04-27T02:05:08.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:07 smithi052 ceph-mon[32586]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:05:08.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:07 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:05:08.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.291+0000 7feddf15f700 1 -- 172.21.15.29:0/2276422798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd8100240 msgr2=0x7fedd8100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:08.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.291+0000 7feddf15f700 1 --2- 172.21.15.29:0/2276422798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd8100240 0x7fedd8100650 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fedcc009910 tx=0x7fedcc009c20 comp rx=0 tx=0).stop 2024-04-27T02:05:08.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.291+0000 7feddf15f700 1 -- 172.21.15.29:0/2276422798 shutdown_connections 2024-04-27T02:05:08.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.291+0000 7feddf15f700 1 --2- 172.21.15.29:0/2276422798 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedd8101440 0x7fedd8101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:08.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.291+0000 7feddf15f700 1 --2- 172.21.15.29:0/2276422798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd8100240 0x7fedd8100650 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:08.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.291+0000 7feddf15f700 1 -- 172.21.15.29:0/2276422798 >> 172.21.15.29:0/2276422798 conn(0x7fedd80fb7f0 msgr2=0x7fedd80fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:08.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.292+0000 7feddf15f700 1 -- 172.21.15.29:0/2276422798 shutdown_connections 2024-04-27T02:05:08.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.292+0000 7feddf15f700 1 -- 172.21.15.29:0/2276422798 wait complete. 2024-04-27T02:05:08.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.292+0000 7feddf15f700 1 Processor -- start 2024-04-27T02:05:08.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.293+0000 7feddf15f700 1 -- start start 2024-04-27T02:05:08.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.293+0000 7feddf15f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd8100240 0x7fedd8195ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:08.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.293+0000 7feddf15f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedd8101440 0x7fedd8195ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:08.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.293+0000 7feddf15f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd8196610 con 0x7fedd8100240 2024-04-27T02:05:08.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.293+0000 7feddf15f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd8196750 con 0x7fedd8101440 2024-04-27T02:05:08.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.293+0000 7feddcefb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd8100240 0x7fedd8195ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:08.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.293+0000 7feddcefb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd8100240 0x7fedd8195ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35242/0 (socket says 172.21.15.29:35242) 2024-04-27T02:05:08.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.293+0000 7feddcefb700 1 -- 172.21.15.29:0/3104556530 learned_addr learned my addr 172.21.15.29:0/3104556530 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:05:08.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.293+0000 7fedd7fff700 1 --2- 172.21.15.29:0/3104556530 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedd8101440 0x7fedd8195ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:08.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.293+0000 7feddcefb700 1 -- 172.21.15.29:0/3104556530 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedd8101440 msgr2=0x7fedd8195ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:08.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.293+0000 7feddcefb700 1 --2- 172.21.15.29:0/3104556530 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedd8101440 0x7fedd8195ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:08.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.293+0000 7feddcefb700 1 -- 172.21.15.29:0/3104556530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedcc0095f0 con 0x7fedd8100240 2024-04-27T02:05:08.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.294+0000 7feddcefb700 1 --2- 172.21.15.29:0/3104556530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd8100240 0x7fedd8195ab0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fedcc000c00 tx=0x7fedcc00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:08.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.294+0000 7fedd5ffb700 1 -- 172.21.15.29:0/3104556530 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedcc01d070 con 0x7fedd8100240 2024-04-27T02:05:08.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.294+0000 7fedd5ffb700 1 -- 172.21.15.29:0/3104556530 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fedcc005dc0 con 0x7fedd8100240 2024-04-27T02:05:08.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.294+0000 7fedd5ffb700 1 -- 172.21.15.29:0/3104556530 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedcc022470 con 0x7fedd8100240 2024-04-27T02:05:08.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.294+0000 7feddf15f700 1 -- 172.21.15.29:0/3104556530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd819b1a0 con 0x7fedd8100240 2024-04-27T02:05:08.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.294+0000 7feddf15f700 1 -- 172.21.15.29:0/3104556530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd819b690 con 0x7fedd8100240 2024-04-27T02:05:08.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.296+0000 7feddf15f700 1 -- 172.21.15.29:0/3104556530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd8108d90 con 0x7fedd8100240 2024-04-27T02:05:08.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.296+0000 7fedd5ffb700 1 -- 172.21.15.29:0/3104556530 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fedcc003810 con 0x7fedd8100240 2024-04-27T02:05:08.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.297+0000 7fedd5ffb700 1 --2- 172.21.15.29:0/3104556530 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fedc806c680 0x7fedc806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:08.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.297+0000 7fedd5ffb700 1 -- 172.21.15.29:0/3104556530 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fedcc08cac0 con 0x7fedd8100240 2024-04-27T02:05:08.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.297+0000 7fedd7fff700 1 --2- 172.21.15.29:0/3104556530 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fedc806c680 0x7fedc806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:08.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.298+0000 7fedd7fff700 1 --2- 172.21.15.29:0/3104556530 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fedc806c680 0x7fedc806eb30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fedc4009c60 tx=0x7fedc4009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:08.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.304+0000 7fedd5ffb700 1 -- 172.21.15.29:0/3104556530 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fedcc05abd0 con 0x7fedd8100240 2024-04-27T02:05:08.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.652+0000 7feddf15f700 1 -- 172.21.15.29:0/3104556530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fedd819bd90 con 0x7fedd8100240 2024-04-27T02:05:08.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.652+0000 7fedd5ffb700 1 -- 172.21.15.29:0/3104556530 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 19 v19) v1 ==== 76+0+1851 (secure 0 0 0) 0x7fedd819bd90 con 0x7fedd8100240 2024-04-27T02:05:08.632 INFO:teuthology.orchestra.run.smithi029.stdout:e19 2024-04-27T02:05:08.632 INFO:teuthology.orchestra.run.smithi029.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T02:05:08.633 INFO:teuthology.orchestra.run.smithi029.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T02:05:08.633 INFO:teuthology.orchestra.run.smithi029.stdout:legacy client fscid: 1 2024-04-27T02:05:08.633 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:05:08.633 INFO:teuthology.orchestra.run.smithi029.stdout:Filesystem 'cephfs' (1) 2024-04-27T02:05:08.633 INFO:teuthology.orchestra.run.smithi029.stdout:fs_name cephfs 2024-04-27T02:05:08.633 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 19 2024-04-27T02:05:08.633 INFO:teuthology.orchestra.run.smithi029.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-27T02:05:08.633 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-04-27T02:04:10.035536+0000 2024-04-27T02:05:08.633 INFO:teuthology.orchestra.run.smithi029.stdout:modified 2024-04-27T02:04:26.892992+0000 2024-04-27T02:05:08.633 INFO:teuthology.orchestra.run.smithi029.stdout:tableserver 0 2024-04-27T02:05:08.633 INFO:teuthology.orchestra.run.smithi029.stdout:root 0 2024-04-27T02:05:08.633 INFO:teuthology.orchestra.run.smithi029.stdout:session_timeout 60 2024-04-27T02:05:08.633 INFO:teuthology.orchestra.run.smithi029.stdout:session_autoclose 300 2024-04-27T02:05:08.633 INFO:teuthology.orchestra.run.smithi029.stdout:max_file_size 1099511627776 2024-04-27T02:05:08.633 INFO:teuthology.orchestra.run.smithi029.stdout:required_client_features {} 2024-04-27T02:05:08.634 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure 0 2024-04-27T02:05:08.634 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure_osd_epoch 46 2024-04-27T02:05:08.634 INFO:teuthology.orchestra.run.smithi029.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T02:05:08.634 INFO:teuthology.orchestra.run.smithi029.stdout:max_mds 1 2024-04-27T02:05:08.634 INFO:teuthology.orchestra.run.smithi029.stdout:in 0 2024-04-27T02:05:08.634 INFO:teuthology.orchestra.run.smithi029.stdout:up {0=14484} 2024-04-27T02:05:08.634 INFO:teuthology.orchestra.run.smithi029.stdout:failed 2024-04-27T02:05:08.634 INFO:teuthology.orchestra.run.smithi029.stdout:damaged 2024-04-27T02:05:08.634 INFO:teuthology.orchestra.run.smithi029.stdout:stopped 2024-04-27T02:05:08.634 INFO:teuthology.orchestra.run.smithi029.stdout:data_pools [3] 2024-04-27T02:05:08.634 INFO:teuthology.orchestra.run.smithi029.stdout:metadata_pool 2 2024-04-27T02:05:08.634 INFO:teuthology.orchestra.run.smithi029.stdout:inline_data enabled 2024-04-27T02:05:08.634 INFO:teuthology.orchestra.run.smithi029.stdout:balancer 2024-04-27T02:05:08.635 INFO:teuthology.orchestra.run.smithi029.stdout:bal_rank_mask -1 2024-04-27T02:05:08.635 INFO:teuthology.orchestra.run.smithi029.stdout:standby_count_wanted 1 2024-04-27T02:05:08.635 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi029.njjezc{0:14484} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.29:6828/1353923457,v1:172.21.15.29:6829/1353923457] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T02:05:08.635 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi052.yylmkb{0:14494} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.52:6824/2352841895,v1:172.21.15.52:6825/2352841895] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T02:05:08.635 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:05:08.635 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:05:08.635 INFO:teuthology.orchestra.run.smithi029.stdout:Standby daemons: 2024-04-27T02:05:08.635 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:05:08.635 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi029.dyrjrd{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.29:6826/3428808494,v1:172.21.15.29:6827/3428808494] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T02:05:08.635 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi052.nbjpjd{-1:24325} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.52:6826/3513632158,v1:172.21.15.52:6827/3513632158] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T02:05:08.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.654+0000 7feddf15f700 1 -- 172.21.15.29:0/3104556530 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fedc806c680 msgr2=0x7fedc806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:08.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.654+0000 7feddf15f700 1 --2- 172.21.15.29:0/3104556530 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fedc806c680 0x7fedc806eb30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fedc4009c60 tx=0x7fedc4009260 comp rx=0 tx=0).stop 2024-04-27T02:05:08.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.654+0000 7feddf15f700 1 -- 172.21.15.29:0/3104556530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd8100240 msgr2=0x7fedd8195ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:08.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.654+0000 7feddf15f700 1 --2- 172.21.15.29:0/3104556530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd8100240 0x7fedd8195ab0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fedcc000c00 tx=0x7fedcc00f690 comp rx=0 tx=0).stop 2024-04-27T02:05:08.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.655+0000 7feddf15f700 1 -- 172.21.15.29:0/3104556530 shutdown_connections 2024-04-27T02:05:08.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.655+0000 7feddf15f700 1 --2- 172.21.15.29:0/3104556530 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fedc806c680 0x7fedc806eb30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:08.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.655+0000 7feddf15f700 1 --2- 172.21.15.29:0/3104556530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd8100240 0x7fedd8195ab0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:08.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.655+0000 7feddf15f700 1 --2- 172.21.15.29:0/3104556530 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedd8101440 0x7fedd8195ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:08.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.655+0000 7feddf15f700 1 -- 172.21.15.29:0/3104556530 >> 172.21.15.29:0/3104556530 conn(0x7fedd80fb7f0 msgr2=0x7fedd8104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:08.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.655+0000 7feddf15f700 1 -- 172.21.15.29:0/3104556530 shutdown_connections 2024-04-27T02:05:08.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.655+0000 7feddf15f700 1 -- 172.21.15.29:0/3104556530 wait complete. 2024-04-27T02:05:08.637 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 19 2024-04-27T02:05:08.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.800+0000 7f2ffe219700 1 -- 172.21.15.29:0/4124229765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ff80ffb60 msgr2=0x7f2ff80fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:08.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.800+0000 7f2ffe219700 1 --2- 172.21.15.29:0/4124229765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ff80ffb60 0x7f2ff80fffd0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2fec009910 tx=0x7f2fec009c20 comp rx=0 tx=0).stop 2024-04-27T02:05:08.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.801+0000 7f2ffe219700 1 -- 172.21.15.29:0/4124229765 shutdown_connections 2024-04-27T02:05:08.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.801+0000 7f2ffe219700 1 --2- 172.21.15.29:0/4124229765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ff80ffb60 0x7f2ff80fffd0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:08.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.801+0000 7f2ffe219700 1 --2- 172.21.15.29:0/4124229765 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2ff80fe8c0 0x7f2ff80fecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:08.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.801+0000 7f2ffe219700 1 -- 172.21.15.29:0/4124229765 >> 172.21.15.29:0/4124229765 conn(0x7f2ff80f9f10 msgr2=0x7f2ff80fc340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:08.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.801+0000 7f2ffe219700 1 -- 172.21.15.29:0/4124229765 shutdown_connections 2024-04-27T02:05:08.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.801+0000 7f2ffe219700 1 -- 172.21.15.29:0/4124229765 wait complete. 2024-04-27T02:05:08.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.801+0000 7f2ffe219700 1 Processor -- start 2024-04-27T02:05:08.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.802+0000 7f2ffe219700 1 -- start start 2024-04-27T02:05:08.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.802+0000 7f2ffe219700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2ff80fe8c0 0x7f2ff8105900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:08.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.802+0000 7f2ffe219700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ff80ffb60 0x7f2ff819a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:08.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.802+0000 7f2ffe219700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff8194590 con 0x7f2ff80ffb60 2024-04-27T02:05:08.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.802+0000 7f2ffe219700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff8194700 con 0x7f2ff80fe8c0 2024-04-27T02:05:08.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.802+0000 7f2ff77fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2ff80fe8c0 0x7f2ff8105900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:08.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.802+0000 7f2ff6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ff80ffb60 0x7f2ff819a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.802+0000 7f2ff6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ff80ffb60 0x7f2ff819a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35250/0 (socket says 172.21.15.29:35250) 2024-04-27T02:05:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.802+0000 7f2ff77fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2ff80fe8c0 0x7f2ff8105900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:59686/0 (socket says 172.21.15.29:59686) 2024-04-27T02:05:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.802+0000 7f2ff6ffd700 1 -- 172.21.15.29:0/1337279337 learned_addr learned my addr 172.21.15.29:0/1337279337 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:05:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.802+0000 7f2ff77fe700 1 -- 172.21.15.29:0/1337279337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ff80ffb60 msgr2=0x7f2ff819a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.802+0000 7f2ff77fe700 1 --2- 172.21.15.29:0/1337279337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ff80ffb60 0x7f2ff819a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.802+0000 7f2ff77fe700 1 -- 172.21.15.29:0/1337279337 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec0095f0 con 0x7f2ff80fe8c0 2024-04-27T02:05:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.803+0000 7f2ff77fe700 1 --2- 172.21.15.29:0/1337279337 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2ff80fe8c0 0x7f2ff8105900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800e970 tx=0x7f2fe800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.803+0000 7f2ff4ff9700 1 -- 172.21.15.29:0/1337279337 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fe800c9d0 con 0x7f2ff80fe8c0 2024-04-27T02:05:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.803+0000 7f2ff4ff9700 1 -- 172.21.15.29:0/1337279337 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2fe8010430 con 0x7f2ff80fe8c0 2024-04-27T02:05:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.803+0000 7f2ffe219700 1 -- 172.21.15.29:0/1337279337 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff81949e0 con 0x7f2ff80fe8c0 2024-04-27T02:05:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.803+0000 7f2ffe219700 1 -- 172.21.15.29:0/1337279337 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff8194f30 con 0x7f2ff80fe8c0 2024-04-27T02:05:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.803+0000 7f2ff4ff9700 1 -- 172.21.15.29:0/1337279337 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fe8018650 con 0x7f2ff80fe8c0 2024-04-27T02:05:08.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.805+0000 7f2ffe219700 1 -- 172.21.15.29:0/1337279337 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe4005320 con 0x7f2ff80fe8c0 2024-04-27T02:05:08.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.805+0000 7f2ff4ff9700 1 -- 172.21.15.29:0/1337279337 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f2fe8018910 con 0x7f2ff80fe8c0 2024-04-27T02:05:08.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.806+0000 7f2ff4ff9700 1 --2- 172.21.15.29:0/1337279337 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f2fe006c680 0x7f2fe006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:08.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.806+0000 7f2ff4ff9700 1 -- 172.21.15.29:0/1337279337 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f2fe8014070 con 0x7f2ff80fe8c0 2024-04-27T02:05:08.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.806+0000 7f2ff6ffd700 1 --2- 172.21.15.29:0/1337279337 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f2fe006c680 0x7f2fe006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:08.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.806+0000 7f2ff6ffd700 1 --2- 172.21.15.29:0/1337279337 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f2fe006c680 0x7f2fe006eb30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8105e80 tx=0x7f2fec005940 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:08.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:08.810+0000 7f2ff4ff9700 1 -- 172.21.15.29:0/1337279337 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2fe8051720 con 0x7f2ff80fe8c0 2024-04-27T02:05:09.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.137+0000 7f2ffe219700 1 -- 172.21.15.29:0/1337279337 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2fe4000bf0 con 0x7f2fe006c680 2024-04-27T02:05:09.117 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:08 smithi029 ceph-mon[26769]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:05:09.118 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:08 smithi029 ceph-mon[26769]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:05:09.118 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:08 smithi029 ceph-mon[26769]: pgmap v214: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-27T02:05:09.118 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:08 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/811390094' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:05:09.118 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:08 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/3104556530' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T02:05:09.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.138+0000 7f2ff4ff9700 1 -- 172.21.15.29:0/1337279337 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f2fe4000bf0 con 0x7f2fe006c680 2024-04-27T02:05:09.118 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-04-27T02:05:09.118 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a", 2024-04-27T02:05:09.119 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-04-27T02:05:09.119 INFO:teuthology.orchestra.run.smithi029.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:05:09.119 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [], 2024-04-27T02:05:09.119 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "", 2024-04-27T02:05:09.119 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a image", 2024-04-27T02:05:09.119 INFO:teuthology.orchestra.run.smithi029.stdout: "is_paused": false 2024-04-27T02:05:09.119 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-04-27T02:05:09.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.140+0000 7f2ffe219700 1 -- 172.21.15.29:0/1337279337 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f2fe006c680 msgr2=0x7f2fe006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:09.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.140+0000 7f2ffe219700 1 --2- 172.21.15.29:0/1337279337 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f2fe006c680 0x7f2fe006eb30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8105e80 tx=0x7f2fec005940 comp rx=0 tx=0).stop 2024-04-27T02:05:09.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.140+0000 7f2ffe219700 1 -- 172.21.15.29:0/1337279337 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2ff80fe8c0 msgr2=0x7f2ff8105900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:09.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.140+0000 7f2ffe219700 1 --2- 172.21.15.29:0/1337279337 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2ff80fe8c0 0x7f2ff8105900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800e970 tx=0x7f2fe800ec80 comp rx=0 tx=0).stop 2024-04-27T02:05:09.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.140+0000 7f2ffe219700 1 -- 172.21.15.29:0/1337279337 shutdown_connections 2024-04-27T02:05:09.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.140+0000 7f2ffe219700 1 --2- 172.21.15.29:0/1337279337 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f2fe006c680 0x7f2fe006eb30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:09.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.140+0000 7f2ffe219700 1 --2- 172.21.15.29:0/1337279337 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2ff80fe8c0 0x7f2ff8105900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:09.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.140+0000 7f2ffe219700 1 --2- 172.21.15.29:0/1337279337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ff80ffb60 0x7f2ff819a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:09.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.140+0000 7f2ffe219700 1 -- 172.21.15.29:0/1337279337 >> 172.21.15.29:0/1337279337 conn(0x7f2ff80f9f10 msgr2=0x7f2ff8102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:09.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.141+0000 7f2ffe219700 1 -- 172.21.15.29:0/1337279337 shutdown_connections 2024-04-27T02:05:09.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.141+0000 7f2ffe219700 1 -- 172.21.15.29:0/1337279337 wait complete. 2024-04-27T02:05:09.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:08 smithi052 ceph-mon[32586]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:05:09.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:08 smithi052 ceph-mon[32586]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:05:09.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:08 smithi052 ceph-mon[32586]: pgmap v214: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-27T02:05:09.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:08 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/811390094' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:05:09.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:08 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/3104556530' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T02:05:09.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.296+0000 7f7d025ee700 1 -- 172.21.15.29:0/2820221977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cfc1003f0 msgr2=0x7f7cfc100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:09.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.296+0000 7f7d025ee700 1 --2- 172.21.15.29:0/2820221977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cfc1003f0 0x7f7cfc100860 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0009910 tx=0x7f7cf0009c20 comp rx=0 tx=0).stop 2024-04-27T02:05:09.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.296+0000 7f7d025ee700 1 -- 172.21.15.29:0/2820221977 shutdown_connections 2024-04-27T02:05:09.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.296+0000 7f7d025ee700 1 --2- 172.21.15.29:0/2820221977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cfc1003f0 0x7f7cfc100860 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:09.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.296+0000 7f7d025ee700 1 --2- 172.21.15.29:0/2820221977 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7cfc0ff150 0x7f7cfc0ff560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:09.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.297+0000 7f7d025ee700 1 -- 172.21.15.29:0/2820221977 >> 172.21.15.29:0/2820221977 conn(0x7f7cfc0fa7a0 msgr2=0x7f7cfc0fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:09.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.297+0000 7f7d025ee700 1 -- 172.21.15.29:0/2820221977 shutdown_connections 2024-04-27T02:05:09.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.297+0000 7f7d025ee700 1 -- 172.21.15.29:0/2820221977 wait complete. 2024-04-27T02:05:09.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.297+0000 7f7d025ee700 1 Processor -- start 2024-04-27T02:05:09.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.297+0000 7f7d025ee700 1 -- start start 2024-04-27T02:05:09.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.298+0000 7f7d025ee700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7cfc0ff150 0x7f7cfc069990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:09.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.298+0000 7f7d025ee700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cfc1003f0 0x7f7cfc19aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:09.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.298+0000 7f7d025ee700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc19b0d0 con 0x7f7cfc1003f0 2024-04-27T02:05:09.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.298+0000 7f7d025ee700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc19b210 con 0x7f7cfc0ff150 2024-04-27T02:05:09.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.298+0000 7f7cfb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cfc1003f0 0x7f7cfc19aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:09.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.298+0000 7f7cfb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cfc1003f0 0x7f7cfc19aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35258/0 (socket says 172.21.15.29:35258) 2024-04-27T02:05:09.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.298+0000 7f7cfb7fe700 1 -- 172.21.15.29:0/2798947149 learned_addr learned my addr 172.21.15.29:0/2798947149 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:05:09.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.298+0000 7f7cfbfff700 1 --2- 172.21.15.29:0/2798947149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7cfc0ff150 0x7f7cfc069990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:09.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.298+0000 7f7cfbfff700 1 -- 172.21.15.29:0/2798947149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cfc1003f0 msgr2=0x7f7cfc19aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:09.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.298+0000 7f7cfbfff700 1 --2- 172.21.15.29:0/2798947149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cfc1003f0 0x7f7cfc19aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:09.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.298+0000 7f7cfbfff700 1 -- 172.21.15.29:0/2798947149 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf00095f0 con 0x7f7cfc0ff150 2024-04-27T02:05:09.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.298+0000 7f7cfb7fe700 1 --2- 172.21.15.29:0/2798947149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cfc1003f0 0x7f7cfc19aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T02:05:09.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.299+0000 7f7cfbfff700 1 --2- 172.21.15.29:0/2798947149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7cfc0ff150 0x7f7cfc069990 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7cec00e970 tx=0x7f7cec00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:09.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.299+0000 7f7cf97fa700 1 -- 172.21.15.29:0/2798947149 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cec00c9d0 con 0x7f7cfc0ff150 2024-04-27T02:05:09.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.299+0000 7f7cf97fa700 1 -- 172.21.15.29:0/2798947149 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7cec010430 con 0x7f7cfc0ff150 2024-04-27T02:05:09.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.299+0000 7f7d025ee700 1 -- 172.21.15.29:0/2798947149 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cfc19b4f0 con 0x7f7cfc0ff150 2024-04-27T02:05:09.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.299+0000 7f7cf97fa700 1 -- 172.21.15.29:0/2798947149 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cec018650 con 0x7f7cfc0ff150 2024-04-27T02:05:09.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.299+0000 7f7d025ee700 1 -- 172.21.15.29:0/2798947149 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cfc19ba40 con 0x7f7cfc0ff150 2024-04-27T02:05:09.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.300+0000 7f7cf97fa700 1 -- 172.21.15.29:0/2798947149 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f7cec0105a0 con 0x7f7cfc0ff150 2024-04-27T02:05:09.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.300+0000 7f7d025ee700 1 -- 172.21.15.29:0/2798947149 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cfc068ab0 con 0x7f7cfc0ff150 2024-04-27T02:05:09.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.301+0000 7f7cf97fa700 1 --2- 172.21.15.29:0/2798947149 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f7ce406c6d0 0x7f7ce406eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:09.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.301+0000 7f7cf97fa700 1 -- 172.21.15.29:0/2798947149 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f7cec014070 con 0x7f7cfc0ff150 2024-04-27T02:05:09.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.301+0000 7f7cfb7fe700 1 --2- 172.21.15.29:0/2798947149 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f7ce406c6d0 0x7f7ce406eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:09.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.301+0000 7f7cfb7fe700 1 --2- 172.21.15.29:0/2798947149 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f7ce406c6d0 0x7f7ce406eb80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0009fd0 tx=0x7f7cf0005940 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:09.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.305+0000 7f7cf97fa700 1 -- 172.21.15.29:0/2798947149 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7cec051720 con 0x7f7cfc0ff150 2024-04-27T02:05:09.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.740+0000 7f7d025ee700 1 -- 172.21.15.29:0/2798947149 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7cfc19bd70 con 0x7f7cfc0ff150 2024-04-27T02:05:09.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.740+0000 7f7cf97fa700 1 -- 172.21.15.29:0/2798947149 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f7cec059d10 con 0x7f7cfc0ff150 2024-04-27T02:05:09.720 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-27T02:05:09.721 INFO:teuthology.orchestra.run.smithi029.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-27T02:05:09.721 INFO:teuthology.orchestra.run.smithi029.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-27T02:05:09.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.742+0000 7f7d025ee700 1 -- 172.21.15.29:0/2798947149 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f7ce406c6d0 msgr2=0x7f7ce406eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:09.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.742+0000 7f7d025ee700 1 --2- 172.21.15.29:0/2798947149 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f7ce406c6d0 0x7f7ce406eb80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0009fd0 tx=0x7f7cf0005940 comp rx=0 tx=0).stop 2024-04-27T02:05:09.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.742+0000 7f7d025ee700 1 -- 172.21.15.29:0/2798947149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7cfc0ff150 msgr2=0x7f7cfc069990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:09.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.742+0000 7f7d025ee700 1 --2- 172.21.15.29:0/2798947149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7cfc0ff150 0x7f7cfc069990 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7cec00e970 tx=0x7f7cec00ec80 comp rx=0 tx=0).stop 2024-04-27T02:05:09.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.742+0000 7f7d025ee700 1 -- 172.21.15.29:0/2798947149 shutdown_connections 2024-04-27T02:05:09.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.742+0000 7f7d025ee700 1 --2- 172.21.15.29:0/2798947149 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f7ce406c6d0 0x7f7ce406eb80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:09.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.742+0000 7f7d025ee700 1 --2- 172.21.15.29:0/2798947149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7cfc0ff150 0x7f7cfc069990 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:09.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.742+0000 7f7d025ee700 1 --2- 172.21.15.29:0/2798947149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cfc1003f0 0x7f7cfc19aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:09.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.742+0000 7f7d025ee700 1 -- 172.21.15.29:0/2798947149 >> 172.21.15.29:0/2798947149 conn(0x7f7cfc0fa7a0 msgr2=0x7f7cfc103720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:09.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.742+0000 7f7d025ee700 1 -- 172.21.15.29:0/2798947149 shutdown_connections 2024-04-27T02:05:09.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:09.742+0000 7f7d025ee700 1 -- 172.21.15.29:0/2798947149 wait complete. 2024-04-27T02:05:10.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:09 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/2798947149' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:05:10.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:09 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/2798947149' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:05:11.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:10 smithi029 ceph-mon[26769]: from='client.24393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:05:11.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:10 smithi029 ceph-mon[26769]: pgmap v215: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:05:11.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:10 smithi052 ceph-mon[32586]: from='client.24393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:05:11.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:10 smithi052 ceph-mon[32586]: pgmap v215: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:05:13.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:12 smithi029 ceph-mon[26769]: pgmap v216: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-27T02:05:13.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:12 smithi052 ceph-mon[32586]: pgmap v216: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-27T02:05:15.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:14 smithi029 ceph-mon[26769]: pgmap v217: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:05:15.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:14 smithi052 ceph-mon[32586]: pgmap v217: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:05:17.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:16 smithi029 ceph-mon[26769]: pgmap v218: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:05:17.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:16 smithi052 ceph-mon[32586]: pgmap v218: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:05:19.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:18 smithi029 ceph-mon[26769]: pgmap v219: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-27T02:05:19.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:18 smithi052 ceph-mon[32586]: pgmap v219: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-27T02:05:21.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:20 smithi029 ceph-mon[26769]: pgmap v220: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:05:21.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:20 smithi052 ceph-mon[32586]: pgmap v220: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:05:23.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:22 smithi029 ceph-mon[26769]: pgmap v221: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T02:05:23.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:22 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:05:23.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:22 smithi052 ceph-mon[32586]: pgmap v221: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T02:05:23.254 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:22 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:05:25.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:24 smithi029 ceph-mon[26769]: pgmap v222: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 255 B/s wr, 1 op/s 2024-04-27T02:05:25.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:24 smithi052 ceph-mon[32586]: pgmap v222: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 255 B/s wr, 1 op/s 2024-04-27T02:05:26.980 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:26 smithi029 ceph-mon[26769]: pgmap v223: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 255 B/s wr, 1 op/s 2024-04-27T02:05:27.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:26 smithi052 ceph-mon[32586]: pgmap v223: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 255 B/s wr, 1 op/s 2024-04-27T02:05:29.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:28 smithi029 ceph-mon[26769]: pgmap v224: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-27T02:05:29.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:28 smithi052 ceph-mon[32586]: pgmap v224: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-27T02:05:31.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:30 smithi029 ceph-mon[26769]: pgmap v225: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-27T02:05:31.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:30 smithi052 ceph-mon[32586]: pgmap v225: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-27T02:05:33.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:32 smithi029 ceph-mon[26769]: pgmap v226: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-27T02:05:33.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:32 smithi052 ceph-mon[32586]: pgmap v226: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-27T02:05:35.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:34 smithi029 ceph-mon[26769]: pgmap v227: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 341 B/s wr, 2 op/s 2024-04-27T02:05:35.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:34 smithi052 ceph-mon[32586]: pgmap v227: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 341 B/s wr, 2 op/s 2024-04-27T02:05:37.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:36 smithi029 ceph-mon[26769]: pgmap v228: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 341 B/s wr, 2 op/s 2024-04-27T02:05:37.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:36 smithi052 ceph-mon[32586]: pgmap v228: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 341 B/s wr, 2 op/s 2024-04-27T02:05:38.154 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:37 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:05:38.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:37 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:05:39.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:38 smithi029 ceph-mon[26769]: pgmap v229: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 341 B/s wr, 3 op/s 2024-04-27T02:05:39.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:38 smithi052 ceph-mon[32586]: pgmap v229: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 341 B/s wr, 3 op/s 2024-04-27T02:05:39.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.905+0000 7ff53b47d700 1 -- 172.21.15.29:0/3913744177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5340fdfc0 msgr2=0x7ff5340fe3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:39.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.905+0000 7ff53b47d700 1 --2- 172.21.15.29:0/3913744177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5340fdfc0 0x7ff5340fe3d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff530009910 tx=0x7ff530009c20 comp rx=0 tx=0).stop 2024-04-27T02:05:39.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.905+0000 7ff53b47d700 1 -- 172.21.15.29:0/3913744177 shutdown_connections 2024-04-27T02:05:39.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.905+0000 7ff53b47d700 1 --2- 172.21.15.29:0/3913744177 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5340fe910 0x7ff534105930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:39.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.905+0000 7ff53b47d700 1 --2- 172.21.15.29:0/3913744177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5340fdfc0 0x7ff5340fe3d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:39.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.905+0000 7ff53b47d700 1 -- 172.21.15.29:0/3913744177 >> 172.21.15.29:0/3913744177 conn(0x7ff5340f9a50 msgr2=0x7ff5340fbe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:39.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.906+0000 7ff53b47d700 1 -- 172.21.15.29:0/3913744177 shutdown_connections 2024-04-27T02:05:39.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.906+0000 7ff53b47d700 1 -- 172.21.15.29:0/3913744177 wait complete. 2024-04-27T02:05:39.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.906+0000 7ff53b47d700 1 Processor -- start 2024-04-27T02:05:39.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.907+0000 7ff53b47d700 1 -- start start 2024-04-27T02:05:39.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.907+0000 7ff53b47d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5340fdfc0 0x7ff5341938f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:39.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.907+0000 7ff53b47d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5340fe910 0x7ff534193e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:39.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.907+0000 7ff53b47d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff534194450 con 0x7ff5340fdfc0 2024-04-27T02:05:39.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.907+0000 7ff53b47d700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff534194590 con 0x7ff5340fe910 2024-04-27T02:05:39.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.908+0000 7ff539219700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5340fdfc0 0x7ff5341938f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:39.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.908+0000 7ff539219700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5340fdfc0 0x7ff5341938f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49646/0 (socket says 172.21.15.29:49646) 2024-04-27T02:05:39.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.908+0000 7ff539219700 1 -- 172.21.15.29:0/3257604541 learned_addr learned my addr 172.21.15.29:0/3257604541 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:05:39.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.908+0000 7ff538a18700 1 --2- 172.21.15.29:0/3257604541 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5340fe910 0x7ff534193e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:39.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.908+0000 7ff538a18700 1 -- 172.21.15.29:0/3257604541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5340fdfc0 msgr2=0x7ff5341938f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:39.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.908+0000 7ff538a18700 1 --2- 172.21.15.29:0/3257604541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5340fdfc0 0x7ff5341938f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:39.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.908+0000 7ff538a18700 1 -- 172.21.15.29:0/3257604541 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5300095f0 con 0x7ff5340fe910 2024-04-27T02:05:39.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.909+0000 7ff539219700 1 --2- 172.21.15.29:0/3257604541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5340fdfc0 0x7ff5341938f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T02:05:39.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.909+0000 7ff538a18700 1 --2- 172.21.15.29:0/3257604541 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5340fe910 0x7ff534193e30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff5340ff520 tx=0x7ff52400ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:39.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.909+0000 7ff52a7fc700 1 -- 172.21.15.29:0/3257604541 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff524004200 con 0x7ff5340fe910 2024-04-27T02:05:39.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.909+0000 7ff52a7fc700 1 -- 172.21.15.29:0/3257604541 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff524004360 con 0x7ff5340fe910 2024-04-27T02:05:39.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.909+0000 7ff52a7fc700 1 -- 172.21.15.29:0/3257604541 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff524003d00 con 0x7ff5340fe910 2024-04-27T02:05:39.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.909+0000 7ff53b47d700 1 -- 172.21.15.29:0/3257604541 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff534195020 con 0x7ff5340fe910 2024-04-27T02:05:39.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.909+0000 7ff53b47d700 1 -- 172.21.15.29:0/3257604541 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff534195570 con 0x7ff5340fe910 2024-04-27T02:05:39.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.910+0000 7ff53b47d700 1 -- 172.21.15.29:0/3257604541 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5341969d0 con 0x7ff5340fe910 2024-04-27T02:05:39.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.912+0000 7ff52a7fc700 1 -- 172.21.15.29:0/3257604541 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7ff52400c7c0 con 0x7ff5340fe910 2024-04-27T02:05:39.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.912+0000 7ff52a7fc700 1 --2- 172.21.15.29:0/3257604541 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ff52006c6d0 0x7ff52006eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:39.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.912+0000 7ff52a7fc700 1 -- 172.21.15.29:0/3257604541 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ff524014070 con 0x7ff5340fe910 2024-04-27T02:05:39.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.913+0000 7ff539219700 1 --2- 172.21.15.29:0/3257604541 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ff52006c6d0 0x7ff52006eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:39.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.913+0000 7ff539219700 1 --2- 172.21.15.29:0/3257604541 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ff52006c6d0 0x7ff52006eb80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff530009fd0 tx=0x7ff530005940 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:39.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:39.921+0000 7ff52a7fc700 1 -- 172.21.15.29:0/3257604541 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff524056660 con 0x7ff5340fe910 2024-04-27T02:05:40.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.309+0000 7ff53b47d700 1 -- 172.21.15.29:0/3257604541 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff534104da0 con 0x7ff52006c6d0 2024-04-27T02:05:40.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.311+0000 7ff52a7fc700 1 -- 172.21.15.29:0/3257604541 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7ff534104da0 con 0x7ff52006c6d0 2024-04-27T02:05:40.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.314+0000 7ff53b47d700 1 -- 172.21.15.29:0/3257604541 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ff52006c6d0 msgr2=0x7ff52006eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:40.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.314+0000 7ff53b47d700 1 --2- 172.21.15.29:0/3257604541 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ff52006c6d0 0x7ff52006eb80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff530009fd0 tx=0x7ff530005940 comp rx=0 tx=0).stop 2024-04-27T02:05:40.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.314+0000 7ff53b47d700 1 -- 172.21.15.29:0/3257604541 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5340fe910 msgr2=0x7ff534193e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:40.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.314+0000 7ff53b47d700 1 --2- 172.21.15.29:0/3257604541 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5340fe910 0x7ff534193e30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff5340ff520 tx=0x7ff52400ec20 comp rx=0 tx=0).stop 2024-04-27T02:05:40.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.314+0000 7ff53b47d700 1 -- 172.21.15.29:0/3257604541 shutdown_connections 2024-04-27T02:05:40.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.314+0000 7ff53b47d700 1 --2- 172.21.15.29:0/3257604541 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7ff52006c6d0 0x7ff52006eb80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:40.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.314+0000 7ff53b47d700 1 --2- 172.21.15.29:0/3257604541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5340fdfc0 0x7ff5341938f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:40.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.314+0000 7ff53b47d700 1 --2- 172.21.15.29:0/3257604541 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5340fe910 0x7ff534193e30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:40.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.314+0000 7ff53b47d700 1 -- 172.21.15.29:0/3257604541 >> 172.21.15.29:0/3257604541 conn(0x7ff5340f9a50 msgr2=0x7ff534101fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:40.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.314+0000 7ff53b47d700 1 -- 172.21.15.29:0/3257604541 shutdown_connections 2024-04-27T02:05:40.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.314+0000 7ff53b47d700 1 -- 172.21.15.29:0/3257604541 wait complete. 2024-04-27T02:05:40.310 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-04-27T02:05:40.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.486+0000 7fe306f42700 1 -- 172.21.15.29:0/1848500264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3001003d0 msgr2=0x7fe300100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:40.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.486+0000 7fe306f42700 1 --2- 172.21.15.29:0/1848500264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3001003d0 0x7fe300100840 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec009910 tx=0x7fe2ec009c20 comp rx=0 tx=0).stop 2024-04-27T02:05:40.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.486+0000 7fe306f42700 1 -- 172.21.15.29:0/1848500264 shutdown_connections 2024-04-27T02:05:40.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.486+0000 7fe306f42700 1 --2- 172.21.15.29:0/1848500264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3001003d0 0x7fe300100840 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:40.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.486+0000 7fe306f42700 1 --2- 172.21.15.29:0/1848500264 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3000ff130 0x7fe3000ff540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:40.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.486+0000 7fe306f42700 1 -- 172.21.15.29:0/1848500264 >> 172.21.15.29:0/1848500264 conn(0x7fe3000fa780 msgr2=0x7fe3000fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:40.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.486+0000 7fe306f42700 1 -- 172.21.15.29:0/1848500264 shutdown_connections 2024-04-27T02:05:40.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.486+0000 7fe306f42700 1 -- 172.21.15.29:0/1848500264 wait complete. 2024-04-27T02:05:40.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.488+0000 7fe306f42700 1 Processor -- start 2024-04-27T02:05:40.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.488+0000 7fe306f42700 1 -- start start 2024-04-27T02:05:40.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.488+0000 7fe306f42700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3000ff130 0x7fe300194960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.488+0000 7fe304cde700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3000ff130 0x7fe300194960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.488+0000 7fe304cde700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3000ff130 0x7fe300194960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49674/0 (socket says 172.21.15.29:49674) 2024-04-27T02:05:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.489+0000 7fe306f42700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3001003d0 0x7fe300194ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.489+0000 7fe304cde700 1 -- 172.21.15.29:0/1458727896 learned_addr learned my addr 172.21.15.29:0/1458727896 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:05:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.489+0000 7fe306f42700 1 -- 172.21.15.29:0/1458727896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3001954c0 con 0x7fe3000ff130 2024-04-27T02:05:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.489+0000 7fe306f42700 1 -- 172.21.15.29:0/1458727896 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe300195600 con 0x7fe3001003d0 2024-04-27T02:05:40.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.489+0000 7fe2fffff700 1 --2- 172.21.15.29:0/1458727896 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3001003d0 0x7fe300194ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:40.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.490+0000 7fe304cde700 1 -- 172.21.15.29:0/1458727896 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3001003d0 msgr2=0x7fe300194ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:40.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.490+0000 7fe304cde700 1 --2- 172.21.15.29:0/1458727896 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3001003d0 0x7fe300194ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:40.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.490+0000 7fe304cde700 1 -- 172.21.15.29:0/1458727896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2ec0095f0 con 0x7fe3000ff130 2024-04-27T02:05:40.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.490+0000 7fe304cde700 1 --2- 172.21.15.29:0/1458727896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3000ff130 0x7fe300194960 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe2f400e920 tx=0x7fe2f400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:40.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.491+0000 7fe2fdffb700 1 -- 172.21.15.29:0/1458727896 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2f400c9d0 con 0x7fe3000ff130 2024-04-27T02:05:40.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.491+0000 7fe2fdffb700 1 -- 172.21.15.29:0/1458727896 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe2f4010430 con 0x7fe3000ff130 2024-04-27T02:05:40.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.491+0000 7fe2fdffb700 1 -- 172.21.15.29:0/1458727896 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2f4018650 con 0x7fe3000ff130 2024-04-27T02:05:40.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.491+0000 7fe306f42700 1 -- 172.21.15.29:0/1458727896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3000685a0 con 0x7fe3000ff130 2024-04-27T02:05:40.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.491+0000 7fe306f42700 1 -- 172.21.15.29:0/1458727896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe300068af0 con 0x7fe3000ff130 2024-04-27T02:05:40.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.493+0000 7fe2fdffb700 1 -- 172.21.15.29:0/1458727896 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fe2f40105a0 con 0x7fe3000ff130 2024-04-27T02:05:40.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.493+0000 7fe306f42700 1 -- 172.21.15.29:0/1458727896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe300107c10 con 0x7fe3000ff130 2024-04-27T02:05:40.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.494+0000 7fe2fdffb700 1 --2- 172.21.15.29:0/1458727896 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fe2f0070bb0 0x7fe2f0073060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:40.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.494+0000 7fe2fdffb700 1 -- 172.21.15.29:0/1458727896 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fe2f4014070 con 0x7fe3000ff130 2024-04-27T02:05:40.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.495+0000 7fe2fffff700 1 --2- 172.21.15.29:0/1458727896 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fe2f0070bb0 0x7fe2f0073060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:40.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.495+0000 7fe2fffff700 1 --2- 172.21.15.29:0/1458727896 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fe2f0070bb0 0x7fe2f0073060 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec009fd0 tx=0x7fe2ec005940 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:40.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.502+0000 7fe2fdffb700 1 -- 172.21.15.29:0/1458727896 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe2f4051720 con 0x7fe3000ff130 2024-04-27T02:05:40.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.772+0000 7fe306f42700 1 -- 172.21.15.29:0/1458727896 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe300104d20 con 0x7fe2f0070bb0 2024-04-27T02:05:40.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.773+0000 7fe2fdffb700 1 -- 172.21.15.29:0/1458727896 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7fe300104d20 con 0x7fe2f0070bb0 2024-04-27T02:05:40.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.776+0000 7fe306f42700 1 -- 172.21.15.29:0/1458727896 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fe2f0070bb0 msgr2=0x7fe2f0073060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:40.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.776+0000 7fe306f42700 1 --2- 172.21.15.29:0/1458727896 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fe2f0070bb0 0x7fe2f0073060 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec009fd0 tx=0x7fe2ec005940 comp rx=0 tx=0).stop 2024-04-27T02:05:40.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.776+0000 7fe306f42700 1 -- 172.21.15.29:0/1458727896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3000ff130 msgr2=0x7fe300194960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:40.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.776+0000 7fe306f42700 1 --2- 172.21.15.29:0/1458727896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3000ff130 0x7fe300194960 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe2f400e920 tx=0x7fe2f400ec30 comp rx=0 tx=0).stop 2024-04-27T02:05:40.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.776+0000 7fe306f42700 1 -- 172.21.15.29:0/1458727896 shutdown_connections 2024-04-27T02:05:40.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.776+0000 7fe306f42700 1 --2- 172.21.15.29:0/1458727896 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fe2f0070bb0 0x7fe2f0073060 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:40.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.776+0000 7fe306f42700 1 --2- 172.21.15.29:0/1458727896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3000ff130 0x7fe300194960 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:40.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.776+0000 7fe306f42700 1 --2- 172.21.15.29:0/1458727896 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3001003d0 0x7fe300194ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:40.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.776+0000 7fe306f42700 1 -- 172.21.15.29:0/1458727896 >> 172.21.15.29:0/1458727896 conn(0x7fe3000fa780 msgr2=0x7fe300103600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:40.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.776+0000 7fe306f42700 1 -- 172.21.15.29:0/1458727896 shutdown_connections 2024-04-27T02:05:40.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.776+0000 7fe306f42700 1 -- 172.21.15.29:0/1458727896 wait complete. 2024-04-27T02:05:40.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.947+0000 7f3707ddd700 1 -- 172.21.15.29:0/3770530822 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f37000fe8c0 msgr2=0x7f37000fecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:40.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.947+0000 7f3707ddd700 1 --2- 172.21.15.29:0/3770530822 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f37000fe8c0 0x7f37000fecd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f36fc009870 tx=0x7f36fc009b80 comp rx=0 tx=0).stop 2024-04-27T02:05:40.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.947+0000 7f3707ddd700 1 -- 172.21.15.29:0/3770530822 shutdown_connections 2024-04-27T02:05:40.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.947+0000 7f3707ddd700 1 --2- 172.21.15.29:0/3770530822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37000ffb60 0x7f37000fffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:40.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.947+0000 7f3707ddd700 1 --2- 172.21.15.29:0/3770530822 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f37000fe8c0 0x7f37000fecd0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:40.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.947+0000 7f3707ddd700 1 -- 172.21.15.29:0/3770530822 >> 172.21.15.29:0/3770530822 conn(0x7f37000f9f10 msgr2=0x7f37000fc340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:40.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.947+0000 7f3707ddd700 1 -- 172.21.15.29:0/3770530822 shutdown_connections 2024-04-27T02:05:40.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.947+0000 7f3707ddd700 1 -- 172.21.15.29:0/3770530822 wait complete. 2024-04-27T02:05:40.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.948+0000 7f3707ddd700 1 Processor -- start 2024-04-27T02:05:40.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.948+0000 7f3707ddd700 1 -- start start 2024-04-27T02:05:40.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.948+0000 7f3707ddd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37000fe8c0 0x7f3700105900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:40.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.948+0000 7f3707ddd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f37000ffb60 0x7f370019cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:40.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.948+0000 7f3707ddd700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3700196680 con 0x7f37000fe8c0 2024-04-27T02:05:40.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.948+0000 7f3707ddd700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370019d150 con 0x7f37000ffb60 2024-04-27T02:05:40.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.948+0000 7f3705b79700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37000fe8c0 0x7f3700105900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:40.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.948+0000 7f3705b79700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37000fe8c0 0x7f3700105900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49684/0 (socket says 172.21.15.29:49684) 2024-04-27T02:05:40.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.948+0000 7f3705b79700 1 -- 172.21.15.29:0/626968456 learned_addr learned my addr 172.21.15.29:0/626968456 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:05:40.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.948+0000 7f3705378700 1 --2- 172.21.15.29:0/626968456 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f37000ffb60 0x7f370019cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:40.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.949+0000 7f3705b79700 1 -- 172.21.15.29:0/626968456 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f37000ffb60 msgr2=0x7f370019cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:40.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.949+0000 7f3705b79700 1 --2- 172.21.15.29:0/626968456 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f37000ffb60 0x7f370019cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:40.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.949+0000 7f3705b79700 1 -- 172.21.15.29:0/626968456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f00095f0 con 0x7f37000fe8c0 2024-04-27T02:05:40.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.949+0000 7f3705b79700 1 --2- 172.21.15.29:0/626968456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37000fe8c0 0x7f3700105900 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f36fc009230 tx=0x7f36fc00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:40.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.949+0000 7f36f6ffd700 1 -- 172.21.15.29:0/626968456 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36fc01c070 con 0x7f37000fe8c0 2024-04-27T02:05:40.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.950+0000 7f36f6ffd700 1 -- 172.21.15.29:0/626968456 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f36fc00be80 con 0x7f37000fe8c0 2024-04-27T02:05:40.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.950+0000 7f3707ddd700 1 -- 172.21.15.29:0/626968456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36fc009520 con 0x7f37000fe8c0 2024-04-27T02:05:40.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.950+0000 7f3707ddd700 1 -- 172.21.15.29:0/626968456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f370019d700 con 0x7f37000fe8c0 2024-04-27T02:05:40.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.950+0000 7f36f6ffd700 1 -- 172.21.15.29:0/626968456 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36fc022e50 con 0x7f37000fe8c0 2024-04-27T02:05:40.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.951+0000 7f3707ddd700 1 -- 172.21.15.29:0/626968456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e4005320 con 0x7f37000fe8c0 2024-04-27T02:05:40.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.951+0000 7f36f6ffd700 1 -- 172.21.15.29:0/626968456 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f36fc020070 con 0x7f37000fe8c0 2024-04-27T02:05:40.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.952+0000 7f36f6ffd700 1 --2- 172.21.15.29:0/626968456 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f36ec06c6d0 0x7f36ec06eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:40.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.952+0000 7f3705378700 1 --2- 172.21.15.29:0/626968456 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f36ec06c6d0 0x7f36ec06eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:40.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.952+0000 7f3705378700 1 --2- 172.21.15.29:0/626968456 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f36ec06c6d0 0x7f36ec06eb80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3700105e80 tx=0x7f36f0009310 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:40.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.953+0000 7f36f6ffd700 1 -- 172.21.15.29:0/626968456 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f36fc08cb00 con 0x7f37000fe8c0 2024-04-27T02:05:40.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:40.961+0000 7f36f6ffd700 1 -- 172.21.15.29:0/626968456 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f36fc05abf0 con 0x7f37000fe8c0 2024-04-27T02:05:41.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:40 smithi029 ceph-mon[26769]: pgmap v230: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-04-27T02:05:41.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:40 smithi052 ceph-mon[32586]: pgmap v230: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-04-27T02:05:41.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.307+0000 7f3707ddd700 1 -- 172.21.15.29:0/626968456 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f36e4000bf0 con 0x7f36ec06c6d0 2024-04-27T02:05:41.307 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:05:41.308 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.smithi029 smithi029 *:9093,9094 running (5m) 60s ago 7m 25.9M - 0.25.0 c8568f914cd2 f713bca72a8f 2024-04-27T02:05:41.308 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-exporter.smithi029 smithi029 running (7m) 60s ago 7m 7428k - 18.2.0 dc2bc1663786 d585d22341b7 2024-04-27T02:05:41.308 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-exporter.smithi052 smithi052 running (6m) 65s ago 6m 7335k - 18.2.0 dc2bc1663786 3555a1bdbadc 2024-04-27T02:05:41.308 INFO:teuthology.orchestra.run.smithi029.stdout:crash.smithi029 smithi029 running (7m) 60s ago 7m 6665k - 18.2.0 dc2bc1663786 aa848ffaabfc 2024-04-27T02:05:41.308 INFO:teuthology.orchestra.run.smithi029.stdout:crash.smithi052 smithi052 running (6m) 65s ago 6m 6660k - 18.2.0 dc2bc1663786 460da511ea40 2024-04-27T02:05:41.308 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.smithi029 smithi029 *:3000 running (5m) 60s ago 6m 87.9M - 9.4.7 954c08fa6188 2e3e5d432724 2024-04-27T02:05:41.308 INFO:teuthology.orchestra.run.smithi029.stdout:mds.cephfs.smithi029.dyrjrd smithi029 running (87s) 60s ago 87s 11.5M - 18.2.0 dc2bc1663786 5e33fd05ca5f 2024-04-27T02:05:41.308 INFO:teuthology.orchestra.run.smithi029.stdout:mds.cephfs.smithi029.njjezc smithi029 running (82s) 60s ago 81s 17.1M - 18.2.0 dc2bc1663786 d278cd21f1d4 2024-04-27T02:05:41.308 INFO:teuthology.orchestra.run.smithi029.stdout:mds.cephfs.smithi052.nbjpjd smithi052 running (79s) 65s ago 78s 14.5M - 18.2.0 dc2bc1663786 b32817e8b1f9 2024-04-27T02:05:41.308 INFO:teuthology.orchestra.run.smithi029.stdout:mds.cephfs.smithi052.yylmkb smithi052 running (85s) 65s ago 84s 13.0M - 18.2.0 dc2bc1663786 9e395db428ee 2024-04-27T02:05:41.308 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.smithi029.ewvdzs smithi029 *:9283,8765,8443 running (8m) 60s ago 8m 486M - 18.2.0 dc2bc1663786 76689e16ce1a 2024-04-27T02:05:41.308 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.smithi052.gbsftx smithi052 *:8443,9283,8765 running (6m) 65s ago 6m 439M - 18.2.0 dc2bc1663786 72bb682980a1 2024-04-27T02:05:41.308 INFO:teuthology.orchestra.run.smithi029.stdout:mon.smithi029 smithi029 running (8m) 60s ago 8m 45.7M 2048M 18.2.0 dc2bc1663786 33130493e80d 2024-04-27T02:05:41.308 INFO:teuthology.orchestra.run.smithi029.stdout:mon.smithi052 smithi052 running (6m) 65s ago 6m 35.7M 2048M 18.2.0 dc2bc1663786 5af8b02bce2f 2024-04-27T02:05:41.308 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.smithi029 smithi029 *:9100 running (7m) 60s ago 7m 15.7M - 1.5.0 0da6a335fe13 f65d14d021a2 2024-04-27T02:05:41.309 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.smithi052 smithi052 *:9100 running (6m) 65s ago 6m 16.6M - 1.5.0 0da6a335fe13 71aadb2b8e2e 2024-04-27T02:05:41.309 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (5m) 60s ago 5m 47.5M 3948M 18.2.0 dc2bc1663786 b4688a5b18ec 2024-04-27T02:05:41.309 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (4m) 60s ago 4m 48.4M 3948M 18.2.0 dc2bc1663786 7123e1ad8219 2024-04-27T02:05:41.309 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (4m) 60s ago 4m 45.1M 3948M 18.2.0 dc2bc1663786 f8ecf2655a40 2024-04-27T02:05:41.309 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi052 running (3m) 65s ago 3m 46.3M 4972M 18.2.0 dc2bc1663786 584fb2e408cd 2024-04-27T02:05:41.309 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi052 running (3m) 65s ago 3m 43.1M 4972M 18.2.0 dc2bc1663786 26fdd5746d2a 2024-04-27T02:05:41.309 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi052 running (2m) 65s ago 2m 43.7M 4972M 18.2.0 dc2bc1663786 6fa7665653a4 2024-04-27T02:05:41.309 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.smithi029 smithi029 *:9095 running (5m) 60s ago 6m 64.6M - 2.43.0 a07b618ecd1d a9ebc9badf18 2024-04-27T02:05:41.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.326+0000 7f36f6ffd700 1 -- 172.21.15.29:0/626968456 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3480 (secure 0 0 0) 0x7f36e4000bf0 con 0x7f36ec06c6d0 2024-04-27T02:05:41.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.328+0000 7f3707ddd700 1 -- 172.21.15.29:0/626968456 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f36ec06c6d0 msgr2=0x7f36ec06eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:41.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.329+0000 7f3707ddd700 1 --2- 172.21.15.29:0/626968456 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f36ec06c6d0 0x7f36ec06eb80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3700105e80 tx=0x7f36f0009310 comp rx=0 tx=0).stop 2024-04-27T02:05:41.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.329+0000 7f3707ddd700 1 -- 172.21.15.29:0/626968456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37000fe8c0 msgr2=0x7f3700105900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:41.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.329+0000 7f3707ddd700 1 --2- 172.21.15.29:0/626968456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37000fe8c0 0x7f3700105900 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f36fc009230 tx=0x7f36fc00f690 comp rx=0 tx=0).stop 2024-04-27T02:05:41.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.329+0000 7f3707ddd700 1 -- 172.21.15.29:0/626968456 shutdown_connections 2024-04-27T02:05:41.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.329+0000 7f3707ddd700 1 --2- 172.21.15.29:0/626968456 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f36ec06c6d0 0x7f36ec06eb80 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:41.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.329+0000 7f3707ddd700 1 --2- 172.21.15.29:0/626968456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37000fe8c0 0x7f3700105900 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:41.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.329+0000 7f3707ddd700 1 --2- 172.21.15.29:0/626968456 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f37000ffb60 0x7f370019cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:41.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.329+0000 7f3707ddd700 1 -- 172.21.15.29:0/626968456 >> 172.21.15.29:0/626968456 conn(0x7f37000f9f10 msgr2=0x7f3700102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:41.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.329+0000 7f3707ddd700 1 -- 172.21.15.29:0/626968456 shutdown_connections 2024-04-27T02:05:41.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.329+0000 7f3707ddd700 1 -- 172.21.15.29:0/626968456 wait complete. 2024-04-27T02:05:41.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.441+0000 7f8da0ba6700 1 -- 172.21.15.29:0/646804695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d9c100d60 msgr2=0x7f8d9c103140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:41.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.441+0000 7f8da0ba6700 1 --2- 172.21.15.29:0/646804695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d9c100d60 0x7f8d9c103140 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c009910 tx=0x7f8d8c009c20 comp rx=0 tx=0).stop 2024-04-27T02:05:41.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.442+0000 7f8da0ba6700 1 -- 172.21.15.29:0/646804695 shutdown_connections 2024-04-27T02:05:41.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.442+0000 7f8da0ba6700 1 --2- 172.21.15.29:0/646804695 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d9c103680 0x7f8d9c105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:41.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.442+0000 7f8da0ba6700 1 --2- 172.21.15.29:0/646804695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d9c100d60 0x7f8d9c103140 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:41.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.442+0000 7f8da0ba6700 1 -- 172.21.15.29:0/646804695 >> 172.21.15.29:0/646804695 conn(0x7f8d9c0fa780 msgr2=0x7f8d9c0fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:41.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.442+0000 7f8da0ba6700 1 -- 172.21.15.29:0/646804695 shutdown_connections 2024-04-27T02:05:41.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.442+0000 7f8da0ba6700 1 -- 172.21.15.29:0/646804695 wait complete. 2024-04-27T02:05:41.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.443+0000 7f8da0ba6700 1 Processor -- start 2024-04-27T02:05:41.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.443+0000 7f8da0ba6700 1 -- start start 2024-04-27T02:05:41.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.443+0000 7f8da0ba6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d9c100d60 0x7f8d9c1006c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:41.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.443+0000 7f8da0ba6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d9c103680 0x7f8d9c0fed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:41.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.443+0000 7f8da0ba6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c199700 con 0x7f8d9c100d60 2024-04-27T02:05:41.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.443+0000 7f8da0ba6700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c199840 con 0x7f8d9c103680 2024-04-27T02:05:41.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.443+0000 7f8d99d9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d9c103680 0x7f8d9c0fed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:41.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.443+0000 7f8d9a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d9c100d60 0x7f8d9c1006c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:41.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.443+0000 7f8d99d9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d9c103680 0x7f8d9c0fed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:37060/0 (socket says 172.21.15.29:37060) 2024-04-27T02:05:41.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.443+0000 7f8d99d9b700 1 -- 172.21.15.29:0/3545660128 learned_addr learned my addr 172.21.15.29:0/3545660128 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:05:41.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.444+0000 7f8d99d9b700 1 -- 172.21.15.29:0/3545660128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d9c100d60 msgr2=0x7f8d9c1006c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:41.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.444+0000 7f8d99d9b700 1 --2- 172.21.15.29:0/3545660128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d9c100d60 0x7f8d9c1006c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:41.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.444+0000 7f8d99d9b700 1 -- 172.21.15.29:0/3545660128 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8c0095f0 con 0x7f8d9c103680 2024-04-27T02:05:41.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.444+0000 7f8d9a59c700 1 --2- 172.21.15.29:0/3545660128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d9c100d60 0x7f8d9c1006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T02:05:41.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.444+0000 7f8d99d9b700 1 --2- 172.21.15.29:0/3545660128 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d9c103680 0x7f8d9c0fed10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8d9000e920 tx=0x7f8d9000ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:41.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.444+0000 7f8d8b7fe700 1 -- 172.21.15.29:0/3545660128 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d9000c9d0 con 0x7f8d9c103680 2024-04-27T02:05:41.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.445+0000 7f8d8b7fe700 1 -- 172.21.15.29:0/3545660128 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8d90010430 con 0x7f8d9c103680 2024-04-27T02:05:41.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.445+0000 7f8da0ba6700 1 -- 172.21.15.29:0/3545660128 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9c0ff250 con 0x7f8d9c103680 2024-04-27T02:05:41.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.445+0000 7f8d8b7fe700 1 -- 172.21.15.29:0/3545660128 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d90018650 con 0x7f8d9c103680 2024-04-27T02:05:41.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.445+0000 7f8da0ba6700 1 -- 172.21.15.29:0/3545660128 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d9c0ff770 con 0x7f8d9c103680 2024-04-27T02:05:41.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.446+0000 7f8da0ba6700 1 -- 172.21.15.29:0/3545660128 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d9c18fbc0 con 0x7f8d9c103680 2024-04-27T02:05:41.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.446+0000 7f8d8b7fe700 1 -- 172.21.15.29:0/3545660128 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f8d900105a0 con 0x7f8d9c103680 2024-04-27T02:05:41.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.447+0000 7f8d8b7fe700 1 --2- 172.21.15.29:0/3545660128 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f8d8406c680 0x7f8d8406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:41.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.447+0000 7f8d8b7fe700 1 -- 172.21.15.29:0/3545660128 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f8d90014070 con 0x7f8d9c103680 2024-04-27T02:05:41.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.447+0000 7f8d9a59c700 1 --2- 172.21.15.29:0/3545660128 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f8d8406c680 0x7f8d8406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:41.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.448+0000 7f8d9a59c700 1 --2- 172.21.15.29:0/3545660128 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f8d8406c680 0x7f8d8406eb30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00b6f0 tx=0x7f8d8c005940 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:41.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:41.455+0000 7f8d8b7fe700 1 -- 172.21.15.29:0/3545660128 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8d90051720 con 0x7f8d9c103680 2024-04-27T02:05:42.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.023+0000 7f8da0ba6700 1 -- 172.21.15.29:0/3545660128 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8d9c0ff950 con 0x7f8d9c103680 2024-04-27T02:05:42.005 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-04-27T02:05:42.005 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-04-27T02:05:42.005 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-04-27T02:05:42.005 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-27T02:05:42.006 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-04-27T02:05:42.006 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-04-27T02:05:42.006 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-27T02:05:42.006 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-04-27T02:05:42.006 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-04-27T02:05:42.006 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-27T02:05:42.006 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": { 2024-04-27T02:05:42.006 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-27T02:05:42.006 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-27T02:05:42.006 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-04-27T02:05:42.006 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 14 2024-04-27T02:05:42.006 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-27T02:05:42.006 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-04-27T02:05:42.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.024+0000 7f8d8b7fe700 1 -- 172.21.15.29:0/3545660128 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7f8d9008be60 con 0x7f8d9c103680 2024-04-27T02:05:42.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.027+0000 7f8da0ba6700 1 -- 172.21.15.29:0/3545660128 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f8d8406c680 msgr2=0x7f8d8406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:42.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.027+0000 7f8da0ba6700 1 --2- 172.21.15.29:0/3545660128 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f8d8406c680 0x7f8d8406eb30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00b6f0 tx=0x7f8d8c005940 comp rx=0 tx=0).stop 2024-04-27T02:05:42.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.027+0000 7f8da0ba6700 1 -- 172.21.15.29:0/3545660128 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d9c103680 msgr2=0x7f8d9c0fed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:42.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.027+0000 7f8da0ba6700 1 --2- 172.21.15.29:0/3545660128 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d9c103680 0x7f8d9c0fed10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8d9000e920 tx=0x7f8d9000ec30 comp rx=0 tx=0).stop 2024-04-27T02:05:42.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.027+0000 7f8da0ba6700 1 -- 172.21.15.29:0/3545660128 shutdown_connections 2024-04-27T02:05:42.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.027+0000 7f8da0ba6700 1 --2- 172.21.15.29:0/3545660128 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f8d8406c680 0x7f8d8406eb30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:42.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.027+0000 7f8da0ba6700 1 --2- 172.21.15.29:0/3545660128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d9c100d60 0x7f8d9c1006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:42.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.027+0000 7f8da0ba6700 1 --2- 172.21.15.29:0/3545660128 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d9c103680 0x7f8d9c0fed10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:42.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.027+0000 7f8da0ba6700 1 -- 172.21.15.29:0/3545660128 >> 172.21.15.29:0/3545660128 conn(0x7f8d9c0fa780 msgr2=0x7f8d9c0fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:42.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.028+0000 7f8da0ba6700 1 -- 172.21.15.29:0/3545660128 shutdown_connections 2024-04-27T02:05:42.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.028+0000 7f8da0ba6700 1 -- 172.21.15.29:0/3545660128 wait complete. 2024-04-27T02:05:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.196+0000 7f08d6efe700 1 -- 172.21.15.29:0/1189487887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08d00ff130 msgr2=0x7f08d00ff540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.196+0000 7f08d6efe700 1 --2- 172.21.15.29:0/1189487887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08d00ff130 0x7f08d00ff540 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f08c4009910 tx=0x7f08c4009c20 comp rx=0 tx=0).stop 2024-04-27T02:05:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.197+0000 7f08d6efe700 1 -- 172.21.15.29:0/1189487887 shutdown_connections 2024-04-27T02:05:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.197+0000 7f08d6efe700 1 --2- 172.21.15.29:0/1189487887 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f08d01003d0 0x7f08d0100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.197+0000 7f08d6efe700 1 --2- 172.21.15.29:0/1189487887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08d00ff130 0x7f08d00ff540 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.197+0000 7f08d6efe700 1 -- 172.21.15.29:0/1189487887 >> 172.21.15.29:0/1189487887 conn(0x7f08d00fa780 msgr2=0x7f08d00fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.197+0000 7f08d6efe700 1 -- 172.21.15.29:0/1189487887 shutdown_connections 2024-04-27T02:05:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.197+0000 7f08d6efe700 1 -- 172.21.15.29:0/1189487887 wait complete. 2024-04-27T02:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.198+0000 7f08d6efe700 1 Processor -- start 2024-04-27T02:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.198+0000 7f08d6efe700 1 -- start start 2024-04-27T02:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.198+0000 7f08d6efe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f08d00ff130 0x7f08d01949b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.198+0000 7f08d6efe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08d01003d0 0x7f08d0194ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.198+0000 7f08d6efe700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d0195510 con 0x7f08d01003d0 2024-04-27T02:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.198+0000 7f08d6efe700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d0195650 con 0x7f08d00ff130 2024-04-27T02:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.198+0000 7f08cffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08d01003d0 0x7f08d0194ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.198+0000 7f08cffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08d01003d0 0x7f08d0194ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49724/0 (socket says 172.21.15.29:49724) 2024-04-27T02:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.198+0000 7f08cffff700 1 -- 172.21.15.29:0/1797246914 learned_addr learned my addr 172.21.15.29:0/1797246914 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.198+0000 7f08d4c9a700 1 --2- 172.21.15.29:0/1797246914 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f08d00ff130 0x7f08d01949b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.199+0000 7f08cffff700 1 -- 172.21.15.29:0/1797246914 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f08d00ff130 msgr2=0x7f08d01949b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:42.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.199+0000 7f08cffff700 1 --2- 172.21.15.29:0/1797246914 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f08d00ff130 0x7f08d01949b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:42.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.199+0000 7f08cffff700 1 -- 172.21.15.29:0/1797246914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c40095f0 con 0x7f08d01003d0 2024-04-27T02:05:42.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.199+0000 7f08d4c9a700 1 --2- 172.21.15.29:0/1797246914 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f08d00ff130 0x7f08d01949b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T02:05:42.182 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:41 smithi029 ceph-mon[26769]: from='client.24401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:05:42.182 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:41 smithi029 ceph-mon[26769]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:05:42.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.199+0000 7f08cffff700 1 --2- 172.21.15.29:0/1797246914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08d01003d0 0x7f08d0194ef0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f08bc00e920 tx=0x7f08bc00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:42.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.199+0000 7f08cdffb700 1 -- 172.21.15.29:0/1797246914 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08bc00cba0 con 0x7f08d01003d0 2024-04-27T02:05:42.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.199+0000 7f08cdffb700 1 -- 172.21.15.29:0/1797246914 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f08bc00cd00 con 0x7f08d01003d0 2024-04-27T02:05:42.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.199+0000 7f08d6efe700 1 -- 172.21.15.29:0/1797246914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d00686e0 con 0x7f08d01003d0 2024-04-27T02:05:42.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.199+0000 7f08cdffb700 1 -- 172.21.15.29:0/1797246914 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08bc018770 con 0x7f08d01003d0 2024-04-27T02:05:42.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.200+0000 7f08d6efe700 1 -- 172.21.15.29:0/1797246914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d0068b70 con 0x7f08d01003d0 2024-04-27T02:05:42.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.201+0000 7f08d6efe700 1 -- 172.21.15.29:0/1797246914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d0107c10 con 0x7f08d01003d0 2024-04-27T02:05:42.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.202+0000 7f08cdffb700 1 -- 172.21.15.29:0/1797246914 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f08bc010430 con 0x7f08d01003d0 2024-04-27T02:05:42.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.202+0000 7f08cdffb700 1 --2- 172.21.15.29:0/1797246914 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f08c006c6d0 0x7f08c006eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:42.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.202+0000 7f08cdffb700 1 -- 172.21.15.29:0/1797246914 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f08bc014070 con 0x7f08d01003d0 2024-04-27T02:05:42.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.203+0000 7f08d4c9a700 1 --2- 172.21.15.29:0/1797246914 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f08c006c6d0 0x7f08c006eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:42.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.203+0000 7f08d4c9a700 1 --2- 172.21.15.29:0/1797246914 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f08c006c6d0 0x7f08c006eb80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f08c4000c00 tx=0x7f08c4009f70 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:42.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.211+0000 7f08cdffb700 1 -- 172.21.15.29:0/1797246914 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f08bc051720 con 0x7f08d01003d0 2024-04-27T02:05:42.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:41 smithi052 ceph-mon[32586]: from='client.24401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:05:42.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:41 smithi052 ceph-mon[32586]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:05:42.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.691+0000 7f08d6efe700 1 -- 172.21.15.29:0/1797246914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f08d0107e00 con 0x7f08d01003d0 2024-04-27T02:05:42.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.691+0000 7f08cdffb700 1 -- 172.21.15.29:0/1797246914 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 19 v19) v1 ==== 76+0+1851 (secure 0 0 0) 0x7f08bc059e60 con 0x7f08d01003d0 2024-04-27T02:05:42.672 INFO:teuthology.orchestra.run.smithi029.stdout:e19 2024-04-27T02:05:42.673 INFO:teuthology.orchestra.run.smithi029.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T02:05:42.673 INFO:teuthology.orchestra.run.smithi029.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T02:05:42.673 INFO:teuthology.orchestra.run.smithi029.stdout:legacy client fscid: 1 2024-04-27T02:05:42.673 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:05:42.673 INFO:teuthology.orchestra.run.smithi029.stdout:Filesystem 'cephfs' (1) 2024-04-27T02:05:42.673 INFO:teuthology.orchestra.run.smithi029.stdout:fs_name cephfs 2024-04-27T02:05:42.673 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 19 2024-04-27T02:05:42.673 INFO:teuthology.orchestra.run.smithi029.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-27T02:05:42.673 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-04-27T02:04:10.035536+0000 2024-04-27T02:05:42.673 INFO:teuthology.orchestra.run.smithi029.stdout:modified 2024-04-27T02:04:26.892992+0000 2024-04-27T02:05:42.673 INFO:teuthology.orchestra.run.smithi029.stdout:tableserver 0 2024-04-27T02:05:42.673 INFO:teuthology.orchestra.run.smithi029.stdout:root 0 2024-04-27T02:05:42.673 INFO:teuthology.orchestra.run.smithi029.stdout:session_timeout 60 2024-04-27T02:05:42.673 INFO:teuthology.orchestra.run.smithi029.stdout:session_autoclose 300 2024-04-27T02:05:42.673 INFO:teuthology.orchestra.run.smithi029.stdout:max_file_size 1099511627776 2024-04-27T02:05:42.674 INFO:teuthology.orchestra.run.smithi029.stdout:required_client_features {} 2024-04-27T02:05:42.674 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure 0 2024-04-27T02:05:42.674 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure_osd_epoch 46 2024-04-27T02:05:42.674 INFO:teuthology.orchestra.run.smithi029.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T02:05:42.674 INFO:teuthology.orchestra.run.smithi029.stdout:max_mds 1 2024-04-27T02:05:42.674 INFO:teuthology.orchestra.run.smithi029.stdout:in 0 2024-04-27T02:05:42.674 INFO:teuthology.orchestra.run.smithi029.stdout:up {0=14484} 2024-04-27T02:05:42.674 INFO:teuthology.orchestra.run.smithi029.stdout:failed 2024-04-27T02:05:42.674 INFO:teuthology.orchestra.run.smithi029.stdout:damaged 2024-04-27T02:05:42.674 INFO:teuthology.orchestra.run.smithi029.stdout:stopped 2024-04-27T02:05:42.674 INFO:teuthology.orchestra.run.smithi029.stdout:data_pools [3] 2024-04-27T02:05:42.674 INFO:teuthology.orchestra.run.smithi029.stdout:metadata_pool 2 2024-04-27T02:05:42.674 INFO:teuthology.orchestra.run.smithi029.stdout:inline_data enabled 2024-04-27T02:05:42.674 INFO:teuthology.orchestra.run.smithi029.stdout:balancer 2024-04-27T02:05:42.674 INFO:teuthology.orchestra.run.smithi029.stdout:bal_rank_mask -1 2024-04-27T02:05:42.675 INFO:teuthology.orchestra.run.smithi029.stdout:standby_count_wanted 1 2024-04-27T02:05:42.675 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi029.njjezc{0:14484} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.29:6828/1353923457,v1:172.21.15.29:6829/1353923457] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T02:05:42.675 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi052.yylmkb{0:14494} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.52:6824/2352841895,v1:172.21.15.52:6825/2352841895] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T02:05:42.675 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:05:42.675 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:05:42.675 INFO:teuthology.orchestra.run.smithi029.stdout:Standby daemons: 2024-04-27T02:05:42.675 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:05:42.675 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi029.dyrjrd{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.29:6826/3428808494,v1:172.21.15.29:6827/3428808494] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T02:05:42.675 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi052.nbjpjd{-1:24325} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.52:6826/3513632158,v1:172.21.15.52:6827/3513632158] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T02:05:42.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.694+0000 7f08d6efe700 1 -- 172.21.15.29:0/1797246914 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f08c006c6d0 msgr2=0x7f08c006eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:42.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.694+0000 7f08d6efe700 1 --2- 172.21.15.29:0/1797246914 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f08c006c6d0 0x7f08c006eb80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f08c4000c00 tx=0x7f08c4009f70 comp rx=0 tx=0).stop 2024-04-27T02:05:42.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.694+0000 7f08d6efe700 1 -- 172.21.15.29:0/1797246914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08d01003d0 msgr2=0x7f08d0194ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:42.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.694+0000 7f08d6efe700 1 --2- 172.21.15.29:0/1797246914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08d01003d0 0x7f08d0194ef0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f08bc00e920 tx=0x7f08bc00ece0 comp rx=0 tx=0).stop 2024-04-27T02:05:42.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.694+0000 7f08d6efe700 1 -- 172.21.15.29:0/1797246914 shutdown_connections 2024-04-27T02:05:42.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.694+0000 7f08d6efe700 1 --2- 172.21.15.29:0/1797246914 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f08c006c6d0 0x7f08c006eb80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:42.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.694+0000 7f08d6efe700 1 --2- 172.21.15.29:0/1797246914 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f08d00ff130 0x7f08d01949b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:42.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.694+0000 7f08d6efe700 1 --2- 172.21.15.29:0/1797246914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08d01003d0 0x7f08d0194ef0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:42.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.694+0000 7f08d6efe700 1 -- 172.21.15.29:0/1797246914 >> 172.21.15.29:0/1797246914 conn(0x7f08d00fa780 msgr2=0x7f08d0103600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:42.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.694+0000 7f08d6efe700 1 -- 172.21.15.29:0/1797246914 shutdown_connections 2024-04-27T02:05:42.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.694+0000 7f08d6efe700 1 -- 172.21.15.29:0/1797246914 wait complete. 2024-04-27T02:05:42.677 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 19 2024-04-27T02:05:42.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.811+0000 7fb96c1c2700 1 -- 172.21.15.29:0/2523634177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb964100350 msgr2=0x7fb9641007a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:42.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.811+0000 7fb96c1c2700 1 --2- 172.21.15.29:0/2523634177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb964100350 0x7fb9641007a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb954009910 tx=0x7fb954009c20 comp rx=0 tx=0).stop 2024-04-27T02:05:42.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.811+0000 7fb96c1c2700 1 -- 172.21.15.29:0/2523634177 shutdown_connections 2024-04-27T02:05:42.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.811+0000 7fb96c1c2700 1 --2- 172.21.15.29:0/2523634177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb964100350 0x7fb9641007a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:42.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.811+0000 7fb96c1c2700 1 --2- 172.21.15.29:0/2523634177 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9640ff150 0x7fb9640ff560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:42.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.811+0000 7fb96c1c2700 1 -- 172.21.15.29:0/2523634177 >> 172.21.15.29:0/2523634177 conn(0x7fb9640fa700 msgr2=0x7fb9640fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:42.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.812+0000 7fb96c1c2700 1 -- 172.21.15.29:0/2523634177 shutdown_connections 2024-04-27T02:05:42.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.812+0000 7fb96c1c2700 1 -- 172.21.15.29:0/2523634177 wait complete. 2024-04-27T02:05:42.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.813+0000 7fb96c1c2700 1 Processor -- start 2024-04-27T02:05:42.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.813+0000 7fb96c1c2700 1 -- start start 2024-04-27T02:05:42.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.813+0000 7fb96c1c2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9640ff150 0x7fb964195d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:42.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.813+0000 7fb96c1c2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb964196260 0x7fb96419b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:42.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.813+0000 7fb96c1c2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9641966d0 con 0x7fb964196260 2024-04-27T02:05:42.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.813+0000 7fb96c1c2700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb964196840 con 0x7fb9640ff150 2024-04-27T02:05:42.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.813+0000 7fb969f5e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9640ff150 0x7fb964195d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:42.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.813+0000 7fb96975d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb964196260 0x7fb96419b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:42.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.813+0000 7fb969f5e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9640ff150 0x7fb964195d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:37100/0 (socket says 172.21.15.29:37100) 2024-04-27T02:05:42.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.813+0000 7fb969f5e700 1 -- 172.21.15.29:0/793795717 learned_addr learned my addr 172.21.15.29:0/793795717 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:05:42.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.814+0000 7fb96975d700 1 -- 172.21.15.29:0/793795717 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9640ff150 msgr2=0x7fb964195d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:42.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.814+0000 7fb96975d700 1 --2- 172.21.15.29:0/793795717 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9640ff150 0x7fb964195d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:42.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.814+0000 7fb96975d700 1 -- 172.21.15.29:0/793795717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9540095f0 con 0x7fb964196260 2024-04-27T02:05:42.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.814+0000 7fb969f5e700 1 --2- 172.21.15.29:0/793795717 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9640ff150 0x7fb964195d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T02:05:42.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.814+0000 7fb96975d700 1 --2- 172.21.15.29:0/793795717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb964196260 0x7fb96419b290 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb954008000 tx=0x7fb95400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:42.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.814+0000 7fb95affd700 1 -- 172.21.15.29:0/793795717 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb95401d070 con 0x7fb964196260 2024-04-27T02:05:42.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.814+0000 7fb95affd700 1 -- 172.21.15.29:0/793795717 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb954005dc0 con 0x7fb964196260 2024-04-27T02:05:42.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.814+0000 7fb96c1c2700 1 -- 172.21.15.29:0/793795717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb96419b7d0 con 0x7fb964196260 2024-04-27T02:05:42.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.815+0000 7fb95affd700 1 -- 172.21.15.29:0/793795717 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb954021410 con 0x7fb964196260 2024-04-27T02:05:42.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.815+0000 7fb96c1c2700 1 -- 172.21.15.29:0/793795717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb96419bbd0 con 0x7fb964196260 2024-04-27T02:05:42.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.816+0000 7fb96c1c2700 1 -- 172.21.15.29:0/793795717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb964108da0 con 0x7fb964196260 2024-04-27T02:05:42.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.817+0000 7fb95affd700 1 -- 172.21.15.29:0/793795717 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fb95400f460 con 0x7fb964196260 2024-04-27T02:05:42.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.817+0000 7fb95affd700 1 --2- 172.21.15.29:0/793795717 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb95006c470 0x7fb95006e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:42.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.817+0000 7fb95affd700 1 -- 172.21.15.29:0/793795717 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fb95408c500 con 0x7fb964196260 2024-04-27T02:05:42.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.818+0000 7fb969f5e700 1 --2- 172.21.15.29:0/793795717 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb95006c470 0x7fb95006e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:42.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.818+0000 7fb969f5e700 1 --2- 172.21.15.29:0/793795717 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb95006c470 0x7fb95006e920 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb964197200 tx=0x7fb960007480 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:42.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:42.826+0000 7fb95affd700 1 -- 172.21.15.29:0/793795717 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb954056c00 con 0x7fb964196260 2024-04-27T02:05:43.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.130+0000 7fb96c1c2700 1 -- 172.21.15.29:0/793795717 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb964104ca0 con 0x7fb95006c470 2024-04-27T02:05:43.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.132+0000 7fb95affd700 1 -- 172.21.15.29:0/793795717 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7fb964104ca0 con 0x7fb95006c470 2024-04-27T02:05:43.113 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-04-27T02:05:43.113 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a", 2024-04-27T02:05:43.113 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-04-27T02:05:43.113 INFO:teuthology.orchestra.run.smithi029.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:05:43.113 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [], 2024-04-27T02:05:43.113 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "", 2024-04-27T02:05:43.113 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a image", 2024-04-27T02:05:43.114 INFO:teuthology.orchestra.run.smithi029.stdout: "is_paused": false 2024-04-27T02:05:43.114 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-04-27T02:05:43.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.134+0000 7fb96c1c2700 1 -- 172.21.15.29:0/793795717 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb95006c470 msgr2=0x7fb95006e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:43.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.134+0000 7fb96c1c2700 1 --2- 172.21.15.29:0/793795717 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb95006c470 0x7fb95006e920 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb964197200 tx=0x7fb960007480 comp rx=0 tx=0).stop 2024-04-27T02:05:43.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.134+0000 7fb96c1c2700 1 -- 172.21.15.29:0/793795717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb964196260 msgr2=0x7fb96419b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:43.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.134+0000 7fb96c1c2700 1 --2- 172.21.15.29:0/793795717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb964196260 0x7fb96419b290 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb954008000 tx=0x7fb95400bd70 comp rx=0 tx=0).stop 2024-04-27T02:05:43.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.135+0000 7fb96c1c2700 1 -- 172.21.15.29:0/793795717 shutdown_connections 2024-04-27T02:05:43.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.135+0000 7fb96c1c2700 1 --2- 172.21.15.29:0/793795717 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fb95006c470 0x7fb95006e920 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:43.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.135+0000 7fb96c1c2700 1 --2- 172.21.15.29:0/793795717 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb9640ff150 0x7fb964195d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:43.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.135+0000 7fb96c1c2700 1 --2- 172.21.15.29:0/793795717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb964196260 0x7fb96419b290 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:43.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.135+0000 7fb96c1c2700 1 -- 172.21.15.29:0/793795717 >> 172.21.15.29:0/793795717 conn(0x7fb9640fa700 msgr2=0x7fb964103580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:43.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.135+0000 7fb96c1c2700 1 -- 172.21.15.29:0/793795717 shutdown_connections 2024-04-27T02:05:43.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.135+0000 7fb96c1c2700 1 -- 172.21.15.29:0/793795717 wait complete. 2024-04-27T02:05:43.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:42 smithi029 ceph-mon[26769]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:05:43.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:42 smithi029 ceph-mon[26769]: pgmap v231: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 2 op/s 2024-04-27T02:05:43.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:42 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/3545660128' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:05:43.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:42 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/1797246914' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T02:05:43.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:42 smithi052 ceph-mon[32586]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:05:43.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:42 smithi052 ceph-mon[32586]: pgmap v231: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 2 op/s 2024-04-27T02:05:43.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:42 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/3545660128' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:05:43.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:42 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/1797246914' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T02:05:43.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.281+0000 7f183a844700 1 -- 172.21.15.29:0/1269688985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1834103680 msgr2=0x7f1834105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:43.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.281+0000 7f183a844700 1 --2- 172.21.15.29:0/1269688985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1834103680 0x7f1834105a60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1828009910 tx=0x7f1828009c20 comp rx=0 tx=0).stop 2024-04-27T02:05:43.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.281+0000 7f183a844700 1 -- 172.21.15.29:0/1269688985 shutdown_connections 2024-04-27T02:05:43.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.281+0000 7f183a844700 1 --2- 172.21.15.29:0/1269688985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1834103680 0x7f1834105a60 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:43.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.281+0000 7f183a844700 1 --2- 172.21.15.29:0/1269688985 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1834100d60 0x7f1834103140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:43.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.281+0000 7f183a844700 1 -- 172.21.15.29:0/1269688985 >> 172.21.15.29:0/1269688985 conn(0x7f18340fa780 msgr2=0x7f18340fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:43.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.281+0000 7f183a844700 1 -- 172.21.15.29:0/1269688985 shutdown_connections 2024-04-27T02:05:43.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.281+0000 7f183a844700 1 -- 172.21.15.29:0/1269688985 wait complete. 2024-04-27T02:05:43.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.282+0000 7f183a844700 1 Processor -- start 2024-04-27T02:05:43.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.282+0000 7f183a844700 1 -- start start 2024-04-27T02:05:43.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.283+0000 7f183a844700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1834100d60 0x7f1834197d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:43.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.283+0000 7f183a844700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1834103680 0x7f1834198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:43.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.283+0000 7f183a844700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1834198870 con 0x7f1834103680 2024-04-27T02:05:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.283+0000 7f183a844700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183419d280 con 0x7f1834100d60 2024-04-27T02:05:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.283+0000 7f1833fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1834100d60 0x7f1834197d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.283+0000 7f18337fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1834103680 0x7f1834198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.283+0000 7f1833fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1834100d60 0x7f1834197d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:37122/0 (socket says 172.21.15.29:37122) 2024-04-27T02:05:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.283+0000 7f1833fff700 1 -- 172.21.15.29:0/2715127750 learned_addr learned my addr 172.21.15.29:0/2715127750 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:05:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.283+0000 7f1833fff700 1 -- 172.21.15.29:0/2715127750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1834103680 msgr2=0x7f1834198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.283+0000 7f1833fff700 1 --2- 172.21.15.29:0/2715127750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1834103680 0x7f1834198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.283+0000 7f1833fff700 1 -- 172.21.15.29:0/2715127750 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18280095f0 con 0x7f1834100d60 2024-04-27T02:05:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.284+0000 7f1833fff700 1 --2- 172.21.15.29:0/2715127750 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1834100d60 0x7f1834197d10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f182400e8f0 tx=0x7f182400ec00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.284+0000 7f18317fa700 1 -- 172.21.15.29:0/2715127750 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1824009790 con 0x7f1834100d60 2024-04-27T02:05:43.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.284+0000 7f183a844700 1 -- 172.21.15.29:0/2715127750 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f183419d480 con 0x7f1834100d60 2024-04-27T02:05:43.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.284+0000 7f18317fa700 1 -- 172.21.15.29:0/2715127750 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f182400c810 con 0x7f1834100d60 2024-04-27T02:05:43.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.284+0000 7f18317fa700 1 -- 172.21.15.29:0/2715127750 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18240055b0 con 0x7f1834100d60 2024-04-27T02:05:43.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.284+0000 7f183a844700 1 -- 172.21.15.29:0/2715127750 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f183419d9a0 con 0x7f1834100d60 2024-04-27T02:05:43.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.285+0000 7f183a844700 1 -- 172.21.15.29:0/2715127750 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18340ffb70 con 0x7f1834100d60 2024-04-27T02:05:43.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.286+0000 7f18317fa700 1 -- 172.21.15.29:0/2715127750 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f182400c980 con 0x7f1834100d60 2024-04-27T02:05:43.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.286+0000 7f18317fa700 1 --2- 172.21.15.29:0/2715127750 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f181c06c680 0x7f181c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:05:43.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.286+0000 7f18317fa700 1 -- 172.21.15.29:0/2715127750 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f1824014070 con 0x7f1834100d60 2024-04-27T02:05:43.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.286+0000 7f18337fe700 1 --2- 172.21.15.29:0/2715127750 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f181c06c680 0x7f181c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:05:43.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.287+0000 7f18337fe700 1 --2- 172.21.15.29:0/2715127750 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f181c06c680 0x7f181c06eb30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1828009fd0 tx=0x7f1828005940 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:05:43.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.295+0000 7f18317fa700 1 -- 172.21.15.29:0/2715127750 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f18240562f0 con 0x7f1834100d60 2024-04-27T02:05:43.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.589+0000 7f183a844700 1 -- 172.21.15.29:0/2715127750 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f183419e060 con 0x7f1834100d60 2024-04-27T02:05:43.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.589+0000 7f18317fa700 1 -- 172.21.15.29:0/2715127750 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f1824059910 con 0x7f1834100d60 2024-04-27T02:05:43.571 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-27T02:05:43.571 INFO:teuthology.orchestra.run.smithi029.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-27T02:05:43.571 INFO:teuthology.orchestra.run.smithi029.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-27T02:05:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.592+0000 7f183a844700 1 -- 172.21.15.29:0/2715127750 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f181c06c680 msgr2=0x7f181c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.592+0000 7f183a844700 1 --2- 172.21.15.29:0/2715127750 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f181c06c680 0x7f181c06eb30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1828009fd0 tx=0x7f1828005940 comp rx=0 tx=0).stop 2024-04-27T02:05:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.592+0000 7f183a844700 1 -- 172.21.15.29:0/2715127750 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1834100d60 msgr2=0x7f1834197d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:05:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.592+0000 7f183a844700 1 --2- 172.21.15.29:0/2715127750 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1834100d60 0x7f1834197d10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f182400e8f0 tx=0x7f182400ec00 comp rx=0 tx=0).stop 2024-04-27T02:05:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.592+0000 7f183a844700 1 -- 172.21.15.29:0/2715127750 shutdown_connections 2024-04-27T02:05:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.592+0000 7f183a844700 1 --2- 172.21.15.29:0/2715127750 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f181c06c680 0x7f181c06eb30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.592+0000 7f183a844700 1 --2- 172.21.15.29:0/2715127750 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1834100d60 0x7f1834197d10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.592+0000 7f183a844700 1 --2- 172.21.15.29:0/2715127750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1834103680 0x7f1834198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:05:43.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.592+0000 7f183a844700 1 -- 172.21.15.29:0/2715127750 >> 172.21.15.29:0/2715127750 conn(0x7f18340fa780 msgr2=0x7f18340fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:05:43.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.592+0000 7f183a844700 1 -- 172.21.15.29:0/2715127750 shutdown_connections 2024-04-27T02:05:43.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:05:43.592+0000 7f183a844700 1 -- 172.21.15.29:0/2715127750 wait complete. 2024-04-27T02:05:44.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:43 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/2715127750' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:05:44.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:43 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/2715127750' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:05:45.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:44 smithi029 ceph-mon[26769]: from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:05:45.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:44 smithi029 ceph-mon[26769]: pgmap v232: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-04-27T02:05:45.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:44 smithi052 ceph-mon[32586]: from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:05:45.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:44 smithi052 ceph-mon[32586]: pgmap v232: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-04-27T02:05:47.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:46 smithi029 ceph-mon[26769]: pgmap v233: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:05:47.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:46 smithi052 ceph-mon[32586]: pgmap v233: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:05:49.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:48 smithi029 ceph-mon[26769]: pgmap v234: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-27T02:05:49.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:48 smithi052 ceph-mon[32586]: pgmap v234: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-27T02:05:51.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:50 smithi029 ceph-mon[26769]: pgmap v235: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:05:51.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:50 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:05:51.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:50 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:05:51.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:50 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:05:51.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:50 smithi052 ceph-mon[32586]: pgmap v235: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:05:51.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:50 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:05:51.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:50 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:05:51.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:50 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' 2024-04-27T02:05:52.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:51 smithi029 ceph-mon[26769]: Upgrade: Target is version 19.0.0-2447-gb22e2ebd (unknown) 2024-04-27T02:05:52.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:51 smithi029 ceph-mon[26769]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, digests ['quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6'] 2024-04-27T02:05:52.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:51 smithi029 ceph-mon[26769]: Upgrade: Need to upgrade myself (mgr.smithi029.ewvdzs) 2024-04-27T02:05:52.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:51 smithi029 ceph-mon[26769]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 on smithi052 2024-04-27T02:05:52.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:51 smithi052 ceph-mon[32586]: Upgrade: Target is version 19.0.0-2447-gb22e2ebd (unknown) 2024-04-27T02:05:52.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:51 smithi052 ceph-mon[32586]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, digests ['quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6'] 2024-04-27T02:05:52.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:51 smithi052 ceph-mon[32586]: Upgrade: Need to upgrade myself (mgr.smithi029.ewvdzs) 2024-04-27T02:05:52.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:51 smithi052 ceph-mon[32586]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 on smithi052 2024-04-27T02:05:53.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:52 smithi029 ceph-mon[26769]: pgmap v236: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-27T02:05:53.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:52 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:05:53.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:52 smithi052 ceph-mon[32586]: pgmap v236: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-27T02:05:53.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:52 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:05:55.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:54 smithi052 ceph-mon[32586]: pgmap v237: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:05:55.450 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:54 smithi029 ceph-mon[26769]: pgmap v237: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:05:57.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:56 smithi052 ceph-mon[32586]: pgmap v238: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:05:57.450 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:56 smithi029 ceph-mon[26769]: pgmap v238: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:05:59.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:05:58 smithi052 ceph-mon[32586]: pgmap v239: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-27T02:05:59.450 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:05:58 smithi029 ceph-mon[26769]: pgmap v239: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-27T02:06:01.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:06:00 smithi052 ceph-mon[32586]: pgmap v240: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:06:01.450 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:06:00 smithi029 ceph-mon[26769]: pgmap v240: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:06:01.745 INFO:tasks.workunit.client.1.smithi052.stderr:Updating files: 87% (10954/12492) Updating files: 88% (10993/12492) Updating files: 89% (11118/12492) Updating files: 90% (11243/12492) Updating files: 91% (11368/12492) Updating files: 92% (11493/12492) Updating files: 93% (11618/12492) Updating files: 94% (11743/12492) Updating files: 95% (11868/12492) Updating files: 96% (11993/12492) Updating files: 97% (12118/12492) Updating files: 98% (12243/12492) Updating files: 99% (12368/12492) Updating files: 100% (12492/12492) Updating files: 100% (12492/12492), done. 2024-04-27T02:06:02.632 INFO:tasks.workunit.client.1.smithi052.stderr:Note: switching to 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a'. 2024-04-27T02:06:02.632 INFO:tasks.workunit.client.1.smithi052.stderr: 2024-04-27T02:06:02.632 INFO:tasks.workunit.client.1.smithi052.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-27T02:06:02.632 INFO:tasks.workunit.client.1.smithi052.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-27T02:06:02.632 INFO:tasks.workunit.client.1.smithi052.stderr:state without impacting any branches by switching back to a branch. 2024-04-27T02:06:02.632 INFO:tasks.workunit.client.1.smithi052.stderr: 2024-04-27T02:06:02.632 INFO:tasks.workunit.client.1.smithi052.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-27T02:06:02.633 INFO:tasks.workunit.client.1.smithi052.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-27T02:06:02.633 INFO:tasks.workunit.client.1.smithi052.stderr: 2024-04-27T02:06:02.633 INFO:tasks.workunit.client.1.smithi052.stderr: git switch -c 2024-04-27T02:06:02.633 INFO:tasks.workunit.client.1.smithi052.stderr: 2024-04-27T02:06:02.633 INFO:tasks.workunit.client.1.smithi052.stderr:Or undo this operation with: 2024-04-27T02:06:02.633 INFO:tasks.workunit.client.1.smithi052.stderr: 2024-04-27T02:06:02.633 INFO:tasks.workunit.client.1.smithi052.stderr: git switch - 2024-04-27T02:06:02.633 INFO:tasks.workunit.client.1.smithi052.stderr: 2024-04-27T02:06:02.633 INFO:tasks.workunit.client.1.smithi052.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-27T02:06:02.633 INFO:tasks.workunit.client.1.smithi052.stderr: 2024-04-27T02:06:02.633 INFO:tasks.workunit.client.1.smithi052.stderr:HEAD is now at b22e2ebdeb2 Merge pull request #57078 from adk3798/squid-cephadm-no-downgrade 2024-04-27T02:06:02.636 DEBUG:teuthology.orchestra.run.smithi052:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-04-27T02:06:02.876 INFO:tasks.workunit.client.1.smithi052.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-27T02:06:02.878 INFO:tasks.workunit.client.1.smithi052.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-27T02:06:02.878 INFO:tasks.workunit.client.1.smithi052.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-27T02:06:03.232 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:06:02 smithi052 ceph-mon[32586]: pgmap v241: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-27T02:06:03.450 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:06:02 smithi029 ceph-mon[26769]: pgmap v241: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-27T02:06:04.286 INFO:tasks.workunit.client.1.smithi052.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-27T02:06:04.386 INFO:tasks.workunit.client.1.smithi052.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-27T02:06:04.427 INFO:tasks.workunit.client.1.smithi052.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-27T02:06:04.429 INFO:tasks.workunit.client.1.smithi052.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-27T02:06:04.429 INFO:tasks.workunit.client.1.smithi052.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-27T02:06:04.471 INFO:tasks.workunit.client.1.smithi052.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-27T02:06:04.475 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-27T02:06:04.475 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-04-27T02:06:04.492 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-04-27T02:06:04.494 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-27T02:06:04.494 DEBUG:teuthology.orchestra.run.smithi052:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b22e2ebdeb24376882b7bda2a7329c8cccc2276a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-04-27T02:06:04.553 INFO:tasks.workunit.client.1.smithi052.stderr:+ mkdir -p fsstress 2024-04-27T02:06:04.555 INFO:tasks.workunit.client.1.smithi052.stderr:+ pushd fsstress 2024-04-27T02:06:04.555 INFO:tasks.workunit.client.1.smithi052.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-27T02:06:04.555 INFO:tasks.workunit.client.1.smithi052.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-27T02:06:05.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:06:04 smithi052 ceph-mon[32586]: pgmap v242: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:06:05.307 INFO:tasks.workunit.client.0.smithi029.stderr:Updating files: 94% (11794/12492) Updating files: 95% (11868/12492) Updating files: 96% (11993/12492) Updating files: 97% (12118/12492) Updating files: 98% (12243/12492) Updating files: 99% (12368/12492) Updating files: 100% (12492/12492) Updating files: 100% (12492/12492), done. 2024-04-27T02:06:05.450 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:06:04 smithi029 ceph-mon[26769]: pgmap v242: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:06:06.442 INFO:tasks.workunit.client.0.smithi029.stderr:Note: switching to 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a'. 2024-04-27T02:06:06.442 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-04-27T02:06:06.442 INFO:tasks.workunit.client.0.smithi029.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-27T02:06:06.443 INFO:tasks.workunit.client.0.smithi029.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-27T02:06:06.443 INFO:tasks.workunit.client.0.smithi029.stderr:state without impacting any branches by switching back to a branch. 2024-04-27T02:06:06.443 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-04-27T02:06:06.443 INFO:tasks.workunit.client.0.smithi029.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-27T02:06:06.443 INFO:tasks.workunit.client.0.smithi029.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-27T02:06:06.443 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-04-27T02:06:06.443 INFO:tasks.workunit.client.0.smithi029.stderr: git switch -c 2024-04-27T02:06:06.443 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-04-27T02:06:06.443 INFO:tasks.workunit.client.0.smithi029.stderr:Or undo this operation with: 2024-04-27T02:06:06.443 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-04-27T02:06:06.443 INFO:tasks.workunit.client.0.smithi029.stderr: git switch - 2024-04-27T02:06:06.443 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-04-27T02:06:06.443 INFO:tasks.workunit.client.0.smithi029.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-27T02:06:06.443 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-04-27T02:06:06.443 INFO:tasks.workunit.client.0.smithi029.stderr:HEAD is now at b22e2ebdeb2 Merge pull request #57078 from adk3798/squid-cephadm-no-downgrade 2024-04-27T02:06:06.447 DEBUG:teuthology.orchestra.run.smithi029:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-27T02:06:07.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:06:06 smithi052 ceph-mon[32586]: pgmap v243: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:06:07.288 INFO:tasks.workunit.client.1.smithi052.stderr:+ tar xzf ltp-full.tgz 2024-04-27T02:06:07.450 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:06:06 smithi029 ceph-mon[26769]: pgmap v243: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-04-27T02:06:08.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:06:07 smithi052 ceph-mon[32586]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:06:08.286 INFO:tasks.workunit.client.0.smithi029.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-27T02:06:08.289 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-27T02:06:08.289 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-27T02:06:08.311 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:06:07 smithi029 ceph-mon[26769]: from='mgr.14215 172.21.15.29:0/44878238' entity='mgr.smithi029.ewvdzs' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:06:09.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:06:08 smithi052 ceph-mon[32586]: pgmap v244: 65 pgs: 65 active+clean; 599 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T02:06:09.450 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:06:08 smithi029 ceph-mon[26769]: pgmap v244: 65 pgs: 65 active+clean; 599 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T02:06:10.450 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:06:10 smithi029 ceph-mon[26769]: pgmap v245: 65 pgs: 65 active+clean; 599 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T02:06:10.503 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:06:10 smithi052 ceph-mon[32586]: pgmap v245: 65 pgs: 65 active+clean; 599 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T02:06:13.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:06:12 smithi029 ceph-mon[26769]: pgmap v246: 65 pgs: 65 active+clean; 6.4 MiB data, 182 MiB used, 536 GiB / 536 GiB avail; 342 KiB/s rd, 521 KiB/s wr, 48 op/s 2024-04-27T02:06:13.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:06:12 smithi052 ceph-mon[32586]: pgmap v246: 65 pgs: 65 active+clean; 6.4 MiB data, 182 MiB used, 536 GiB / 536 GiB avail; 342 KiB/s rd, 521 KiB/s wr, 48 op/s 2024-04-27T02:06:13.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.742+0000 7f7896ce8700 1 -- 172.21.15.29:0/2397344765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7890101440 msgr2=0x7f7890101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:13.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.742+0000 7f7896ce8700 1 --2- 172.21.15.29:0/2397344765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7890101440 0x7f7890101890 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f787c009910 tx=0x7f787c009c20 comp rx=0 tx=0).stop 2024-04-27T02:06:13.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.742+0000 7f7896ce8700 1 -- 172.21.15.29:0/2397344765 shutdown_connections 2024-04-27T02:06:13.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.742+0000 7f7896ce8700 1 --2- 172.21.15.29:0/2397344765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7890101440 0x7f7890101890 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:13.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.742+0000 7f7896ce8700 1 --2- 172.21.15.29:0/2397344765 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7890100240 0x7f7890100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:13.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.742+0000 7f7896ce8700 1 -- 172.21.15.29:0/2397344765 >> 172.21.15.29:0/2397344765 conn(0x7f78900fb7f0 msgr2=0x7f78900fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:06:13.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.742+0000 7f7896ce8700 1 -- 172.21.15.29:0/2397344765 shutdown_connections 2024-04-27T02:06:13.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.742+0000 7f7896ce8700 1 -- 172.21.15.29:0/2397344765 wait complete. 2024-04-27T02:06:13.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.743+0000 7f7896ce8700 1 Processor -- start 2024-04-27T02:06:13.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.743+0000 7f7896ce8700 1 -- start start 2024-04-27T02:06:13.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.743+0000 7f7896ce8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7890100240 0x7f7890195ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:06:13.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.743+0000 7f7896ce8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7890101440 0x7f7890195ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:06:13.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.743+0000 7f7896ce8700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7890196610 con 0x7f7890100240 2024-04-27T02:06:13.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.743+0000 7f7896ce8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7890196750 con 0x7f7890101440 2024-04-27T02:06:13.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.743+0000 7f788ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7890101440 0x7f7890195ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:06:13.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.743+0000 7f7894a84700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7890100240 0x7f7890195ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:06:13.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.743+0000 7f788ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7890101440 0x7f7890195ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:56618/0 (socket says 172.21.15.29:56618) 2024-04-27T02:06:13.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.743+0000 7f788ffff700 1 -- 172.21.15.29:0/507496319 learned_addr learned my addr 172.21.15.29:0/507496319 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:06:13.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.744+0000 7f788ffff700 1 -- 172.21.15.29:0/507496319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7890100240 msgr2=0x7f7890195ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:13.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.744+0000 7f788ffff700 1 --2- 172.21.15.29:0/507496319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7890100240 0x7f7890195ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:13.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.744+0000 7f788ffff700 1 -- 172.21.15.29:0/507496319 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f787c0095f0 con 0x7f7890101440 2024-04-27T02:06:13.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.744+0000 7f788ffff700 1 --2- 172.21.15.29:0/507496319 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7890101440 0x7f7890195ff0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f787c000c00 tx=0x7f787c00f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:06:13.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.744+0000 7f788dffb700 1 -- 172.21.15.29:0/507496319 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f787c01d070 con 0x7f7890101440 2024-04-27T02:06:13.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.744+0000 7f788dffb700 1 -- 172.21.15.29:0/507496319 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f787c005dc0 con 0x7f7890101440 2024-04-27T02:06:13.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.744+0000 7f788dffb700 1 -- 172.21.15.29:0/507496319 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f787c0226a0 con 0x7f7890101440 2024-04-27T02:06:13.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.744+0000 7f7896ce8700 1 -- 172.21.15.29:0/507496319 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f789019b1a0 con 0x7f7890101440 2024-04-27T02:06:13.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.745+0000 7f7896ce8700 1 -- 172.21.15.29:0/507496319 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f789019b690 con 0x7f7890101440 2024-04-27T02:06:13.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.745+0000 7f7896ce8700 1 -- 172.21.15.29:0/507496319 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7890108d90 con 0x7f7890101440 2024-04-27T02:06:13.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.746+0000 7f788dffb700 1 -- 172.21.15.29:0/507496319 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f787c0038f0 con 0x7f7890101440 2024-04-27T02:06:13.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.747+0000 7f788dffb700 1 --2- 172.21.15.29:0/507496319 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f788006c680 0x7f788006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:06:13.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.747+0000 7f788dffb700 1 -- 172.21.15.29:0/507496319 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f787c08ce20 con 0x7f7890101440 2024-04-27T02:06:13.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.747+0000 7f7894a84700 1 --2- 172.21.15.29:0/507496319 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f788006c680 0x7f788006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:06:13.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.748+0000 7f7894a84700 1 --2- 172.21.15.29:0/507496319 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f788006c680 0x7f788006eb30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f78901012a0 tx=0x7f7884009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:06:13.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.752+0000 7f788dffb700 1 -- 172.21.15.29:0/507496319 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f787c05aed0 con 0x7f7890101440 2024-04-27T02:06:13.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.991+0000 7f7896ce8700 1 -- 172.21.15.29:0/507496319 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7890105d90 con 0x7f788006c680 2024-04-27T02:06:13.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.992+0000 7f788dffb700 1 -- 172.21.15.29:0/507496319 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f7890105d90 con 0x7f788006c680 2024-04-27T02:06:13.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.994+0000 7f7896ce8700 1 -- 172.21.15.29:0/507496319 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f788006c680 msgr2=0x7f788006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:13.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.994+0000 7f7896ce8700 1 --2- 172.21.15.29:0/507496319 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f788006c680 0x7f788006eb30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f78901012a0 tx=0x7f7884009260 comp rx=0 tx=0).stop 2024-04-27T02:06:13.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.994+0000 7f7896ce8700 1 -- 172.21.15.29:0/507496319 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7890101440 msgr2=0x7f7890195ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:13.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.994+0000 7f7896ce8700 1 --2- 172.21.15.29:0/507496319 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7890101440 0x7f7890195ff0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f787c000c00 tx=0x7f787c00f690 comp rx=0 tx=0).stop 2024-04-27T02:06:13.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.994+0000 7f7896ce8700 1 -- 172.21.15.29:0/507496319 shutdown_connections 2024-04-27T02:06:13.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.994+0000 7f7896ce8700 1 --2- 172.21.15.29:0/507496319 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f788006c680 0x7f788006eb30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:13.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.994+0000 7f7896ce8700 1 --2- 172.21.15.29:0/507496319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7890100240 0x7f7890195ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:13.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.994+0000 7f7896ce8700 1 --2- 172.21.15.29:0/507496319 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7890101440 0x7f7890195ff0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:13.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.994+0000 7f7896ce8700 1 -- 172.21.15.29:0/507496319 >> 172.21.15.29:0/507496319 conn(0x7f78900fb7f0 msgr2=0x7f7890104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:06:13.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.994+0000 7f7896ce8700 1 -- 172.21.15.29:0/507496319 shutdown_connections 2024-04-27T02:06:13.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:13.994+0000 7f7896ce8700 1 -- 172.21.15.29:0/507496319 wait complete. 2024-04-27T02:06:13.986 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-04-27T02:06:14.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.141+0000 7f13b81f9700 1 -- 172.21.15.29:0/270324032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b0101350 msgr2=0x7f13b01017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:14.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.141+0000 7f13b81f9700 1 --2- 172.21.15.29:0/270324032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b0101350 0x7f13b01017a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f13a0009910 tx=0x7f13a0009c20 comp rx=0 tx=0).stop 2024-04-27T02:06:14.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.141+0000 7f13b81f9700 1 -- 172.21.15.29:0/270324032 shutdown_connections 2024-04-27T02:06:14.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.141+0000 7f13b81f9700 1 --2- 172.21.15.29:0/270324032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b0101350 0x7f13b01017a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:14.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.141+0000 7f13b81f9700 1 --2- 172.21.15.29:0/270324032 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f13b0100150 0x7f13b0100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:14.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.141+0000 7f13b81f9700 1 -- 172.21.15.29:0/270324032 >> 172.21.15.29:0/270324032 conn(0x7f13b00fb6e0 msgr2=0x7f13b00fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:06:14.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.141+0000 7f13b81f9700 1 -- 172.21.15.29:0/270324032 shutdown_connections 2024-04-27T02:06:14.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.141+0000 7f13b81f9700 1 -- 172.21.15.29:0/270324032 wait complete. 2024-04-27T02:06:14.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.142+0000 7f13b81f9700 1 Processor -- start 2024-04-27T02:06:14.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.142+0000 7f13b81f9700 1 -- start start 2024-04-27T02:06:14.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.142+0000 7f13b81f9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b0100150 0x7f13b0195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:06:14.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.142+0000 7f13b81f9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f13b0101350 0x7f13b0195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:06:14.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.142+0000 7f13b81f9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b0196470 con 0x7f13b0100150 2024-04-27T02:06:14.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.142+0000 7f13b81f9700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b01965b0 con 0x7f13b0101350 2024-04-27T02:06:14.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.142+0000 7f13b5f95700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b0100150 0x7f13b0195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:06:14.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.142+0000 7f13b5f95700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b0100150 0x7f13b0195910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35802/0 (socket says 172.21.15.29:35802) 2024-04-27T02:06:14.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.142+0000 7f13b5f95700 1 -- 172.21.15.29:0/1161391700 learned_addr learned my addr 172.21.15.29:0/1161391700 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:06:14.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.142+0000 7f13b5794700 1 --2- 172.21.15.29:0/1161391700 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f13b0101350 0x7f13b0195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:06:14.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.143+0000 7f13b5f95700 1 -- 172.21.15.29:0/1161391700 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f13b0101350 msgr2=0x7f13b0195e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:14.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.143+0000 7f13b5f95700 1 --2- 172.21.15.29:0/1161391700 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f13b0101350 0x7f13b0195e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:14.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.143+0000 7f13b5f95700 1 -- 172.21.15.29:0/1161391700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a00095f0 con 0x7f13b0100150 2024-04-27T02:06:14.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.143+0000 7f13b5f95700 1 --2- 172.21.15.29:0/1161391700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b0100150 0x7f13b0195910 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f13ac00e8f0 tx=0x7f13ac00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:06:14.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.143+0000 7f13a6ffd700 1 -- 172.21.15.29:0/1161391700 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13ac009790 con 0x7f13b0100150 2024-04-27T02:06:14.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.143+0000 7f13a6ffd700 1 -- 172.21.15.29:0/1161391700 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f13ac00c810 con 0x7f13b0100150 2024-04-27T02:06:14.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.143+0000 7f13b81f9700 1 -- 172.21.15.29:0/1161391700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b019b060 con 0x7f13b0100150 2024-04-27T02:06:14.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.143+0000 7f13a6ffd700 1 -- 172.21.15.29:0/1161391700 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13ac003d20 con 0x7f13b0100150 2024-04-27T02:06:14.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.144+0000 7f13b81f9700 1 -- 172.21.15.29:0/1161391700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b019b5b0 con 0x7f13b0100150 2024-04-27T02:06:14.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.145+0000 7f13b81f9700 1 -- 172.21.15.29:0/1161391700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b0108ca0 con 0x7f13b0100150 2024-04-27T02:06:14.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.146+0000 7f13a6ffd700 1 -- 172.21.15.29:0/1161391700 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f13ac0041f0 con 0x7f13b0100150 2024-04-27T02:06:14.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.146+0000 7f13a6ffd700 1 --2- 172.21.15.29:0/1161391700 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f139c06c680 0x7f139c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:06:14.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.146+0000 7f13a6ffd700 1 -- 172.21.15.29:0/1161391700 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f13ac014070 con 0x7f13b0100150 2024-04-27T02:06:14.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.146+0000 7f13b5794700 1 --2- 172.21.15.29:0/1161391700 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f139c06c680 0x7f139c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:06:14.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.147+0000 7f13b5794700 1 --2- 172.21.15.29:0/1161391700 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f139c06c680 0x7f139c06eb30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f13a0009fd0 tx=0x7f13a0005940 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:06:14.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.151+0000 7f13a6ffd700 1 -- 172.21.15.29:0/1161391700 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f13ac059e00 con 0x7f13b0100150 2024-04-27T02:06:14.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.395+0000 7f13b81f9700 1 -- 172.21.15.29:0/1161391700 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f13b0105ca0 con 0x7f139c06c680 2024-04-27T02:06:14.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.396+0000 7f13a6ffd700 1 -- 172.21.15.29:0/1161391700 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f13b0105ca0 con 0x7f139c06c680 2024-04-27T02:06:14.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.398+0000 7f13b81f9700 1 -- 172.21.15.29:0/1161391700 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f139c06c680 msgr2=0x7f139c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:14.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.398+0000 7f13b81f9700 1 --2- 172.21.15.29:0/1161391700 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f139c06c680 0x7f139c06eb30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f13a0009fd0 tx=0x7f13a0005940 comp rx=0 tx=0).stop 2024-04-27T02:06:14.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.398+0000 7f13b81f9700 1 -- 172.21.15.29:0/1161391700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b0100150 msgr2=0x7f13b0195910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:14.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.398+0000 7f13b81f9700 1 --2- 172.21.15.29:0/1161391700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b0100150 0x7f13b0195910 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f13ac00e8f0 tx=0x7f13ac00ec00 comp rx=0 tx=0).stop 2024-04-27T02:06:14.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.398+0000 7f13b81f9700 1 -- 172.21.15.29:0/1161391700 shutdown_connections 2024-04-27T02:06:14.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.398+0000 7f13b81f9700 1 --2- 172.21.15.29:0/1161391700 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f139c06c680 0x7f139c06eb30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:14.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.398+0000 7f13b81f9700 1 --2- 172.21.15.29:0/1161391700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b0100150 0x7f13b0195910 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:14.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.398+0000 7f13b81f9700 1 --2- 172.21.15.29:0/1161391700 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f13b0101350 0x7f13b0195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:14.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.398+0000 7f13b81f9700 1 -- 172.21.15.29:0/1161391700 >> 172.21.15.29:0/1161391700 conn(0x7f13b00fb6e0 msgr2=0x7f13b0104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:06:14.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.399+0000 7f13b81f9700 1 -- 172.21.15.29:0/1161391700 shutdown_connections 2024-04-27T02:06:14.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.399+0000 7f13b81f9700 1 -- 172.21.15.29:0/1161391700 wait complete. 2024-04-27T02:06:14.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.510+0000 7f4880bf2700 1 -- 172.21.15.29:0/732559806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f487c069250 msgr2=0x7f487c1051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:14.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.510+0000 7f4880bf2700 1 --2- 172.21.15.29:0/732559806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f487c069250 0x7f487c1051f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f486c009910 tx=0x7f486c009c20 comp rx=0 tx=0).stop 2024-04-27T02:06:14.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.510+0000 7f4880bf2700 1 -- 172.21.15.29:0/732559806 shutdown_connections 2024-04-27T02:06:14.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.510+0000 7f4880bf2700 1 --2- 172.21.15.29:0/732559806 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f487c105730 0x7f487c107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:14.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.510+0000 7f4880bf2700 1 --2- 172.21.15.29:0/732559806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f487c069250 0x7f487c1051f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:14.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.510+0000 7f4880bf2700 1 -- 172.21.15.29:0/732559806 >> 172.21.15.29:0/732559806 conn(0x7f487c0fa780 msgr2=0x7f487c0fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:06:14.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.510+0000 7f4880bf2700 1 -- 172.21.15.29:0/732559806 shutdown_connections 2024-04-27T02:06:14.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.510+0000 7f4880bf2700 1 -- 172.21.15.29:0/732559806 wait complete. 2024-04-27T02:06:14.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.511+0000 7f4880bf2700 1 Processor -- start 2024-04-27T02:06:14.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.511+0000 7f4880bf2700 1 -- start start 2024-04-27T02:06:14.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.512+0000 7f4880bf2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f487c069250 0x7f487c197d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:06:14.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.512+0000 7f4880bf2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f487c105730 0x7f487c198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:06:14.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.512+0000 7f4880bf2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487c198880 con 0x7f487c069250 2024-04-27T02:06:14.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.512+0000 7f4880bf2700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487c19d290 con 0x7f487c105730 2024-04-27T02:06:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.512+0000 7f487a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f487c069250 0x7f487c197d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:06:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.512+0000 7f487a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f487c069250 0x7f487c197d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35808/0 (socket says 172.21.15.29:35808) 2024-04-27T02:06:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.512+0000 7f4879d9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f487c105730 0x7f487c198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:06:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.512+0000 7f487a59c700 1 -- 172.21.15.29:0/1516400702 learned_addr learned my addr 172.21.15.29:0/1516400702 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:06:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.512+0000 7f4879d9b700 1 -- 172.21.15.29:0/1516400702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f487c069250 msgr2=0x7f487c197d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.512+0000 7f4879d9b700 1 --2- 172.21.15.29:0/1516400702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f487c069250 0x7f487c197d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.512+0000 7f4879d9b700 1 -- 172.21.15.29:0/1516400702 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486c0095f0 con 0x7f487c105730 2024-04-27T02:06:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.512+0000 7f487a59c700 1 --2- 172.21.15.29:0/1516400702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f487c069250 0x7f487c197d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T02:06:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.513+0000 7f4879d9b700 1 --2- 172.21.15.29:0/1516400702 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f487c105730 0x7f487c198260 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f487000e970 tx=0x7f487000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:06:14.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.513+0000 7f486b7fe700 1 -- 172.21.15.29:0/1516400702 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f487000c9d0 con 0x7f487c105730 2024-04-27T02:06:14.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.513+0000 7f486b7fe700 1 -- 172.21.15.29:0/1516400702 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4870010430 con 0x7f487c105730 2024-04-27T02:06:14.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.513+0000 7f4880bf2700 1 -- 172.21.15.29:0/1516400702 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f487c19d490 con 0x7f487c105730 2024-04-27T02:06:14.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.513+0000 7f486b7fe700 1 -- 172.21.15.29:0/1516400702 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4870018650 con 0x7f487c105730 2024-04-27T02:06:14.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.513+0000 7f4880bf2700 1 -- 172.21.15.29:0/1516400702 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f487c19d8f0 con 0x7f487c105730 2024-04-27T02:06:14.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.514+0000 7f4880bf2700 1 -- 172.21.15.29:0/1516400702 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f487c191dd0 con 0x7f487c105730 2024-04-27T02:06:14.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.514+0000 7f486b7fe700 1 -- 172.21.15.29:0/1516400702 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f48700105a0 con 0x7f487c105730 2024-04-27T02:06:14.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.515+0000 7f486b7fe700 1 --2- 172.21.15.29:0/1516400702 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f486406c680 0x7f486406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:06:14.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.515+0000 7f486b7fe700 1 -- 172.21.15.29:0/1516400702 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f4870014070 con 0x7f487c105730 2024-04-27T02:06:14.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.515+0000 7f487a59c700 1 --2- 172.21.15.29:0/1516400702 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f486406c680 0x7f486406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:06:14.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.516+0000 7f487a59c700 1 --2- 172.21.15.29:0/1516400702 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f486406c680 0x7f486406eb30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f486c009fd0 tx=0x7f486c005940 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:06:14.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.520+0000 7f486b7fe700 1 -- 172.21.15.29:0/1516400702 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4870051720 con 0x7f487c105730 2024-04-27T02:06:14.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.804+0000 7f4880bf2700 1 -- 172.21.15.29:0/1516400702 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f487c061190 con 0x7f486406c680 2024-04-27T02:06:14.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.818+0000 7f486b7fe700 1 -- 172.21.15.29:0/1516400702 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3528 (secure 0 0 0) 0x7f487c061190 con 0x7f486406c680 2024-04-27T02:06:14.798 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:06:14.798 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.smithi029 smithi029 *:9093,9094 running (6m) 93s ago 7m 25.9M - 0.25.0 c8568f914cd2 f713bca72a8f 2024-04-27T02:06:14.798 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-exporter.smithi029 smithi029 running (7m) 93s ago 7m 7428k - 18.2.0 dc2bc1663786 d585d22341b7 2024-04-27T02:06:14.798 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-exporter.smithi052 smithi052 running (6m) 99s ago 6m 7335k - 18.2.0 dc2bc1663786 3555a1bdbadc 2024-04-27T02:06:14.798 INFO:teuthology.orchestra.run.smithi029.stdout:crash.smithi029 smithi029 running (7m) 93s ago 7m 6665k - 18.2.0 dc2bc1663786 aa848ffaabfc 2024-04-27T02:06:14.798 INFO:teuthology.orchestra.run.smithi029.stdout:crash.smithi052 smithi052 running (6m) 99s ago 6m 6660k - 18.2.0 dc2bc1663786 460da511ea40 2024-04-27T02:06:14.798 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.smithi029 smithi029 *:3000 running (6m) 93s ago 7m 87.9M - 9.4.7 954c08fa6188 2e3e5d432724 2024-04-27T02:06:14.798 INFO:teuthology.orchestra.run.smithi029.stdout:mds.cephfs.smithi029.dyrjrd smithi029 running (2m) 93s ago 2m 11.5M - 18.2.0 dc2bc1663786 5e33fd05ca5f 2024-04-27T02:06:14.798 INFO:teuthology.orchestra.run.smithi029.stdout:mds.cephfs.smithi029.njjezc smithi029 running (115s) 93s ago 114s 17.1M - 18.2.0 dc2bc1663786 d278cd21f1d4 2024-04-27T02:06:14.798 INFO:teuthology.orchestra.run.smithi029.stdout:mds.cephfs.smithi052.nbjpjd smithi052 running (112s) 99s ago 112s 14.5M - 18.2.0 dc2bc1663786 b32817e8b1f9 2024-04-27T02:06:14.798 INFO:teuthology.orchestra.run.smithi029.stdout:mds.cephfs.smithi052.yylmkb smithi052 running (118s) 99s ago 118s 13.0M - 18.2.0 dc2bc1663786 9e395db428ee 2024-04-27T02:06:14.799 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.smithi029.ewvdzs smithi029 *:9283,8765,8443 running (9m) 93s ago 9m 486M - 18.2.0 dc2bc1663786 76689e16ce1a 2024-04-27T02:06:14.799 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.smithi052.gbsftx smithi052 *:8443,9283,8765 running (6m) 99s ago 6m 439M - 18.2.0 dc2bc1663786 72bb682980a1 2024-04-27T02:06:14.799 INFO:teuthology.orchestra.run.smithi029.stdout:mon.smithi029 smithi029 running (9m) 93s ago 9m 45.7M 2048M 18.2.0 dc2bc1663786 33130493e80d 2024-04-27T02:06:14.799 INFO:teuthology.orchestra.run.smithi029.stdout:mon.smithi052 smithi052 running (6m) 99s ago 6m 35.7M 2048M 18.2.0 dc2bc1663786 5af8b02bce2f 2024-04-27T02:06:14.799 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.smithi029 smithi029 *:9100 running (7m) 93s ago 7m 15.7M - 1.5.0 0da6a335fe13 f65d14d021a2 2024-04-27T02:06:14.799 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.smithi052 smithi052 *:9100 running (6m) 99s ago 6m 16.6M - 1.5.0 0da6a335fe13 71aadb2b8e2e 2024-04-27T02:06:14.799 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (5m) 93s ago 5m 47.5M 3948M 18.2.0 dc2bc1663786 b4688a5b18ec 2024-04-27T02:06:14.799 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (5m) 93s ago 5m 48.4M 3948M 18.2.0 dc2bc1663786 7123e1ad8219 2024-04-27T02:06:14.799 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (4m) 93s ago 4m 45.1M 3948M 18.2.0 dc2bc1663786 f8ecf2655a40 2024-04-27T02:06:14.799 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi052 running (4m) 99s ago 4m 46.3M 4972M 18.2.0 dc2bc1663786 584fb2e408cd 2024-04-27T02:06:14.799 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi052 running (4m) 99s ago 3m 43.1M 4972M 18.2.0 dc2bc1663786 26fdd5746d2a 2024-04-27T02:06:14.799 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi052 running (3m) 99s ago 3m 43.7M 4972M 18.2.0 dc2bc1663786 6fa7665653a4 2024-04-27T02:06:14.799 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.smithi029 smithi029 *:9095 running (5m) 93s ago 7m 64.6M - 2.43.0 a07b618ecd1d a9ebc9badf18 2024-04-27T02:06:14.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.820+0000 7f4880bf2700 1 -- 172.21.15.29:0/1516400702 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f486406c680 msgr2=0x7f486406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:14.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.820+0000 7f4880bf2700 1 --2- 172.21.15.29:0/1516400702 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f486406c680 0x7f486406eb30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f486c009fd0 tx=0x7f486c005940 comp rx=0 tx=0).stop 2024-04-27T02:06:14.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.820+0000 7f4880bf2700 1 -- 172.21.15.29:0/1516400702 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f487c105730 msgr2=0x7f487c198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:14.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.820+0000 7f4880bf2700 1 --2- 172.21.15.29:0/1516400702 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f487c105730 0x7f487c198260 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f487000e970 tx=0x7f487000ec80 comp rx=0 tx=0).stop 2024-04-27T02:06:14.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.820+0000 7f4880bf2700 1 -- 172.21.15.29:0/1516400702 shutdown_connections 2024-04-27T02:06:14.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.820+0000 7f4880bf2700 1 --2- 172.21.15.29:0/1516400702 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f486406c680 0x7f486406eb30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:14.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.820+0000 7f4880bf2700 1 --2- 172.21.15.29:0/1516400702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f487c069250 0x7f487c197d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:14.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.820+0000 7f4880bf2700 1 --2- 172.21.15.29:0/1516400702 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f487c105730 0x7f487c198260 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:14.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.820+0000 7f4880bf2700 1 -- 172.21.15.29:0/1516400702 >> 172.21.15.29:0/1516400702 conn(0x7f487c0fa780 msgr2=0x7f487c0fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:06:14.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.820+0000 7f4880bf2700 1 -- 172.21.15.29:0/1516400702 shutdown_connections 2024-04-27T02:06:14.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.820+0000 7f4880bf2700 1 -- 172.21.15.29:0/1516400702 wait complete. 2024-04-27T02:06:14.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.938+0000 7fa28e32d700 1 -- 172.21.15.29:0/3022022685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa288103310 msgr2=0x7fa288105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:14.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.938+0000 7fa28e32d700 1 --2- 172.21.15.29:0/3022022685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa288103310 0x7fa288105790 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa27c009910 tx=0x7fa27c009c20 comp rx=0 tx=0).stop 2024-04-27T02:06:14.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.938+0000 7fa28e32d700 1 -- 172.21.15.29:0/3022022685 shutdown_connections 2024-04-27T02:06:14.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.938+0000 7fa28e32d700 1 --2- 172.21.15.29:0/3022022685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa288103310 0x7fa288105790 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:14.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.938+0000 7fa28e32d700 1 --2- 172.21.15.29:0/3022022685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa288069270 0x7fa288102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:14.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.938+0000 7fa28e32d700 1 -- 172.21.15.29:0/3022022685 >> 172.21.15.29:0/3022022685 conn(0x7fa2880fa780 msgr2=0x7fa2880fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:06:14.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.938+0000 7fa28e32d700 1 -- 172.21.15.29:0/3022022685 shutdown_connections 2024-04-27T02:06:14.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.938+0000 7fa28e32d700 1 -- 172.21.15.29:0/3022022685 wait complete. 2024-04-27T02:06:14.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.939+0000 7fa28e32d700 1 Processor -- start 2024-04-27T02:06:14.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.939+0000 7fa28e32d700 1 -- start start 2024-04-27T02:06:14.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.939+0000 7fa28e32d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa288103310 0x7fa288100720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:06:14.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.939+0000 7fa28e32d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2880fed70 0x7fa2880ff1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:06:14.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.939+0000 7fa28e32d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2880ff720 con 0x7fa2880fed70 2024-04-27T02:06:14.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.939+0000 7fa28e32d700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2880ff890 con 0x7fa288103310 2024-04-27T02:06:14.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.939+0000 7fa2877fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2880fed70 0x7fa2880ff1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:06:14.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.939+0000 7fa2877fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2880fed70 0x7fa2880ff1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35834/0 (socket says 172.21.15.29:35834) 2024-04-27T02:06:14.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.939+0000 7fa2877fe700 1 -- 172.21.15.29:0/576526052 learned_addr learned my addr 172.21.15.29:0/576526052 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:06:14.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.939+0000 7fa287fff700 1 --2- 172.21.15.29:0/576526052 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa288103310 0x7fa288100720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:06:14.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.940+0000 7fa2877fe700 1 -- 172.21.15.29:0/576526052 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa288103310 msgr2=0x7fa288100720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:14.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.940+0000 7fa2877fe700 1 --2- 172.21.15.29:0/576526052 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa288103310 0x7fa288100720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:14.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.940+0000 7fa2877fe700 1 -- 172.21.15.29:0/576526052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27c0095f0 con 0x7fa2880fed70 2024-04-27T02:06:14.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.940+0000 7fa287fff700 1 --2- 172.21.15.29:0/576526052 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa288103310 0x7fa288100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T02:06:14.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.940+0000 7fa2877fe700 1 --2- 172.21.15.29:0/576526052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2880fed70 0x7fa2880ff1e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa27c000c00 tx=0x7fa27c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:06:14.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.940+0000 7fa2857fa700 1 -- 172.21.15.29:0/576526052 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa27c01d070 con 0x7fa2880fed70 2024-04-27T02:06:14.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.940+0000 7fa28e32d700 1 -- 172.21.15.29:0/576526052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2880ffb10 con 0x7fa2880fed70 2024-04-27T02:06:14.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.940+0000 7fa2857fa700 1 -- 172.21.15.29:0/576526052 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa27c005dc0 con 0x7fa2880fed70 2024-04-27T02:06:14.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.940+0000 7fa2857fa700 1 -- 172.21.15.29:0/576526052 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa27c0226a0 con 0x7fa2880fed70 2024-04-27T02:06:14.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.940+0000 7fa28e32d700 1 -- 172.21.15.29:0/576526052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa28819dd00 con 0x7fa2880fed70 2024-04-27T02:06:14.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.941+0000 7fa28e32d700 1 -- 172.21.15.29:0/576526052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa28818fbf0 con 0x7fa2880fed70 2024-04-27T02:06:14.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.942+0000 7fa2857fa700 1 -- 172.21.15.29:0/576526052 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fa27c0038f0 con 0x7fa2880fed70 2024-04-27T02:06:14.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.943+0000 7fa2857fa700 1 --2- 172.21.15.29:0/576526052 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa27006c680 0x7fa27006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:06:14.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.943+0000 7fa2857fa700 1 -- 172.21.15.29:0/576526052 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa27c08cda0 con 0x7fa2880fed70 2024-04-27T02:06:14.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.943+0000 7fa287fff700 1 --2- 172.21.15.29:0/576526052 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa27006c680 0x7fa27006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:06:14.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.943+0000 7fa287fff700 1 --2- 172.21.15.29:0/576526052 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa27006c680 0x7fa27006eb30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa288068fa0 tx=0x7fa278009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:06:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:14.951+0000 7fa2857fa700 1 -- 172.21.15.29:0/576526052 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa27c05ae50 con 0x7fa2880fed70 2024-04-27T02:06:15.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:06:14 smithi029 ceph-mon[26769]: pgmap v247: 65 pgs: 65 active+clean; 25 MiB data, 245 MiB used, 536 GiB / 536 GiB avail; 683 KiB/s rd, 2.2 MiB/s wr, 163 op/s 2024-04-27T02:06:15.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:06:14 smithi052 ceph-mon[32586]: pgmap v247: 65 pgs: 65 active+clean; 25 MiB data, 245 MiB used, 536 GiB / 536 GiB avail; 683 KiB/s rd, 2.2 MiB/s wr, 163 op/s 2024-04-27T02:06:15.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.297+0000 7fa28e32d700 1 -- 172.21.15.29:0/576526052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa288069270 con 0x7fa2880fed70 2024-04-27T02:06:15.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.298+0000 7fa2857fa700 1 -- 172.21.15.29:0/576526052 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7fa27c003680 con 0x7fa2880fed70 2024-04-27T02:06:15.278 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-04-27T02:06:15.278 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-04-27T02:06:15.278 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-04-27T02:06:15.278 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-27T02:06:15.278 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-04-27T02:06:15.278 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-04-27T02:06:15.278 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-27T02:06:15.278 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-04-27T02:06:15.278 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-04-27T02:06:15.278 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-27T02:06:15.278 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": { 2024-04-27T02:06:15.279 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-27T02:06:15.279 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-27T02:06:15.279 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-04-27T02:06:15.279 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 14 2024-04-27T02:06:15.279 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-27T02:06:15.279 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-04-27T02:06:15.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.301+0000 7fa28e32d700 1 -- 172.21.15.29:0/576526052 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa27006c680 msgr2=0x7fa27006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:15.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.301+0000 7fa28e32d700 1 --2- 172.21.15.29:0/576526052 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa27006c680 0x7fa27006eb30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa288068fa0 tx=0x7fa278009260 comp rx=0 tx=0).stop 2024-04-27T02:06:15.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.301+0000 7fa28e32d700 1 -- 172.21.15.29:0/576526052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2880fed70 msgr2=0x7fa2880ff1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:15.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.301+0000 7fa28e32d700 1 --2- 172.21.15.29:0/576526052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2880fed70 0x7fa2880ff1e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa27c000c00 tx=0x7fa27c00f690 comp rx=0 tx=0).stop 2024-04-27T02:06:15.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.301+0000 7fa28e32d700 1 -- 172.21.15.29:0/576526052 shutdown_connections 2024-04-27T02:06:15.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.301+0000 7fa28e32d700 1 --2- 172.21.15.29:0/576526052 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7fa27006c680 0x7fa27006eb30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:15.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.301+0000 7fa28e32d700 1 --2- 172.21.15.29:0/576526052 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa288103310 0x7fa288100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:15.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.301+0000 7fa28e32d700 1 --2- 172.21.15.29:0/576526052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2880fed70 0x7fa2880ff1e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:15.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.301+0000 7fa28e32d700 1 -- 172.21.15.29:0/576526052 >> 172.21.15.29:0/576526052 conn(0x7fa2880fa780 msgr2=0x7fa2880fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:06:15.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.301+0000 7fa28e32d700 1 -- 172.21.15.29:0/576526052 shutdown_connections 2024-04-27T02:06:15.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.301+0000 7fa28e32d700 1 -- 172.21.15.29:0/576526052 wait complete. 2024-04-27T02:06:15.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.431+0000 7f85f9438700 1 -- 172.21.15.29:0/3555907174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85f4105710 msgr2=0x7f85f4107af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:15.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.431+0000 7f85f9438700 1 --2- 172.21.15.29:0/3555907174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85f4105710 0x7f85f4107af0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f85e8009910 tx=0x7f85e8009c20 comp rx=0 tx=0).stop 2024-04-27T02:06:15.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.431+0000 7f85f9438700 1 -- 172.21.15.29:0/3555907174 shutdown_connections 2024-04-27T02:06:15.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.431+0000 7f85f9438700 1 --2- 172.21.15.29:0/3555907174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85f4105710 0x7f85f4107af0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:15.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.431+0000 7f85f9438700 1 --2- 172.21.15.29:0/3555907174 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85f4069270 0x7f85f41051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:15.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.431+0000 7f85f9438700 1 -- 172.21.15.29:0/3555907174 >> 172.21.15.29:0/3555907174 conn(0x7f85f40fa780 msgr2=0x7f85f40fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:06:15.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.431+0000 7f85f9438700 1 -- 172.21.15.29:0/3555907174 shutdown_connections 2024-04-27T02:06:15.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.431+0000 7f85f9438700 1 -- 172.21.15.29:0/3555907174 wait complete. 2024-04-27T02:06:15.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.432+0000 7f85f9438700 1 Processor -- start 2024-04-27T02:06:15.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.432+0000 7f85f9438700 1 -- start start 2024-04-27T02:06:15.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.432+0000 7f85f9438700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85f4105710 0x7f85f4197cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:06:15.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.432+0000 7f85f9438700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85f4198230 0x7f85f419d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:06:15.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.432+0000 7f85f9438700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f41986a0 con 0x7f85f4105710 2024-04-27T02:06:15.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.432+0000 7f85f9438700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f4198810 con 0x7f85f4198230 2024-04-27T02:06:15.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.432+0000 7f85f2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85f4105710 0x7f85f4197cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:06:15.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.432+0000 7f85f2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85f4105710 0x7f85f4197cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35852/0 (socket says 172.21.15.29:35852) 2024-04-27T02:06:15.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.432+0000 7f85f2ffd700 1 -- 172.21.15.29:0/3423607635 learned_addr learned my addr 172.21.15.29:0/3423607635 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:06:15.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.432+0000 7f85f27fc700 1 --2- 172.21.15.29:0/3423607635 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85f4198230 0x7f85f419d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:06:15.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.433+0000 7f85f2ffd700 1 -- 172.21.15.29:0/3423607635 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85f4198230 msgr2=0x7f85f419d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:15.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.433+0000 7f85f2ffd700 1 --2- 172.21.15.29:0/3423607635 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85f4198230 0x7f85f419d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:15.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.433+0000 7f85f2ffd700 1 -- 172.21.15.29:0/3423607635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e80095f0 con 0x7f85f4105710 2024-04-27T02:06:15.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.433+0000 7f85f2ffd700 1 --2- 172.21.15.29:0/3423607635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85f4105710 0x7f85f4197cf0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f85e400e8f0 tx=0x7f85e400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:06:15.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.433+0000 7f85dbfff700 1 -- 172.21.15.29:0/3423607635 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85e4009790 con 0x7f85f4105710 2024-04-27T02:06:15.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.433+0000 7f85f9438700 1 -- 172.21.15.29:0/3423607635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f419d800 con 0x7f85f4105710 2024-04-27T02:06:15.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.433+0000 7f85dbfff700 1 -- 172.21.15.29:0/3423607635 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f85e400c810 con 0x7f85f4105710 2024-04-27T02:06:15.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.433+0000 7f85dbfff700 1 -- 172.21.15.29:0/3423607635 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85e40055b0 con 0x7f85f4105710 2024-04-27T02:06:15.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.433+0000 7f85f9438700 1 -- 172.21.15.29:0/3423607635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f419dd20 con 0x7f85f4105710 2024-04-27T02:06:15.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.434+0000 7f85f9438700 1 -- 172.21.15.29:0/3423607635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f4191df0 con 0x7f85f4105710 2024-04-27T02:06:15.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.435+0000 7f85dbfff700 1 -- 172.21.15.29:0/3423607635 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f85e400c980 con 0x7f85f4105710 2024-04-27T02:06:15.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.435+0000 7f85dbfff700 1 --2- 172.21.15.29:0/3423607635 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f85dc06c680 0x7f85dc06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:06:15.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.435+0000 7f85dbfff700 1 -- 172.21.15.29:0/3423607635 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f85e4014070 con 0x7f85f4105710 2024-04-27T02:06:15.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.435+0000 7f85f27fc700 1 --2- 172.21.15.29:0/3423607635 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f85dc06c680 0x7f85dc06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:06:15.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.436+0000 7f85f27fc700 1 --2- 172.21.15.29:0/3423607635 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f85dc06c680 0x7f85dc06eb30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f85e8009fd0 tx=0x7f85e8005fd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:06:15.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.443+0000 7f85dbfff700 1 -- 172.21.15.29:0/3423607635 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f85e40562e0 con 0x7f85f4105710 2024-04-27T02:06:15.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.709+0000 7f85f9438700 1 -- 172.21.15.29:0/3423607635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f85f402cf30 con 0x7f85f4105710 2024-04-27T02:06:15.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.709+0000 7f85dbfff700 1 -- 172.21.15.29:0/3423607635 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 19 v19) v1 ==== 76+0+1851 (secure 0 0 0) 0x7f85e4059900 con 0x7f85f4105710 2024-04-27T02:06:15.689 INFO:teuthology.orchestra.run.smithi029.stdout:e19 2024-04-27T02:06:15.690 INFO:teuthology.orchestra.run.smithi029.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T02:06:15.690 INFO:teuthology.orchestra.run.smithi029.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T02:06:15.690 INFO:teuthology.orchestra.run.smithi029.stdout:legacy client fscid: 1 2024-04-27T02:06:15.690 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:06:15.690 INFO:teuthology.orchestra.run.smithi029.stdout:Filesystem 'cephfs' (1) 2024-04-27T02:06:15.690 INFO:teuthology.orchestra.run.smithi029.stdout:fs_name cephfs 2024-04-27T02:06:15.690 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 19 2024-04-27T02:06:15.690 INFO:teuthology.orchestra.run.smithi029.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-27T02:06:15.690 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-04-27T02:04:10.035536+0000 2024-04-27T02:06:15.690 INFO:teuthology.orchestra.run.smithi029.stdout:modified 2024-04-27T02:04:26.892992+0000 2024-04-27T02:06:15.690 INFO:teuthology.orchestra.run.smithi029.stdout:tableserver 0 2024-04-27T02:06:15.690 INFO:teuthology.orchestra.run.smithi029.stdout:root 0 2024-04-27T02:06:15.691 INFO:teuthology.orchestra.run.smithi029.stdout:session_timeout 60 2024-04-27T02:06:15.691 INFO:teuthology.orchestra.run.smithi029.stdout:session_autoclose 300 2024-04-27T02:06:15.691 INFO:teuthology.orchestra.run.smithi029.stdout:max_file_size 1099511627776 2024-04-27T02:06:15.691 INFO:teuthology.orchestra.run.smithi029.stdout:required_client_features {} 2024-04-27T02:06:15.691 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure 0 2024-04-27T02:06:15.691 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure_osd_epoch 46 2024-04-27T02:06:15.691 INFO:teuthology.orchestra.run.smithi029.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T02:06:15.691 INFO:teuthology.orchestra.run.smithi029.stdout:max_mds 1 2024-04-27T02:06:15.691 INFO:teuthology.orchestra.run.smithi029.stdout:in 0 2024-04-27T02:06:15.691 INFO:teuthology.orchestra.run.smithi029.stdout:up {0=14484} 2024-04-27T02:06:15.691 INFO:teuthology.orchestra.run.smithi029.stdout:failed 2024-04-27T02:06:15.691 INFO:teuthology.orchestra.run.smithi029.stdout:damaged 2024-04-27T02:06:15.691 INFO:teuthology.orchestra.run.smithi029.stdout:stopped 2024-04-27T02:06:15.691 INFO:teuthology.orchestra.run.smithi029.stdout:data_pools [3] 2024-04-27T02:06:15.692 INFO:teuthology.orchestra.run.smithi029.stdout:metadata_pool 2 2024-04-27T02:06:15.692 INFO:teuthology.orchestra.run.smithi029.stdout:inline_data enabled 2024-04-27T02:06:15.692 INFO:teuthology.orchestra.run.smithi029.stdout:balancer 2024-04-27T02:06:15.692 INFO:teuthology.orchestra.run.smithi029.stdout:bal_rank_mask -1 2024-04-27T02:06:15.692 INFO:teuthology.orchestra.run.smithi029.stdout:standby_count_wanted 1 2024-04-27T02:06:15.692 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi029.njjezc{0:14484} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.29:6828/1353923457,v1:172.21.15.29:6829/1353923457] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T02:06:15.692 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi052.yylmkb{0:14494} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.52:6824/2352841895,v1:172.21.15.52:6825/2352841895] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T02:06:15.692 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:06:15.692 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:06:15.692 INFO:teuthology.orchestra.run.smithi029.stdout:Standby daemons: 2024-04-27T02:06:15.692 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-27T02:06:15.692 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi029.dyrjrd{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.29:6826/3428808494,v1:172.21.15.29:6827/3428808494] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T02:06:15.692 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.cephfs.smithi052.nbjpjd{-1:24325} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.52:6826/3513632158,v1:172.21.15.52:6827/3513632158] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T02:06:15.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.712+0000 7f85f9438700 1 -- 172.21.15.29:0/3423607635 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f85dc06c680 msgr2=0x7f85dc06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:15.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.712+0000 7f85f9438700 1 --2- 172.21.15.29:0/3423607635 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f85dc06c680 0x7f85dc06eb30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f85e8009fd0 tx=0x7f85e8005fd0 comp rx=0 tx=0).stop 2024-04-27T02:06:15.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.712+0000 7f85f9438700 1 -- 172.21.15.29:0/3423607635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85f4105710 msgr2=0x7f85f4197cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:15.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.712+0000 7f85f9438700 1 --2- 172.21.15.29:0/3423607635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85f4105710 0x7f85f4197cf0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f85e400e8f0 tx=0x7f85e400ec00 comp rx=0 tx=0).stop 2024-04-27T02:06:15.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.712+0000 7f85f9438700 1 -- 172.21.15.29:0/3423607635 shutdown_connections 2024-04-27T02:06:15.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.712+0000 7f85f9438700 1 --2- 172.21.15.29:0/3423607635 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f85dc06c680 0x7f85dc06eb30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:15.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.712+0000 7f85f9438700 1 --2- 172.21.15.29:0/3423607635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85f4105710 0x7f85f4197cf0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:15.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.712+0000 7f85f9438700 1 --2- 172.21.15.29:0/3423607635 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85f4198230 0x7f85f419d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:15.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.712+0000 7f85f9438700 1 -- 172.21.15.29:0/3423607635 >> 172.21.15.29:0/3423607635 conn(0x7f85f40fa780 msgr2=0x7f85f40fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:06:15.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.712+0000 7f85f9438700 1 -- 172.21.15.29:0/3423607635 shutdown_connections 2024-04-27T02:06:15.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.712+0000 7f85f9438700 1 -- 172.21.15.29:0/3423607635 wait complete. 2024-04-27T02:06:15.694 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 19 2024-04-27T02:06:15.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.825+0000 7f34069e9700 1 -- 172.21.15.29:0/4013832453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3400100350 msgr2=0x7f34001007a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:15.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.825+0000 7f34069e9700 1 --2- 172.21.15.29:0/4013832453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3400100350 0x7f34001007a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f33f4009910 tx=0x7f33f4009c20 comp rx=0 tx=0).stop 2024-04-27T02:06:15.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.825+0000 7f34069e9700 1 -- 172.21.15.29:0/4013832453 shutdown_connections 2024-04-27T02:06:15.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.825+0000 7f34069e9700 1 --2- 172.21.15.29:0/4013832453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3400100350 0x7f34001007a0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:15.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.825+0000 7f34069e9700 1 --2- 172.21.15.29:0/4013832453 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34000ff150 0x7f34000ff560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:15.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.825+0000 7f34069e9700 1 -- 172.21.15.29:0/4013832453 >> 172.21.15.29:0/4013832453 conn(0x7f34000fa700 msgr2=0x7f34000fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:06:15.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.825+0000 7f34069e9700 1 -- 172.21.15.29:0/4013832453 shutdown_connections 2024-04-27T02:06:15.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.825+0000 7f34069e9700 1 -- 172.21.15.29:0/4013832453 wait complete. 2024-04-27T02:06:15.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.826+0000 7f34069e9700 1 Processor -- start 2024-04-27T02:06:15.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.826+0000 7f34069e9700 1 -- start start 2024-04-27T02:06:15.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.826+0000 7f34069e9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34000ff150 0x7f3400195aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:06:15.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.826+0000 7f34069e9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3400100350 0x7f3400195fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:06:15.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.826+0000 7f34069e9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3400196600 con 0x7f3400100350 2024-04-27T02:06:15.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.826+0000 7f34069e9700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3400196740 con 0x7f34000ff150 2024-04-27T02:06:15.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.826+0000 7f33ff7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3400100350 0x7f3400195fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:06:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.826+0000 7f33ff7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3400100350 0x7f3400195fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35866/0 (socket says 172.21.15.29:35866) 2024-04-27T02:06:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.826+0000 7f33ff7fe700 1 -- 172.21.15.29:0/1281421338 learned_addr learned my addr 172.21.15.29:0/1281421338 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:06:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.826+0000 7f33fffff700 1 --2- 172.21.15.29:0/1281421338 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34000ff150 0x7f3400195aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:06:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.827+0000 7f33ff7fe700 1 -- 172.21.15.29:0/1281421338 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34000ff150 msgr2=0x7f3400195aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.827+0000 7f33ff7fe700 1 --2- 172.21.15.29:0/1281421338 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34000ff150 0x7f3400195aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.827+0000 7f33ff7fe700 1 -- 172.21.15.29:0/1281421338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f40095f0 con 0x7f3400100350 2024-04-27T02:06:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.827+0000 7f33ff7fe700 1 --2- 172.21.15.29:0/1281421338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3400100350 0x7f3400195fe0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f33f40092e0 tx=0x7f33f400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:06:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.827+0000 7f33fd7fa700 1 -- 172.21.15.29:0/1281421338 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33f401d070 con 0x7f3400100350 2024-04-27T02:06:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.827+0000 7f33fd7fa700 1 -- 172.21.15.29:0/1281421338 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f33f4005dc0 con 0x7f3400100350 2024-04-27T02:06:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.827+0000 7f34069e9700 1 -- 172.21.15.29:0/1281421338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f340019b190 con 0x7f3400100350 2024-04-27T02:06:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.827+0000 7f33fd7fa700 1 -- 172.21.15.29:0/1281421338 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33f4017630 con 0x7f3400100350 2024-04-27T02:06:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.827+0000 7f34069e9700 1 -- 172.21.15.29:0/1281421338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f340019b680 con 0x7f3400100350 2024-04-27T02:06:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.828+0000 7f34069e9700 1 -- 172.21.15.29:0/1281421338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3400107ca0 con 0x7f3400100350 2024-04-27T02:06:15.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.829+0000 7f33fd7fa700 1 -- 172.21.15.29:0/1281421338 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f33f400faf0 con 0x7f3400100350 2024-04-27T02:06:15.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.830+0000 7f33fd7fa700 1 --2- 172.21.15.29:0/1281421338 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f33e806c680 0x7f33e806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:06:15.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.830+0000 7f33fd7fa700 1 -- 172.21.15.29:0/1281421338 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f33f408cda0 con 0x7f3400100350 2024-04-27T02:06:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.830+0000 7f33fffff700 1 --2- 172.21.15.29:0/1281421338 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f33e806c680 0x7f33e806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:06:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.830+0000 7f33fffff700 1 --2- 172.21.15.29:0/1281421338 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f33e806c680 0x7f33e806eb30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f34001001b0 tx=0x7f33f0009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:06:15.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:15.838+0000 7f33fd7fa700 1 -- 172.21.15.29:0/1281421338 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f33f405ae10 con 0x7f3400100350 2024-04-27T02:06:15.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:06:15 smithi029 ceph-mon[26769]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:06:15.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:06:15 smithi029 ceph-mon[26769]: from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:06:15.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:06:15 smithi029 ceph-mon[26769]: from='client.24435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:06:15.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:06:15 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/576526052' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:06:15.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:06:15 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/3423607635' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T02:06:16.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.089+0000 7f34069e9700 1 -- 172.21.15.29:0/1281421338 --> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f340019bd80 con 0x7f33e806c680 2024-04-27T02:06:16.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.090+0000 7f33fd7fa700 1 -- 172.21.15.29:0/1281421338 <== mgr.14215 v2:172.21.15.29:6800/298222579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f340019bd80 con 0x7f33e806c680 2024-04-27T02:06:16.071 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-04-27T02:06:16.071 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:06:16.071 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-04-27T02:06:16.071 INFO:teuthology.orchestra.run.smithi029.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:06:16.071 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [], 2024-04-27T02:06:16.071 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "0/23 daemons upgraded", 2024-04-27T02:06:16.071 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 image on host smithi052", 2024-04-27T02:06:16.071 INFO:teuthology.orchestra.run.smithi029.stdout: "is_paused": false 2024-04-27T02:06:16.071 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-04-27T02:06:16.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.093+0000 7f34069e9700 1 -- 172.21.15.29:0/1281421338 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f33e806c680 msgr2=0x7f33e806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:16.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.093+0000 7f34069e9700 1 --2- 172.21.15.29:0/1281421338 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f33e806c680 0x7f33e806eb30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f34001001b0 tx=0x7f33f0009260 comp rx=0 tx=0).stop 2024-04-27T02:06:16.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.093+0000 7f34069e9700 1 -- 172.21.15.29:0/1281421338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3400100350 msgr2=0x7f3400195fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:16.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.093+0000 7f34069e9700 1 --2- 172.21.15.29:0/1281421338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3400100350 0x7f3400195fe0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f33f40092e0 tx=0x7f33f400f690 comp rx=0 tx=0).stop 2024-04-27T02:06:16.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.093+0000 7f34069e9700 1 -- 172.21.15.29:0/1281421338 shutdown_connections 2024-04-27T02:06:16.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.093+0000 7f34069e9700 1 --2- 172.21.15.29:0/1281421338 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f33e806c680 0x7f33e806eb30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:16.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.093+0000 7f34069e9700 1 --2- 172.21.15.29:0/1281421338 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34000ff150 0x7f3400195aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:16.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.093+0000 7f34069e9700 1 --2- 172.21.15.29:0/1281421338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3400100350 0x7f3400195fe0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:16.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.093+0000 7f34069e9700 1 -- 172.21.15.29:0/1281421338 >> 172.21.15.29:0/1281421338 conn(0x7f34000fa700 msgr2=0x7f3400068dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:06:16.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.093+0000 7f34069e9700 1 -- 172.21.15.29:0/1281421338 shutdown_connections 2024-04-27T02:06:16.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.093+0000 7f34069e9700 1 -- 172.21.15.29:0/1281421338 wait complete. 2024-04-27T02:06:16.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.209+0000 7f7701264700 1 -- 172.21.15.29:0/840171364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76fc0ff130 msgr2=0x7f76fc0ff540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:16.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.209+0000 7f7701264700 1 --2- 172.21.15.29:0/840171364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76fc0ff130 0x7f76fc0ff540 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f76ec009910 tx=0x7f76ec009c20 comp rx=0 tx=0).stop 2024-04-27T02:06:16.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.209+0000 7f7701264700 1 -- 172.21.15.29:0/840171364 shutdown_connections 2024-04-27T02:06:16.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.209+0000 7f7701264700 1 --2- 172.21.15.29:0/840171364 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76fc1003d0 0x7f76fc100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:16.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.209+0000 7f7701264700 1 --2- 172.21.15.29:0/840171364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76fc0ff130 0x7f76fc0ff540 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:16.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.209+0000 7f7701264700 1 -- 172.21.15.29:0/840171364 >> 172.21.15.29:0/840171364 conn(0x7f76fc0fa780 msgr2=0x7f76fc0fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:06:16.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.209+0000 7f7701264700 1 -- 172.21.15.29:0/840171364 shutdown_connections 2024-04-27T02:06:16.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.209+0000 7f7701264700 1 -- 172.21.15.29:0/840171364 wait complete. 2024-04-27T02:06:16.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.210+0000 7f7701264700 1 Processor -- start 2024-04-27T02:06:16.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.210+0000 7f7701264700 1 -- start start 2024-04-27T02:06:16.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.210+0000 7f7701264700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76fc0ff130 0x7f76fc0690a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:06:16.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.210+0000 7f7701264700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76fc1003d0 0x7f76fc19a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:06:16.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.210+0000 7f7701264700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76fc19ade0 con 0x7f76fc1003d0 2024-04-27T02:06:16.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.210+0000 7f7701264700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76fc19af20 con 0x7f76fc0ff130 2024-04-27T02:06:16.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.210+0000 7f76fa7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76fc1003d0 0x7f76fc19a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:06:16.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.211+0000 7f76fa7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76fc1003d0 0x7f76fc19a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35874/0 (socket says 172.21.15.29:35874) 2024-04-27T02:06:16.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.211+0000 7f76fa7fc700 1 -- 172.21.15.29:0/3041339207 learned_addr learned my addr 172.21.15.29:0/3041339207 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-27T02:06:16.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.211+0000 7f76faffd700 1 --2- 172.21.15.29:0/3041339207 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76fc0ff130 0x7f76fc0690a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:06:16.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.211+0000 7f76fa7fc700 1 -- 172.21.15.29:0/3041339207 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76fc0ff130 msgr2=0x7f76fc0690a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:16.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.211+0000 7f76fa7fc700 1 --2- 172.21.15.29:0/3041339207 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76fc0ff130 0x7f76fc0690a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:16.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.211+0000 7f76fa7fc700 1 -- 172.21.15.29:0/3041339207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec0095f0 con 0x7f76fc1003d0 2024-04-27T02:06:16.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.211+0000 7f76fa7fc700 1 --2- 172.21.15.29:0/3041339207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76fc1003d0 0x7f76fc19a810 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f76f000d8d0 tx=0x7f76f000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:06:16.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.211+0000 7f76e3fff700 1 -- 172.21.15.29:0/3041339207 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76f00098a0 con 0x7f76fc1003d0 2024-04-27T02:06:16.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.211+0000 7f7701264700 1 -- 172.21.15.29:0/3041339207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76fc19b200 con 0x7f76fc1003d0 2024-04-27T02:06:16.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.211+0000 7f76e3fff700 1 -- 172.21.15.29:0/3041339207 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f76f0010460 con 0x7f76fc1003d0 2024-04-27T02:06:16.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.211+0000 7f7701264700 1 -- 172.21.15.29:0/3041339207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76fc19b750 con 0x7f76fc1003d0 2024-04-27T02:06:16.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.211+0000 7f76e3fff700 1 -- 172.21.15.29:0/3041339207 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76f000f5f0 con 0x7f76fc1003d0 2024-04-27T02:06:16.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.212+0000 7f7701264700 1 -- 172.21.15.29:0/3041339207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76fc107c10 con 0x7f76fc1003d0 2024-04-27T02:06:16.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.213+0000 7f76e3fff700 1 -- 172.21.15.29:0/3041339207 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f76f0009a00 con 0x7f76fc1003d0 2024-04-27T02:06:16.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.214+0000 7f76e3fff700 1 --2- 172.21.15.29:0/3041339207 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f76e406c680 0x7f76e406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T02:06:16.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.214+0000 7f76e3fff700 1 -- 172.21.15.29:0/3041339207 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f76f008b7c0 con 0x7f76fc1003d0 2024-04-27T02:06:16.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.214+0000 7f76faffd700 1 --2- 172.21.15.29:0/3041339207 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f76e406c680 0x7f76e406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T02:06:16.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.214+0000 7f76faffd700 1 --2- 172.21.15.29:0/3041339207 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f76e406c680 0x7f76e406eb30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f76ec00b5c0 tx=0x7f76ec005ee0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T02:06:16.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.218+0000 7f76e3fff700 1 -- 172.21.15.29:0/3041339207 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f76f00598d0 con 0x7f76fc1003d0 2024-04-27T02:06:16.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:06:15 smithi052 ceph-mon[32586]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:06:16.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:06:15 smithi052 ceph-mon[32586]: from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:06:16.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:06:15 smithi052 ceph-mon[32586]: from='client.24435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:06:16.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:06:15 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/576526052' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:06:16.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:06:15 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/3423607635' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T02:06:16.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.509+0000 7f7701264700 1 -- 172.21.15.29:0/3041339207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f76fc107e30 con 0x7f76fc1003d0 2024-04-27T02:06:16.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.510+0000 7f76e3fff700 1 -- 172.21.15.29:0/3041339207 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f76f00160e0 con 0x7f76fc1003d0 2024-04-27T02:06:16.490 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-27T02:06:16.490 INFO:teuthology.orchestra.run.smithi029.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-27T02:06:16.490 INFO:teuthology.orchestra.run.smithi029.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-27T02:06:16.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.512+0000 7f7701264700 1 -- 172.21.15.29:0/3041339207 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f76e406c680 msgr2=0x7f76e406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:16.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.512+0000 7f7701264700 1 --2- 172.21.15.29:0/3041339207 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f76e406c680 0x7f76e406eb30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f76ec00b5c0 tx=0x7f76ec005ee0 comp rx=0 tx=0).stop 2024-04-27T02:06:16.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.512+0000 7f7701264700 1 -- 172.21.15.29:0/3041339207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76fc1003d0 msgr2=0x7f76fc19a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T02:06:16.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.512+0000 7f7701264700 1 --2- 172.21.15.29:0/3041339207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76fc1003d0 0x7f76fc19a810 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f76f000d8d0 tx=0x7f76f000dbe0 comp rx=0 tx=0).stop 2024-04-27T02:06:16.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.512+0000 7f7701264700 1 -- 172.21.15.29:0/3041339207 shutdown_connections 2024-04-27T02:06:16.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.512+0000 7f7701264700 1 --2- 172.21.15.29:0/3041339207 >> [v2:172.21.15.29:6800/298222579,v1:172.21.15.29:6801/298222579] conn(0x7f76e406c680 0x7f76e406eb30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:16.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.512+0000 7f7701264700 1 --2- 172.21.15.29:0/3041339207 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76fc0ff130 0x7f76fc0690a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:16.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.512+0000 7f7701264700 1 --2- 172.21.15.29:0/3041339207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76fc1003d0 0x7f76fc19a810 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T02:06:16.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.512+0000 7f7701264700 1 -- 172.21.15.29:0/3041339207 >> 172.21.15.29:0/3041339207 conn(0x7f76fc0fa780 msgr2=0x7f76fc103600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T02:06:16.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.513+0000 7f7701264700 1 -- 172.21.15.29:0/3041339207 shutdown_connections 2024-04-27T02:06:16.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:06:16.513+0000 7f7701264700 1 -- 172.21.15.29:0/3041339207 wait complete. 2024-04-27T02:06:17.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:06:16 smithi029 ceph-mon[26769]: pgmap v248: 65 pgs: 65 active+clean; 33 MiB data, 278 MiB used, 536 GiB / 536 GiB avail; 683 KiB/s rd, 2.9 MiB/s wr, 257 op/s 2024-04-27T02:06:17.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:06:16 smithi029 ceph-mon[26769]: from='client.? 172.21.15.29:0/3041339207' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:06:17.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:06:16 smithi052 ceph-mon[32586]: pgmap v248: 65 pgs: 65 active+clean; 33 MiB data, 278 MiB used, 536 GiB / 536 GiB avail; 683 KiB/s rd, 2.9 MiB/s wr, 257 op/s 2024-04-27T02:06:17.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:06:16 smithi052 ceph-mon[32586]: from='client.? 172.21.15.29:0/3041339207' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:06:17.441 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-27T02:06:18.060 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-27T02:06:18.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:06:17 smithi029 ceph-mon[26769]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:06:18.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:06:17 smithi052 ceph-mon[32586]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:06:18.333 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-27T02:06:18.335 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-27T02:06:18.335 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-27T02:06:18.560 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-27T02:06:18.564 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-27T02:06:18.564 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-27T02:06:18.711 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-04-27T02:06:18.712 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-27T02:06:18.712 DEBUG:teuthology.orchestra.run.smithi029:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b22e2ebdeb24376882b7bda2a7329c8cccc2276a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-04-27T02:06:18.798 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir -p fsstress 2024-04-27T02:06:18.800 INFO:tasks.workunit.client.0.smithi029.stderr:+ pushd fsstress 2024-04-27T02:06:18.800 INFO:tasks.workunit.client.0.smithi029.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-27T02:06:18.800 INFO:tasks.workunit.client.0.smithi029.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-27T02:06:18.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:06:18 smithi029 ceph-mon[26769]: pgmap v249: 65 pgs: 65 active+clean; 41 MiB data, 298 MiB used, 536 GiB / 536 GiB avail; 1.3 MiB/s rd, 3.5 MiB/s wr, 267 op/s 2024-04-27T02:06:19.253 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Apr 27 02:06:18 smithi052 ceph-mon[32586]: pgmap v249: 65 pgs: 65 active+clean; 41 MiB data, 298 MiB used, 536 GiB / 536 GiB avail; 1.3 MiB/s rd, 3.5 MiB/s wr, 267 op/s 2024-04-27T02:06:19.707 INFO:tasks.workunit.client.1.smithi052.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-27T02:06:19.707 INFO:tasks.workunit.client.1.smithi052.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-27T02:06:19.708 INFO:tasks.workunit.client.1.smithi052.stderr:+ make 2024-04-27T02:06:19.766 INFO:tasks.workunit.client.0.smithi029.stderr:+ tar xzf ltp-full.tgz 2024-04-27T02:06:19.771 INFO:tasks.workunit.client.1.smithi052.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-27T02:06:21.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:06:20 smithi029 ceph-mon[26769]: pgmap v250: 65 pgs: 65 active+clean; 42 MiB data, 306 MiB used, 536 GiB / 536 GiB avail; 1.7 MiB/s rd, 3.6 MiB/s wr, 266 op/s 2024-04-27T02:06:41.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:41.459+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:41.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:41.459+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:41.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:41.459+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:42.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:42.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:42.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:42.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:42.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:42.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:42.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:42.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:42.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:42.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:42.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:42.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:43.490 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:43.250+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:43.490 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:43.250+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:43.490 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:43.250+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:43.490 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:43.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:43.491 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:43.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:43.491 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:43.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:43.491 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:43.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:43.491 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:43.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:43.491 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:43.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:44.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:44.221+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:44.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:44.221+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:44.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:44.221+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:44.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:44.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:44.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:44.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:44.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:44.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:44.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:44.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:44.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:44.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:44.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:44.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:45.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:45.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:45.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:45.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:45.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:45.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:45.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:45.239+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:45.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:45.239+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:45.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:45.239+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:45.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:45.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:45.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:45.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:45.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:45.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:46.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:46.362+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:46.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:46.362+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:46.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:46.362+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:46.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:46.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:46.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:46.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:46.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:46.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:46.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:46.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:46.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:46.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:46.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:46.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:47.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:47.350+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:47.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:47.350+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:47.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:47.350+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:47.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:47.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:47.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:47.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:47.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:47.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:47.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:47.260+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:47.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:47.260+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:47.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:47.260+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:48.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:48.311+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:48.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:48.311+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:48.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:48.311+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:48.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:48.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:48.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:48.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:48.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:48.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:48.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:48.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:48.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:48.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:48.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:48.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:49.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:49.320+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:49.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:49.320+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:49.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:49.321+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:49.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:49.227+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:49.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:49.227+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:49.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:49.227+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:49.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:49.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:49.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:49.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:49.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:49.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:50.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:50.232+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:50.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:50.232+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:50.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:50.232+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:50.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:50.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:50.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:50.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:50.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:50.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:50.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:50.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:50.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:50.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:50.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:50.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:51.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:51.410+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:51.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:51.410+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:51.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:51.410+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:51.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:51.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:51.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:51.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:51.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:51.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:51.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:51.279+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:51.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:51.279+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:51.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:51.279+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:52.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:52.275+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:52.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:52.275+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:52.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:52.275+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:52.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:52.414+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:52.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:52.414+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:52.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:52.414+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:52.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:52.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:52.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:52.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:52.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:52.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:53.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:53.303+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:53.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:53.303+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:53.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:53.303+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:53.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:53.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:53.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:53.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:53.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:53.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:53.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:53.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:53.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:53.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:53.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:53.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:54.419 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:54.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:54.420 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:54.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:54.420 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:54.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:54.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:54.439+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:54.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:54.439+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:54.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:54.439+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:54.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:54.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:54.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:54.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:54.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:54.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:55.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:55.333+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:55.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:55.333+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:55.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:55.333+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:55.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:55.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:55.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:55.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:55.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:55.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:55.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:55.462+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:55.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:55.462+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:55.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:55.462+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:56.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:56.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:56.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:56.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:56.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:56.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:56.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:56.460+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:56.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:56.460+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:56.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:56.460+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:56.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:56.310+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:56.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:56.310+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:56.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:56.310+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:57.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:57.443+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:57.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:57.443+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:57.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:57.443+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:57.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:57.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:57.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:57.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:57.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:57.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:57.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:57.265+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:57.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:57.265+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:57.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:57.265+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:58.702 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:58.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:58.702 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:58.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:58.702 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:58.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:58.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:58.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:58.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:58.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:58.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:58.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:58.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:58.279+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:58.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:58.279+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:58.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:58.279+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:59.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:59.435+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:59.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:59.435+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:59.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:06:59.435+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:06:59.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:59.236+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:59.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:59.236+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:59.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:06:59.236+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:06:59.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:59.490+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:59.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:59.490+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:06:59.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:06:59.490+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:00.518 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:00.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:00.518 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:00.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:00.518 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:00.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:00.518 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:00.258+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:00.518 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:00.258+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:00.518 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:00.258+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:00.875 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:00.538+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:00.875 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:00.538+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:00.875 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:00.538+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:01.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:01.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:01.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:01.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:01.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:01.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:01.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:01.274+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:01.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:01.274+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:01.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:01.274+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:01.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:01.416+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:01.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:01.416+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:01.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:01.416+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:02.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:02.288+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:02.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:02.288+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:02.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:02.288+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:02.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:02.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:02.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:02.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:02.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:02.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:02.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:02.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:02.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:02.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:02.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:02.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:03.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:03.278+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:03.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:03.278+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:03.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:03.278+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:03.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:03.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:03.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:03.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:03.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:03.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:03.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:03.513+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:03.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:03.513+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:03.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:03.513+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:04.496 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:04.372+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:04.497 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:04.372+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:04.497 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:04.372+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:04.497 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:04.263+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:04.497 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:04.263+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:04.497 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:04.263+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:04.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:04.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:04.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:04.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:04.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:04.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:05.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:05.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:05.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:05.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:05.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:05.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:05.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:05.487+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:05.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:05.487+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:05.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:05.487+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:05.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:05.326+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:05.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:05.326+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:05.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:05.326+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:06.483 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:06.238+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:06.483 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:06.238+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:06.483 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:06.238+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:06.483 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:06.312+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:06.483 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:06.312+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:06.483 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:06.312+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:06.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:06.503+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:06.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:06.503+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:06.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:06.503+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:07.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:07.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:07.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:07.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:07.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:07.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:07.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:07.218+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:07.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:07.218+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:07.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:07.218+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:07.451 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:07.345+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:07.451 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:07.345+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:07.451 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:07.345+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:08.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:08.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:08.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:08.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:08.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:08.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:08.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:08.254+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:08.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:08.254+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:08.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:08.254+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:08.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:08.376+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:08.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:08.376+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:08.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:08.376+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:09.484 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:09.234+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:09.485 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:09.234+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:09.485 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:09.234+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:09.485 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:09.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:09.485 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:09.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:09.485 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:09.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:09.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:09.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:09.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:09.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:09.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:09.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:10.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:10.226+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:10.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:10.226+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:10.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:10.226+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:10.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:10.362+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:10.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:10.362+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:10.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:10.362+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:10.868 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:10.487+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:10.869 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:10.487+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:10.869 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:10.487+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:11.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:11.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:11.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:11.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:11.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:11.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:11.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:11.197+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:11.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:11.197+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:11.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:11.197+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:11.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:11.472+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:11.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:11.472+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:11.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:11.472+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:12.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:12.374+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:12.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:12.374+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:12.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:12.374+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:12.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:12.463+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:12.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:12.463+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:12.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:12.463+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:12.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:12.223+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:12.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:12.223+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:12.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:12.223+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:13.492 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:13.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:13.492 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:13.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:13.492 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:13.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:13.492 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:13.238+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:13.493 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:13.238+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:13.493 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:13.238+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:13.493 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:13.400+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:13.493 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:13.400+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:13.493 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:13.400+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:14.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:14.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:14.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:14.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:14.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:14.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:14.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:14.430+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:14.451 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:14.430+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:14.451 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:14.430+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:14.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:14.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:14.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:14.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:14.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:14.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:15.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:15.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:15.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:15.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:15.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:15.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:15.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:15.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:15.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:15.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:15.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:15.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:15.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:15.231+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:15.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:15.231+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:15.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:15.231+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:16.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:16.392+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:16.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:16.392+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:16.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:16.392+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:16.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:16.213+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:16.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:16.213+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:16.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:16.213+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:16.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:16.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:16.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:16.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:16.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:16.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:17.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:17.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:17.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:17.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:17.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:17.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:17.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:17.422+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:17.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:17.422+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:17.451 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:17.422+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:17.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:17.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:17.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:17.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:17.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:17.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:18.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:18.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:18.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:18.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:18.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:18.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:18.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:18.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:18.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:18.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:18.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:18.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:18.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:18.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:18.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:18.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:18.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:18.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:19.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:19.453+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:19.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:19.453+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:19.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:19.453+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:19.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:19.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:19.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:19.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:19.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:19.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:19.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:19.209+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:19.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:19.209+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:19.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:19.209+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:20.426 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:20.188+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:20.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:20.188+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:20.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:20.188+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:20.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:20.327+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:20.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:20.327+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:20.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:20.327+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:20.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:20.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:20.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:20.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:20.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:20.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:21.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:21.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:21.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:21.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:21.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:21.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:21.410 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:21.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:21.410 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:21.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:21.410 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:21.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:21.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:21.430+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:21.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:21.430+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:21.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:21.430+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:22.414 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:22.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:22.414 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:22.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:22.414 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:22.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:22.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:22.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:22.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:22.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:22.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:22.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:22.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:22.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:22.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:22.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:22.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:22.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:23.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:23.144+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:23.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:23.144+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:23.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:23.144+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:23.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:23.298+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:23.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:23.298+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:23.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:23.298+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:23.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:23.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:23.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:23.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:23.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:23.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:24.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:24.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:24.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:24.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:24.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:24.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:24.403 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:24.268+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:24.403 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:24.268+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:24.403 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:24.268+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:24.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:24.422+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:24.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:24.422+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:24.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:24.422+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:25.413 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:25.083+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:25.413 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:25.083+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:25.413 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:25.083+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:25.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:25.256+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:25.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:25.256+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:25.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:25.256+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:25.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:25.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:25.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:25.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:25.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:25.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:26.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:26.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:26.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:26.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:26.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:26.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:26.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:26.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:26.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:26.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:26.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:26.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:26.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:26.306+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:26.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:26.306+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:26.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:26.306+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:27.389 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:27.125+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:27.390 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:27.125+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:27.390 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:27.125+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:27.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:27.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:27.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:27.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:27.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:27.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:27.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:27.410+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:27.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:27.410+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:27.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:27.410+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:28.327 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:28.077+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:28.327 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:28.077+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:28.327 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:28.077+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:28.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:28.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:28.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:28.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:28.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:28.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:28.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:28.432+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:28.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:28.432+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:28.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:28.432+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:29.314 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:29.084+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:29.314 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:29.084+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:29.314 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:29.084+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:29.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:29.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:29.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:29.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:29.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:29.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:29.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:29.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:29.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:29.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:29.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:29.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:30.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:30.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:30.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:30.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:30.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:30.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:30.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:30.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:30.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:30.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:30.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:30.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:30.880 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:30.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:30.880 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:30.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:30.880 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:30.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:31.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:31.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:31.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:31.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:31.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:31.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:31.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:31.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:31.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:31.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:31.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:31.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:31.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:31.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:31.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:31.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:31.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:31.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:32.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:32.080+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:32.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:32.080+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:32.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:32.080+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:32.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:32.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:32.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:32.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:32.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:32.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:32.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:32.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:32.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:32.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:32.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:32.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:33.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:33.100+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:33.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:33.100+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:33.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:33.100+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:33.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:33.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:33.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:33.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:33.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:33.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:33.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:33.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:33.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:33.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:33.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:33.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:34.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:34.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:34.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:34.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:34.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:34.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:34.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:34.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:34.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:34.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:34.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:34.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:34.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:34.510+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:34.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:34.510+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:34.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:34.510+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:35.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:35.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:35.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:35.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:35.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:35.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:35.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:35.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:35.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:35.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:35.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:35.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:35.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:35.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:35.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:35.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:35.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:35.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:36.130 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:36.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:36.131 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:36.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:36.131 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:36.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:36.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:36.317+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:36.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:36.317+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:36.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:36.317+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:36.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:36.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:36.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:36.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:36.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:36.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:37.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:37.152+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:37.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:37.152+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:37.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:37.152+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:37.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:37.271+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:37.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:37.271+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:37.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:37.271+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:37.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:37.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:37.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:37.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:37.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:37.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:38.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:38.201+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:38.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:38.201+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:38.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:38.201+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:38.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:38.392+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:38.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:38.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:38.451 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:38.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:38.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:38.258+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:38.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:38.258+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:38.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:38.258+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:39.555 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:39.238+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:39.556 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:39.238+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:39.556 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:39.238+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:39.556 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:39.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:39.556 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:39.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:39.556 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:39.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:39.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:39.268+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:39.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:39.268+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:39.557 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:39.268+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:40.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:40.391+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:40.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:40.391+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:40.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:40.391+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:40.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:40.263+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:40.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:40.263+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:40.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:40.263+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:40.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:40.261+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:40.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:40.261+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:40.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:40.261+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:41.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:41.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:41.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:41.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:41.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:41.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:41.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:41.256+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:41.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:41.256+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:41.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:41.256+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:41.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:41.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:41.451 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:41.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:41.451 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:41.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:42.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:42.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:42.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:42.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:42.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:42.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:42.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:42.259+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:42.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:42.259+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:42.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:42.260+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:42.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:42.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:42.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:42.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:42.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:42.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:43.494 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:43.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:43.494 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:43.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:43.494 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:43.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:43.495 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:43.244+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:43.495 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:43.244+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:43.495 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:43.244+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:43.495 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:43.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:43.495 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:43.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:43.495 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:43.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:44.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:44.416+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:44.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:44.416+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:44.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:44.416+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:44.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:44.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:44.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:44.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:44.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:44.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:44.604 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:44.248+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:44.604 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:44.248+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:44.604 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:44.248+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:45.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:45.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:45.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:45.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:45.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:45.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:45.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:45.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:45.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:45.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:45.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:45.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:45.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:45.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:45.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:45.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:45.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:45.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:46.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:46.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:46.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:46.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:46.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:46.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:46.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:46.266+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:46.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:46.266+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:46.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:46.266+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:46.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:46.447+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:46.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:46.447+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:46.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:46.447+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:47.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:47.224+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:47.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:47.224+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:47.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:47.225+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:47.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:47.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:47.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:47.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:47.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:47.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:47.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:47.241+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:47.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:47.241+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:47.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:47.241+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:48.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:48.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:48.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:48.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:48.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:48.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:48.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:48.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:48.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:48.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:48.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:48.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:48.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:48.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:48.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:48.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:48.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:48.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:49.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:49.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:49.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:49.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:49.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:49.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:49.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:49.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:49.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:49.389+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:49.451 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:49.389+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:49.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:49.219+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:49.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:49.219+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:49.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:49.219+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:50.410 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:50.223+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:50.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:50.223+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:50.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:50.223+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:50.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:50.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:50.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:50.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:50.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:50.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:50.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:50.431+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:50.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:50.431+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:50.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:50.431+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:51.441 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:51.196+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:51.441 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:51.196+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:51.441 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:51.196+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:51.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:51.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:51.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:51.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:51.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:51.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:51.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:51.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:51.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:51.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:51.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:51.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:52.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:52.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:52.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:52.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:52.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:52.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:52.431 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:52.172+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:52.431 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:52.172+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:52.431 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:52.172+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:52.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:52.451+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:52.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:52.451+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:52.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:52.451+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:53.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:53.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:53.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:53.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:53.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:53.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:53.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:53.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:53.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:53.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:53.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:53.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:53.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:53.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:53.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:53.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:53.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:53.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:54.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:54.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:54.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:54.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:54.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:54.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:54.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:54.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:54.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:54.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:54.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:54.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:54.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:54.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:54.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:54.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:54.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:54.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:55.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:55.163+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:55.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:55.163+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:55.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:55.163+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:55.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:55.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:55.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:55.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:55.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:55.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:55.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:55.503+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:55.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:55.503+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:55.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:55.503+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:56.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:56.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:56.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:56.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:56.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:56.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:56.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:56.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:56.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:56.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:56.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:56.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:56.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:56.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:56.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:56.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:56.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:56.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:57.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:57.136+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:57.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:57.136+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:57.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:57.136+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:57.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:57.063+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:57.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:57.063+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:57.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:57.063+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:57.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:57.487+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:57.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:57.487+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:57.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:57.487+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:58.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:58.092+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:58.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:58.092+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:58.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:58.092+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:58.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:58.030+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:58.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:58.030+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:58.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:58.030+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:58.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:58.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:58.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:58.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:58.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:58.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:59.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:59.028+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:59.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:59.028+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:59.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:59.028+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:07:59.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:59.115+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:59.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:59.115+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:59.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:07:59.115+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:07:59.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:59.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:59.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:59.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:07:59.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:07:59.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:00.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:59.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:00.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:59.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:00.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:07:59.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:00.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:00.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:00.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:00.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:00.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:00.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:00.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:00.525+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:00.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:00.525+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:00.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:00.525+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:01.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:00.960+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:01.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:00.960+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:01.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:00.960+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:01.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:01.136+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:01.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:01.136+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:01.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:01.136+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:01.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:01.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:01.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:01.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:01.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:01.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:02.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:01.987+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:02.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:01.987+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:02.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:01.987+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:02.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:02.119+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:02.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:02.119+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:02.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:02.119+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:02.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:02.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:02.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:02.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:02.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:02.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:03.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:02.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:03.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:02.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:03.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:02.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:03.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:03.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:03.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:03.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:03.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:03.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:03.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:03.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:03.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:03.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:03.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:03.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:04.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:04.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:04.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:04.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:04.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:04.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:04.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:04.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:04.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:04.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:04.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:04.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:04.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:04.601+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:04.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:04.601+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:04.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:04.601+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:05.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:04.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:05.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:04.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:05.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:04.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:05.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:05.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:05.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:05.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:05.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:05.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:05.873 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:05.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:05.873 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:05.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:05.873 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:05.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:06.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:06.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:06.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:06.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:06.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:06.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:06.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:06.005+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:06.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:06.005+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:06.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:06.005+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:06.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:06.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:06.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:06.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:06.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:06.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:07.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:07.134+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:07.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:07.134+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:07.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:07.134+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:07.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:07.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:07.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:07.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:07.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:07.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:07.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:07.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:07.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:07.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:07.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:07.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:08.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:08.116+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:08.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:08.116+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:08.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:08.116+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:08.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:08.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:08.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:08.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:08.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:08.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:08.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:08.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:08.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:08.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:08.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:08.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:09.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:08.978+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:09.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:08.978+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:09.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:08.978+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:09.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:09.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:09.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:09.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:09.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:09.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:09.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:09.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:09.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:09.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:09.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:09.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:10.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:10.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:10.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:10.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:10.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:10.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:10.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:09.970+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:10.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:09.970+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:10.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:09.970+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:10.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:10.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:10.885 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:10.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:10.885 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:10.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:11.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:10.940+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:11.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:10.940+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:11.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:10.940+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:11.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:11.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:11.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:11.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:11.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:11.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:11.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:11.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:11.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:11.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:11.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:11.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:12.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:12.081+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:12.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:12.081+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:12.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:12.081+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:12.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:11.965+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:12.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:11.965+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:12.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:11.965+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:12.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:12.514+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:12.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:12.514+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:12.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:12.514+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:13.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:13.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:13.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:13.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:13.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:13.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:13.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:12.952+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:13.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:12.952+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:13.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:12.952+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:13.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:13.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:13.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:13.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:13.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:13.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:14.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:14.102+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:14.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:14.102+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:14.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:14.102+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:14.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:13.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:14.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:13.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:14.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:13.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:14.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:14.481+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:14.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:14.481+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:14.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:14.481+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:15.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:14.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:15.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:14.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:15.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:14.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:15.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:15.109+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:15.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:15.109+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:15.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:15.109+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:15.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:15.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:15.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:15.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:15.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:15.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:16.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:16.017+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:16.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:16.017+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:16.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:16.017+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:16.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:16.134+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:16.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:16.134+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:16.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:16.134+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:16.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:16.476+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:16.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:16.476+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:16.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:16.476+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:17.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:17.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:17.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:17.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:17.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:17.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:17.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:17.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:17.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:17.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:17.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:17.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:17.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:17.503+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:17.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:17.503+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:17.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:17.503+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:18.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:18.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:18.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:18.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:18.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:18.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:18.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:18.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:18.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:18.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:18.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:18.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:18.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:18.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:18.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:18.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:18.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:18.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:19.398 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:19.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:19.399 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:19.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:19.399 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:19.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:19.399 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:19.191+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:19.399 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:19.191+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:19.399 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:19.191+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:19.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:19.419+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:19.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:19.419+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:19.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:19.419+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:20.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:19.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:20.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:19.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:20.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:19.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:20.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:20.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:20.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:20.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:20.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:20.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:20.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:20.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:20.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:20.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:20.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:20.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:21.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:20.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:21.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:20.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:21.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:20.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:21.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:21.125+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:21.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:21.125+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:21.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:21.125+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:21.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:21.514+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:21.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:21.514+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:21.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:21.514+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:22.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:22.021+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:22.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:22.021+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:22.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:22.021+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:22.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:22.117+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:22.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:22.117+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:22.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:22.117+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:22.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:22.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:22.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:22.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:22.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:22.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:23.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:23.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:23.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:23.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:23.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:23.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:23.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:23.076+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:23.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:23.076+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:23.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:23.076+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:23.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:23.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:23.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:23.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:23.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:23.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:24.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:24.104+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:24.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:24.104+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:24.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:24.104+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:24.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:24.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:24.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:24.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:24.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:24.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:24.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:24.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:24.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:24.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:24.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:24.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:25.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:25.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:25.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:25.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:25.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:25.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:25.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:25.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:25.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:25.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:25.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:25.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:25.875 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:25.561+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:25.875 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:25.561+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:25.875 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:25.561+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:26.199 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:08:25.895+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:08:26.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:26.086+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:26.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:26.086+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:26.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:26.086+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:26.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:25.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:26.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:25.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:26.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:25.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:26.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:26.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:26.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:26.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:26.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:26.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:27.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:27.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:27.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:27.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:27.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:27.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:27.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:27.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:27.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:27.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:27.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:27.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:27.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:27.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:27.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:27.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:27.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:27.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:28.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:28.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:28.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:28.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:28.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:28.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:28.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:28.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:28.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:28.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:28.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:28.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:28.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:28.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:28.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:28.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:28.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:28.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:29.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:29.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:29.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:29.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:29.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:29.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:29.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:28.986+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:29.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:28.986+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:29.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:28.986+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:29.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:29.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:29.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:29.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:29.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:29.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:30.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:30.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:30.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:30.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:30.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:30.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:30.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:29.979+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:30.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:29.979+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:30.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:29.979+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:30.949 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:08:30.895+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:08:30.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:30.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:30.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:30.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:30.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:30.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:31.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:31.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:31.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:31.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:31.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:31.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:31.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:30.986+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:31.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:30.986+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:31.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:30.986+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:31.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:31.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:31.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:31.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:31.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:31.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:32.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:32.045+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:32.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:32.045+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:32.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:32.045+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:32.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:31.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:32.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:31.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:32.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:31.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:32.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:32.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:32.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:32.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:32.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:32.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:33.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:33.059+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:33.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:33.059+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:33.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:33.059+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:33.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:32.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:33.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:32.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:33.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:32.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:33.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:33.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:33.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:33.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:33.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:33.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:34.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:33.907+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:34.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:33.907+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:34.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:33.907+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:34.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:34.045+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:34.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:34.045+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:34.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:34.045+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:34.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:34.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:34.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:34.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:34.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:34.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:35.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:35.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:35.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:35.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:35.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:35.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:35.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:34.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:35.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:34.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:35.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:34.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:35.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:35.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:35.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:35.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:35.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:35.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:36.199 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:08:35.895+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 31 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:08:36.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:35.983+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:36.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:35.983+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:36.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:35.983+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:36.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:35.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:36.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:35.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:36.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:35.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:36.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:36.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:36.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:36.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:36.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:36.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:37.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:37.003+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:37.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:37.003+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:37.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:37.003+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:37.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:36.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:37.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:36.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:37.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:36.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:37.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:37.639+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:37.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:37.639+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:37.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:37.639+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:38.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:37.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:38.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:37.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:38.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:37.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:38.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:37.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:38.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:37.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:38.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:37.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:38.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:38.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:38.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:38.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:38.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:38.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:39.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:38.975+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:39.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:38.975+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:39.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:38.975+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:39.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:38.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:39.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:38.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:39.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:38.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:39.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:39.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:39.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:39.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:39.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:39.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:40.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:39.949+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:40.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:39.949+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:40.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:39.949+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:40.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:40.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:40.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:40.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:40.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:40.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:40.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:40.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:40.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:40.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:40.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:40.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:41.199 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:08:40.895+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:08:41.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:40.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:41.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:40.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:41.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:40.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:41.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:41.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:41.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:41.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:41.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:41.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:41.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:41.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:41.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:41.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:41.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:41.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:42.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:41.965+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:42.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:41.965+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:42.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:41.965+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:42.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:42.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:42.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:42.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:42.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:42.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:42.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:42.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:42.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:42.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:42.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:42.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:43.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:42.988+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:43.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:42.988+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:43.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:42.988+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:43.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:43.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:43.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:43.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:43.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:43.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:43.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:43.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:43.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:43.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:43.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:43.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:44.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:43.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:44.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:43.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:44.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:43.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:44.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:44.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:44.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:44.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:44.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:44.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:44.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:44.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:44.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:44.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:44.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:44.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:45.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:44.960+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:45.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:44.960+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:45.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:44.960+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:45.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:45.035+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:45.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:45.035+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:45.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:45.035+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:45.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:45.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:45.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:45.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:45.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:45.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:46.184 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:08:45.897+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 90 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:08:46.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:46.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:46.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:46.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:46.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:46.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:46.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:46.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:46.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:46.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:46.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:46.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:46.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:46.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:46.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:46.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:46.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:46.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:47.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:47.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:47.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:47.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:47.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:47.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:47.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:46.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:47.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:46.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:47.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:46.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:47.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:47.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:47.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:47.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:47.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:47.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:48.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:48.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:48.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:48.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:48.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:48.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:48.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:48.025+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:48.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:48.025+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:48.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:48.025+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:48.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:48.611+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:48.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:48.611+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:48.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:48.611+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:49.257 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:49.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:49.257 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:49.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:49.257 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:49.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:49.258 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:48.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:49.258 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:48.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:49.258 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:48.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:49.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:49.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:49.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:49.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:49.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:49.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:50.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:50.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:50.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:50.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:50.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:50.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:50.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:49.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:50.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:49.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:50.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:49.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:50.889 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:08:50.897+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 118 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:08:50.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:50.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:50.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:50.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:50.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:50.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:51.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:51.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:51.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:51.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:51.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:51.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:51.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:51.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:51.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:51.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:51.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:51.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:51.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:51.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:51.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:51.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:51.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:51.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:52.329 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:52.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:52.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:52.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:52.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:52.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:52.330 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:51.997+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:52.330 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:51.997+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:52.330 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:51.997+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:52.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:52.635+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:52.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:52.635+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:52.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:52.635+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:53.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:53.047+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:53.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:53.047+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:53.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:53.047+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:53.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:53.031+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:53.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:53.031+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:53.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:53.031+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:53.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:53.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:53.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:53.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:53.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:53.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:54.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:54.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:54.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:54.039+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:54.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:54.039+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:54.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:53.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:54.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:53.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:54.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:53.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:54.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:54.603+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:54.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:54.603+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:54.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:54.603+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:55.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:55.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:55.272 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:55.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:55.272 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:55.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:55.272 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:55.032+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:55.272 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:55.032+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:55.272 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:55.032+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:55.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:55.605+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:55.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:55.605+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:55.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:55.605+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:56.199 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:08:55.898+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 145 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:08:56.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:56.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:56.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:56.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:56.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:56.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:56.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:56.097+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:56.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:56.097+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:56.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:56.097+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:56.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:56.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:56.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:56.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:56.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:56.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:57.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:57.044+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:57.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:57.044+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:57.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:57.044+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:57.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:57.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:57.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:57.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:57.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:57.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:57.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:57.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:57.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:57.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:57.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:57.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:58.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:58.094+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:58.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:58.094+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:58.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:58.094+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:58.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:58.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:58.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:58.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:58.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:58.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:58.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:58.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:58.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:58.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:58.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:58.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:59.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:59.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:59.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:59.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:59.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:08:59.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:08:59.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:59.175+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:59.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:59.175+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:59.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:08:59.175+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:08:59.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:59.690+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:59.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:59.690+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:08:59.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:08:59.690+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:00.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:00.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:00.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:00.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:00.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:00.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:00.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:00.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:00.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:00.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:00.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:00.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:00.949 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:09:00.898+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 176 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:09:00.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:00.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:00.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:00.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:00.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:00.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:01.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:01.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:01.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:01.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:01.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:01.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:01.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:01.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:01.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:01.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:01.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:01.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:01.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:01.730+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:01.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:01.730+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:01.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:01.730+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:02.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:02.102+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:02.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:02.102+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:02.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:02.102+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:02.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:02.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:02.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:02.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:02.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:02.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:03.061 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:02.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:03.061 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:02.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:03.061 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:02.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:03.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:03.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:03.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:03.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:03.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:03.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:03.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:03.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:03.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:03.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:03.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:03.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:04.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:03.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:04.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:03.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:04.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:03.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:04.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:04.048+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:04.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:04.048+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:04.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:04.048+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:04.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:04.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:04.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:04.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:04.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:04.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:04.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:04.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:04.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:04.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:04.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:04.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:05.364 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:05.005+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:05.365 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:05.005+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:05.365 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:05.005+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:05.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:05.109+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:05.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:05.109+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:05.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:05.109+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:06.002 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:09:05.899+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 204 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:09:06.003 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:05.732+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:06.003 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:05.732+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:06.003 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:05.732+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:06.348 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:06.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:06.348 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:06.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:06.348 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:06.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:06.348 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:06.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:06.348 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:06.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:06.349 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:06.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:07.034 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:06.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:07.034 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:06.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:07.034 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:06.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:07.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:07.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:07.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:07.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:07.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:07.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:07.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:07.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:07.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:07.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:07.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:07.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:08.014 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:07.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:08.014 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:07.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:08.015 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:07.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:08.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:08.035+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:08.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:08.035+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:08.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:08.035+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:08.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:08.107+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:08.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:08.107+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:08.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:08.107+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:09.035 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:08.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:09.035 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:08.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:09.035 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:08.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:09.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:09.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:09.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:09.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:09.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:09.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:09.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:09.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:09.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:09.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:09.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:09.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:10.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:09.780+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:10.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:09.780+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:10.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:09.780+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:10.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:10.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:10.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:10.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:10.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:10.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:10.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:10.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:10.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:10.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:10.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:10.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:11.076 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:09:10.899+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 231 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:09:11.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:10.774+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:11.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:10.774+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:11.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:10.774+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:11.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:11.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:11.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:11.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:11.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:11.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:11.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:11.125+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:11.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:11.125+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:11.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:11.125+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:12.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:11.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:12.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:11.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:12.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:11.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:12.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:12.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:12.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:12.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:12.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:12.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:12.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:12.080+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:12.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:12.080+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:12.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:12.080+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:13.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:12.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:13.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:12.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:13.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:12.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:13.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:13.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:13.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:13.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:13.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:13.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:13.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:13.087+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:13.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:13.087+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:13.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:13.087+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:13.832 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:13.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:13.832 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:13.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:13.832 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:13.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:14.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:14.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:14.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:14.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:14.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:14.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:14.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:14.128+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:14.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:14.128+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:14.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:14.128+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:15.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:14.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:15.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:14.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:15.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:14.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:15.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:15.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:15.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:15.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:15.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:15.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:15.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:15.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:15.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:15.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:15.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:15.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:16.034 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:09:15.900+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 258 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:09:16.035 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:15.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:16.035 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:15.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:16.035 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:15.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:16.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:16.104+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:16.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:16.104+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:16.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:16.104+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:16.346 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:16.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:16.346 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:16.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:16.346 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:16.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:17.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:16.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:17.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:16.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:17.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:16.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:17.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:17.142+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:17.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:17.142+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:17.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:17.142+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:17.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:17.073+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:17.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:17.073+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:17.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:17.073+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:17.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:17.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:17.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:17.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:17.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:17.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:18.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:18.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:18.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:18.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:18.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:18.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:18.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:18.105+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:18.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:18.105+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:18.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:18.105+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:19.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:18.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:19.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:18.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:19.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:18.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:19.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:19.094+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:19.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:19.094+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:19.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:19.094+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:19.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:19.086+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:19.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:19.086+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:19.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:19.086+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:20.035 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:19.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:20.035 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:19.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:20.035 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:19.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:20.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:20.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:20.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:20.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:20.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:20.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:20.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:20.062+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:20.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:20.062+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:20.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:20.062+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:20.880 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:20.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:20.880 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:20.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:20.880 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:20.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:21.199 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:09:20.900+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 288 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:09:21.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:21.082+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:21.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:21.082+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:21.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:21.082+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:21.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:21.084+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:21.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:21.084+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:21.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:21.084+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:22.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:21.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:22.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:21.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:22.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:21.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:22.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:22.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:22.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:22.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:22.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:22.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:22.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:22.122+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:22.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:22.122+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:22.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:22.122+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:23.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:23.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:23.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:23.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:23.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:23.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:23.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:22.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:23.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:22.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:23.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:22.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:23.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:23.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:23.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:23.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:23.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:23.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:24.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:23.814+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:24.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:23.814+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:24.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:23.814+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:24.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:23.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:24.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:23.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:24.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:23.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:24.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:24.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:24.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:24.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:24.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:24.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:25.133 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:24.779+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:25.133 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:24.779+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:25.133 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:24.779+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:25.133 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:24.780+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:25.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:24.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:25.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:24.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:25.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:24.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:25.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:24.936+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:25.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:25.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:25.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:25.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:25.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:25.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:25.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:25.155+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:25.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:25.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:25.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:25.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:25.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:25.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:25.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:25.787+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:26.199 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:09:25.901+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 317 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:09:26.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:25.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:26.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:25.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:26.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:25.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:26.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:25.943+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:26.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:26.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:26.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:26.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:26.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:26.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:26.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:26.135+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:27.074 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:26.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:27.074 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:26.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:27.074 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:26.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:27.074 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:26.988+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:27.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:26.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:27.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:26.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:27.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:26.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:27.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:26.758+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:27.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:27.095+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:27.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:27.095+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:27.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:27.095+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:27.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:27.096+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:28.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:27.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:28.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:27.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:28.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:27.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:28.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:27.765+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:28.110 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:27.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:28.111 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:27.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:28.111 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:27.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:28.111 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:27.973+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:28.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:28.130+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:28.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:28.130+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:28.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:28.130+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:28.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:28.131+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:29.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:28.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:29.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:28.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:29.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:28.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:29.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:28.803+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:29.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:28.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:29.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:28.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:29.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:28.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:29.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:28.988+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:29.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:29.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:29.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:29.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:29.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:29.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:29.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:29.177+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:30.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:29.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:30.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:29.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:30.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:29.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:30.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:29.776+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:30.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:29.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:30.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:29.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:30.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:29.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:30.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:29.966+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:30.604 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:30.227+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:30.604 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:30.227+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:30.604 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:30.227+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:30.605 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:30.228+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:30.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:30.808+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:30.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:30.808+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:30.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:30.808+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:30.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:30.809+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:31.178 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:09:30.901+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 349 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:09:31.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:30.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:31.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:30.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:31.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:30.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:31.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:30.988+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:31.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:31.199+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:31.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:31.199+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:31.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:31.199+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:31.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:31.200+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:32.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:31.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:32.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:31.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:32.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:31.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:32.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:31.805+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:32.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:31.977+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:32.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:31.977+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:32.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:31.977+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:32.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:31.977+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:32.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:32.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:32.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:32.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:32.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:32.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:32.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:32.234+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:33.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:32.814+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:33.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:32.814+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:33.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:32.814+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:33.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:32.815+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:33.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:32.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:33.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:32.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:33.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:32.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:33.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:32.996+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:33.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:33.272+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:33.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:33.272+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:33.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:33.272+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:33.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:33.273+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:34.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:33.842+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:34.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:33.842+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:34.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:33.842+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:34.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:33.844+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:34.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:34.040+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:34.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:34.040+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:34.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:34.040+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:34.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:34.041+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:34.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:34.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:34.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:34.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:34.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:34.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:34.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:34.254+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:35.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:34.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:35.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:34.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:35.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:34.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:35.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:34.802+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:35.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:35.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:35.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:35.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:35.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:35.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:35.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:35.044+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:35.651 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:35.276+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:35.652 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:35.276+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:35.652 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:35.276+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:35.652 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:35.277+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:35.902 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:09:35.901+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 381 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:09:35.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:35.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:35.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:35.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:35.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:35.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:35.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:35.755+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:36.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:35.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:36.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:35.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:36.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:35.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:36.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:35.995+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:36.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:36.249+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:36.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:36.249+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:36.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:36.249+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:36.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:36.250+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:36.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:36.950+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:36.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:36.950+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:36.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:36.950+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:36.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:36.950+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:36.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:36.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:36.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:36.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:36.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:36.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:36.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:36.716+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:37.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:37.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:37.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:37.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:37.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:37.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:37.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:37.213+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:38.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:37.986+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:38.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:37.986+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:38.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:37.986+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:38.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:37.986+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:38.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:37.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:38.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:37.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:38.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:37.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:38.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:37.764+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:38.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:38.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:38.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:38.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:38.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:38.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:38.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:38.216+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:39.014 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:38.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:39.014 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:38.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:39.014 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:38.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:39.014 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:38.745+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:39.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:39.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:39.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:39.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:39.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:39.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:39.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:39.171+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:39.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:39.034+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:39.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:39.034+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:39.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:39.034+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:39.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:39.034+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:40.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:39.736+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:40.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:39.736+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:40.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:39.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:40.040 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:39.737+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:40.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:40.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:40.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:40.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:40.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:40.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:40.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:40.138+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:40.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:40.060+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:40.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:40.060+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:40.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:40.060+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:40.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:40.060+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:40.949 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:09:40.902+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 409 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:09:40.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:40.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:40.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:40.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:40.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:40.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:40.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:40.714+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:41.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:41.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:41.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:41.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:41.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:41.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:41.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:41.186+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:41.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:41.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:41.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:41.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:41.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:41.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:41.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:41.013+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:41.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:41.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:41.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:41.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:41.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:41.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:41.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:41.743+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:42.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:42.167+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:42.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:42.167+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:42.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:42.167+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:42.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:42.168+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:42.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:41.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:42.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:41.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:42.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:41.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:42.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:41.970+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:42.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:42.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:42.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:42.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:42.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:42.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:42.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:42.698+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:43.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:43.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:43.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:43.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:43.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:43.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:43.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:43.119+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:43.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:42.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:43.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:42.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:43.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:42.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:43.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:42.992+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:43.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:43.687+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:43.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:43.687+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:43.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:43.687+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:43.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:43.687+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:44.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:44.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:44.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:44.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:44.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:44.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:44.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:44.086+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:44.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:44.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:44.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:44.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:44.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:44.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:44.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:44.042+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:44.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:44.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:44.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:44.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:44.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:44.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:44.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:44.652+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:45.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:45.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:45.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:45.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:45.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:45.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:45.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:45.089+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:45.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:45.082+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:45.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:45.082+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:45.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:45.082+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:45.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:45.082+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:45.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:45.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:45.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:45.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:45.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:45.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:45.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:45.624+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:46.199 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:09:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:09:45.903+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 437 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:09:46.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:46.089+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:46.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:46.089+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:46.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:46.089+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:46.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:46.090+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:46.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:46.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:46.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:46.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:46.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:46.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:46.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:46.055+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:46.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:46.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:46.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:46.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:46.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:46.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:46.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:46.603+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:47.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:47.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:47.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:47.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:47.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:47.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:47.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:47.044+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:47.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:47.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:47.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:47.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:47.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:47.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:47.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:47.042+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:47.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:47.616+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:47.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:47.616+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:47.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:47.616+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:47.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:47.617+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:48.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:48.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:48.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:48.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:48.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:48.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:48.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:48.058+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:48.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:48.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:48.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:48.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:48.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:48.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:48.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:48.083+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:48.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:48.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:48.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:48.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:48.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:48.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:48.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:48.664+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:49.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:49.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:49.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:49.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:49.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:49.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:49.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:49.100+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:49.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:49.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:49.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:49.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:49.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:49.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:49.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:49.051+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:49.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:49.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:49.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:49.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:49.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:49.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:49.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:49.703+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:50.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:50.079+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:50.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:50.079+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:50.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:50.079+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:50.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:50.080+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:50.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:50.019+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:50.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:50.019+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:50.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:50.019+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:50.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:50.020+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:50.949 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:09:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:09:50.904+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 464 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:09:50.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:50.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:50.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:50.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:50.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:50.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:50.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:50.718+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:51.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:51.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:51.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:51.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:51.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:51.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:51.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:51.044+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:51.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:51.035+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:51.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:51.035+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:51.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:51.035+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:51.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:51.036+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:51.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:51.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:51.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:51.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:51.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:51.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:51.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:51.703+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:52.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:52.091+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:52.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:52.091+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:52.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:52.091+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:52.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:52.092+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:52.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:52.038+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:52.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:52.038+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:52.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:52.038+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:52.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:52.039+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:53.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:52.749+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:53.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:52.749+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:53.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:52.749+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:53.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:52.749+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:53.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:53.071+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:53.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:53.071+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:53.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:53.071+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:53.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:53.071+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:53.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:53.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:53.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:53.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:53.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:53.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:53.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:53.053+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:54.026 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:54.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:54.026 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:54.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:54.026 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:54.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:54.026 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:54.004+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:54.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:53.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:54.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:53.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:54.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:53.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:54.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:53.778+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:54.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:54.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:54.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:54.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:54.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:54.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:54.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:54.047+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:55.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:55.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:55.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:55.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:55.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:55.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:55.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:55.008+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:55.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:54.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:55.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:54.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:55.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:54.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:55.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:54.778+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:55.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:55.019+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:55.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:55.019+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:55.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:55.019+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:55.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:55.019+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:55.965 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:09:55.904+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 491 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:09:55.965 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:55.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:55.965 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:55.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:55.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:55.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:55.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:55.729+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:56.346 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:55.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:56.346 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:56.000+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:56.347 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:56.000+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:56.347 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:56.000+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:56.347 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:55.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:56.347 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:55.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:56.347 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:55.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:56.348 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:55.987+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:56.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:56.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:56.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:56.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:56.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:56.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:56.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:56.707+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:56.941 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:56.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:56.941 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:56.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:56.941 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:56.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:56.941 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:56.938+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:57.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:56.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:57.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:56.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:57.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:56.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:57.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:56.961+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:57.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:57.888+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:57.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:57.888+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:57.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:57.888+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:57.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:57.889+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:57.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:57.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:57.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:57.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:57.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:57.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:57.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:57.672+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:58.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:57.976+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:58.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:57.976+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:58.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:57.976+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:58.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:57.977+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:58.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:58.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:58.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:58.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:58.908 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:58.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:58.908 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:58.656+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:59.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:58.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:59.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:58.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:59.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:58.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:59.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:58.944+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:59.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:58.928+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:59.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:58.928+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:59.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:58.928+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:09:59.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:58.929+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:59.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:59.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:59.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:59.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:59.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:59.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:09:59.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:09:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:09:59.663+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:09:59.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:59.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:59.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:59.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:59.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:59.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:09:59.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:09:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:09:59.910+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:00.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:59.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:00.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:59.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:00.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:59.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:00.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:09:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:09:59.974+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:00.949 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:10:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:10:00.904+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 519 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:10:00.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:00.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:00.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:00.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:00.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:00.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:00.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:00.708+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:00.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:00.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:00.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:00.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:00.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:00.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:00.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:00.870+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:01.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:00.985+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:01.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:00.985+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:01.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:00.985+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:01.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:00.986+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:01.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:01.736+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:01.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:01.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:01.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:01.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:01.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:01.737+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:01.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:01.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:01.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:01.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:01.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:01.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:01.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:01.871+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:02.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:01.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:02.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:01.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:02.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:01.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:02.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:01.993+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:02.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:02.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:02.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:02.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:02.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:02.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:02.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:02.703+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:02.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:02.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:02.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:02.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:02.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:02.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:02.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:02.916+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:03.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:03.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:03.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:03.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:03.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:03.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:03.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:03.015+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:03.859 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:03.749+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:03.859 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:03.749+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:03.859 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:03.749+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:03.859 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:03.750+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:04.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:04.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:04.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:04.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:04.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:04.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:04.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:04.008+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:04.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:03.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:04.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:03.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:04.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:03.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:04.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:03.880+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:05.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:04.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:05.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:04.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:05.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:04.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:05.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:04.874+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:05.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:04.960+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:05.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:04.960+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:05.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:04.960+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:05.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:04.961+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:05.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:04.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:05.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:04.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:05.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:04.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:05.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:04.796+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:06.056 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:10:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:10:05.905+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 552 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:10:06.056 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:05.949+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:06.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:05.949+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:06.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:05.949+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:06.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:05.950+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:06.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:05.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:06.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:05.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:06.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:05.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:06.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:05.803+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:06.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:05.919+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:06.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:05.919+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:06.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:05.919+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:06.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:05.920+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:07.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:06.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:07.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:06.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:07.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:06.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:07.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:06.772+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:07.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:06.962+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:07.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:06.962+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:07.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:06.962+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:07.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:06.962+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:07.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:06.975+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:07.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:06.975+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:07.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:06.975+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:07.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:06.976+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:08.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:07.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:08.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:07.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:08.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:07.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:08.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:07.998+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:08.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:07.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:08.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:07.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:08.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:07.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:08.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:07.803+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:08.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:08.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:08.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:08.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:08.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:08.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:08.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:08.006+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:09.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:08.962+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:09.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:08.962+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:09.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:08.962+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:09.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:08.963+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:09.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:08.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:09.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:08.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:09.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:08.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:09.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:08.808+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:09.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:09.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:09.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:09.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:09.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:09.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:09.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:09.033+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:10.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:09.836+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:10.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:09.836+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:10.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:09.836+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:10.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:09.836+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:10.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:10.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:10.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:10.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:10.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:10.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:10.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:10.001+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:10.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:09.983+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:10.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:09.983+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:10.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:09.983+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:10.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:09.984+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:11.100 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:10:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:10:10.905+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 590 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:10:11.101 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:11.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:11.101 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:11.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:11.101 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:11.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:11.101 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:11.005+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:11.101 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:10.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:11.101 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:10.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:11.101 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:10.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:11.101 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:10.870+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:11.102 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:11.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:11.102 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:11.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:11.102 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:11.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:11.102 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:11.015+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:12.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:12.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:12.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:12.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:12.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:12.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:12.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:12.055+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:12.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:11.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:12.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:11.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:12.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:11.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:12.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:11.967+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:12.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:11.879+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:12.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:11.879+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:12.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:11.879+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:12.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:11.880+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:13.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:12.896+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:13.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:12.896+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:13.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:12.896+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:13.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:12.897+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:13.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:13.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:13.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:13.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:13.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:13.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:13.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:13.037+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:13.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:13.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:13.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:13.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:13.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:13.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:13.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:13.002+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:14.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:14.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:14.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:14.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:14.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:14.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:14.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:14.041+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:14.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:13.918+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:14.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:13.918+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:14.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:13.918+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:14.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:13.918+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:14.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:14.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:14.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:14.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:14.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:14.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:14.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:14.045+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:15.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:15.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:15.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:15.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:15.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:15.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:15.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:15.021+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:15.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:15.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:15.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:15.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:15.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:15.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:15.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:15.002+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:15.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:14.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:15.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:14.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:15.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:14.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:15.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:14.957+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:16.199 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:10:15.906+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 617 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:10:16.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:15.945+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:16.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:15.945+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:16.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:15.945+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:16.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:15.946+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:16.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:15.980+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:16.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:15.980+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:16.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:15.980+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:16.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:15.980+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:16.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:16.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:16.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:16.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:16.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:16.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:16.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:16.015+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:17.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:16.934+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:17.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:16.934+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:17.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:16.934+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:17.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:16.935+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:17.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:17.028+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:17.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:17.028+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:17.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:17.028+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:17.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:17.030+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:17.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:17.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:17.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:17.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:17.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:17.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:17.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:17.065+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:18.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:17.935+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:18.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:17.935+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:18.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:17.935+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:18.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:17.935+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:18.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:18.042+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:18.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:18.042+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:18.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:18.042+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:18.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:18.043+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:18.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:18.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:18.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:18.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:18.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:18.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:18.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:18.036+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:19.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:18.894+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:19.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:18.894+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:19.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:18.894+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:19.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:18.895+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:19.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:19.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:19.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:19.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:19.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:19.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:19.201 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:19.014+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:19.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:18.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:19.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:18.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:19.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:18.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:19.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:18.994+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:20.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:19.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:20.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:19.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:20.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:19.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:20.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:19.854+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:20.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:20.026+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:20.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:20.026+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:20.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:20.026+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:20.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:20.026+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:20.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:20.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:20.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:20.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:20.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:20.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:20.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:20.056+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:21.199 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:10:20.906+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 645 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:10:21.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:21.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:21.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:21.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:21.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:21.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:21.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:21.054+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:21.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:21.063+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:21.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:21.063+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:21.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:21.063+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:21.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:21.063+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:21.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:20.870+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:21.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:20.870+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:21.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:20.870+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:21.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:20.871+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:22.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:21.837+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:22.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:21.837+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:22.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:21.837+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:22.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:21.838+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:22.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:22.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:22.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:22.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:22.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:22.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:22.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:22.021+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:22.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:22.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:22.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:22.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:22.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:22.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:22.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:22.055+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:23.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:23.092+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:23.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:23.092+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:23.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:23.092+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:23.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:23.093+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:23.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:23.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:23.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:23.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:23.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:23.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:23.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:23.005+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:23.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:22.876+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:23.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:22.876+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:23.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:22.876+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:23.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:22.876+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:24.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:23.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:24.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:23.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:24.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:23.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:24.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:23.831+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:24.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:24.051+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:24.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:24.051+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:24.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:24.051+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:24.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:24.051+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:24.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:24.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:24.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:24.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:24.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:24.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:24.201 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:24.077+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:25.058 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:25.043+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:25.058 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:25.043+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:25.058 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:25.043+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:25.058 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:25.044+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:25.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:24.803+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:25.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:24.803+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:25.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:24.803+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:25.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:24.804+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:25.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:25.079+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:25.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:25.079+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:25.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:25.079+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:25.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:25.079+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:26.093 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:10:25.907+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 673 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:10:26.094 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:25.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:26.094 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:25.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:26.094 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:25.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:26.094 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:25.850+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:26.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:26.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:26.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:26.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:26.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:26.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:26.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:26.010+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:26.346 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:26.114+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:26.346 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:26.114+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:26.346 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:26.114+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:26.346 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:26.115+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:27.113 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:26.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:27.113 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:26.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:27.113 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:26.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:27.113 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:26.998+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:27.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:26.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:27.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:26.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:27.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:26.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:27.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:26.852+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:27.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:27.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:27.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:27.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:27.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:27.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:27.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:27.134+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:28.139 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:28.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:28.139 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:28.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:28.139 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:28.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:28.139 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:28.014+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:28.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:27.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:28.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:27.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:28.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:27.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:28.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:27.822+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:28.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:28.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:28.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:28.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:28.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:28.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:28.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:28.160+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:29.101 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:28.826+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:29.101 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:28.826+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:29.101 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:28.826+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:29.101 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:28.827+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:29.101 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:29.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:29.101 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:29.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:29.101 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:29.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:29.101 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:29.001+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:29.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:29.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:29.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:29.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:29.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:29.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:29.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:29.122+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:30.113 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:29.988+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:30.113 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:29.988+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:30.113 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:29.988+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:30.114 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:29.988+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:30.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:29.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:30.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:29.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:30.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:29.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:30.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:29.831+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:30.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:30.134+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:30.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:30.134+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:30.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:30.134+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:30.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:30.134+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:30.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:30.836+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:30.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:30.836+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:30.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:30.836+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:30.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:30.837+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:31.137 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:10:30.907+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 700 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:10:31.138 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:31.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:31.138 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:31.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:31.138 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:31.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:31.138 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:31.013+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:31.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:31.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:31.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:31.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:31.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:31.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:31.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:31.159+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:32.131 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:31.980+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:32.131 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:31.980+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:32.131 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:31.980+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:32.131 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:31.981+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:32.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:31.883+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:32.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:31.883+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:32.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:31.883+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:32.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:31.884+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:32.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:32.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:32.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:32.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:32.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:32.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:32.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:32.151+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:33.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:33.102+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:33.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:33.102+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:33.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:33.102+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:33.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:33.103+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:33.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:32.999+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:33.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:32.999+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:33.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:32.999+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:33.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:32.999+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:33.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:32.918+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:33.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:32.918+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:33.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:32.918+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:33.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:32.919+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:34.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:34.114+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:34.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:34.114+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:34.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:34.114+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:34.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:34.115+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:34.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:33.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:34.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:33.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:34.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:33.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:34.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:33.990+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:34.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:33.877+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:34.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:33.877+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:34.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:33.877+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:34.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:33.878+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:35.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:34.983+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:35.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:34.983+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:35.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:34.983+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:35.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:34.984+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:35.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:34.905+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:35.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:34.905+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:35.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:34.905+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:35.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:34.906+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:35.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:35.076+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:35.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:35.076+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:35.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:35.076+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:35.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:35.076+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:36.199 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:10:35.907+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 727 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:10:36.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:36.078+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:36.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:36.078+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:36.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:36.078+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:36.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:36.078+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:36.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:35.861+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:36.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:35.861+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:36.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:35.861+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:36.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:35.861+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:36.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:35.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:36.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:35.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:36.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:35.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:36.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:35.946+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:37.083 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:36.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:37.083 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:36.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:37.083 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:36.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:37.083 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:36.997+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:37.083 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:36.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:37.083 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:36.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:37.083 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:36.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:37.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:36.850+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:37.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:37.103+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:37.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:37.103+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:37.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:37.103+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:37.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:37.103+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:38.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:38.006+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:38.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:38.006+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:38.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:38.006+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:38.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:38.007+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:38.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:37.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:38.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:37.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:38.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:37.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:38.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:37.830+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:38.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:38.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:38.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:38.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:38.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:38.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:38.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:38.136+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:39.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:38.843+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:39.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:38.843+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:39.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:38.843+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:39.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:38.844+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:39.130 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:39.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:39.130 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:39.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:39.130 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:39.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:39.130 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:39.048+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:39.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:39.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:39.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:39.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:39.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:39.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:39.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:39.150+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:40.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:40.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:40.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:40.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:40.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:40.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:40.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:40.040+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:40.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:39.842+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:40.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:39.842+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:40.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:39.842+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:40.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:39.843+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:40.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:40.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:40.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:40.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:40.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:40.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:40.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:40.110+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:41.042 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:10:40.907+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 755 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:10:41.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:40.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:41.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:40.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:41.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:40.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:41.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:40.813+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:41.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:41.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:41.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:41.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:41.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:41.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:41.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:41.096+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:41.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:41.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:41.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:41.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:41.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:41.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:41.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:41.063+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:42.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:41.784+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:42.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:41.784+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:42.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:41.784+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:42.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:41.784+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:42.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:42.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:42.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:42.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:42.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:42.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:42.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:42.092+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:42.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:42.112+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:42.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:42.112+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:42.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:42.112+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:42.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:42.112+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:43.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:42.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:43.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:42.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:43.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:42.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:43.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:42.770+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:43.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:43.051+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:43.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:43.051+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:43.033 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:43.051+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:43.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:43.052+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:43.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:43.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:43.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:43.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:43.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:43.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:43.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:43.162+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:43.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:43.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:43.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:43.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:43.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:43.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:43.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:43.776+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:44.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:44.049+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:44.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:44.049+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:44.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:44.049+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:44.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:44.051+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:44.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:44.126+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:44.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:44.126+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:44.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:44.126+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:44.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:44.126+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:45.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:44.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:45.010 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:44.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:45.010 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:44.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:45.010 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:44.739+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:45.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:45.030+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:45.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:45.030+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:45.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:45.030+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:45.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:45.031+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:45.536 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:45.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:45.537 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:45.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:45.537 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:45.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:45.537 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:45.174+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:45.992 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:10:45.909+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 783 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:10:45.992 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:45.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:45.992 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:45.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:45.992 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:45.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:45.992 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:45.724+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:46.327 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:46.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:46.327 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:46.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:46.327 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:46.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:46.327 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:46.048+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:46.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:46.137+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:46.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:46.138+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:46.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:46.138+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:46.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:46.138+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:47.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:46.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:47.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:46.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:47.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:46.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:47.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:46.731+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:47.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:47.092+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:47.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:47.092+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:47.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:47.092+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:47.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:47.092+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:47.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:47.045+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:47.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:47.045+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:47.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:47.045+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:47.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:47.046+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:47.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:47.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:47.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:47.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:47.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:47.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:47.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:47.713+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:48.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:48.067+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:48.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:48.067+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:48.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:48.067+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:48.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:48.067+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:48.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:48.044+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:48.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:48.044+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:48.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:48.044+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:48.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:48.045+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:48.987 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:48.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:48.988 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:48.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:48.988 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:48.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:48.988 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:48.734+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:49.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:49.087+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:49.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:49.087+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:49.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:49.087+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:49.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:49.091+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:49.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:49.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:49.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:49.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:49.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:49.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:49.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:49.009+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:49.994 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:49.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:49.994 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:49.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:49.994 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:49.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:49.994 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:49.757+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:50.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:50.069+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:50.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:50.069+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:50.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:50.069+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:50.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:50.070+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:50.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:50.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:50.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:50.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:50.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:50.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:50.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:50.012+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:50.970 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:10:50.909+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 810 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:10:50.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:50.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:50.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:50.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:50.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:50.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:50.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:50.737+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:51.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:50.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:51.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:50.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:51.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:50.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:51.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:50.992+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:51.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:51.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:51.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:51.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:51.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:51.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:51.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:51.103+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:51.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:51.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:51.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:51.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:51.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:51.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:51.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:51.705+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:52.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:52.122+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:52.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:52.122+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:52.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:52.122+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:52.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:52.123+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:52.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:51.970+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:52.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:51.970+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:52.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:51.970+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:52.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:51.971+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:52.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:52.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:52.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:52.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:52.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:52.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:52.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:52.668+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:53.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:53.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:53.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:53.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:53.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:53.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:53.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:53.007+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:53.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:53.167+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:53.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:53.167+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:53.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:53.167+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:53.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:53.167+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:53.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:53.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:53.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:53.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:53.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:53.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:53.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:53.665+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:54.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:54.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:54.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:54.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:54.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:54.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:54.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:54.173+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:54.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:53.972+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:54.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:53.972+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:54.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:53.972+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:54.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:53.973+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:54.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:54.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:54.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:54.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:54.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:54.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:54.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:54.679+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:55.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:55.201+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:55.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:55.201+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:55.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:55.201+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:55.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:55.202+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:55.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:55.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:55.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:55.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:55.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:55.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:55.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:55.021+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:55.910 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:10:55.910+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 837 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:10:55.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:55.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:55.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:55.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:55.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:55.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:55.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:55.674+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:56.190 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:56.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:56.190 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:56.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:56.190 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:56.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:56.191 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:56.023+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:56.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:56.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:56.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:56.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:56.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:56.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:56.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:56.212+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:56.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:56.670+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:56.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:56.670+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:56.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:56.670+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:56.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:56.671+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:57.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:56.974+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:57.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:56.974+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:57.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:56.974+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:57.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:56.974+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:57.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:57.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:57.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:57.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:57.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:57.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:57.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:57.212+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:57.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:57.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:57.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:57.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:57.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:57.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:57.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:57.670+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:58.280 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:58.241+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:58.281 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:58.241+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:58.281 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:58.241+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:58.281 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:58.242+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:58.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:57.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:58.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:57.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:58.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:57.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:58.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:57.993+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:58.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:58.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:58.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:58.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:58.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:58.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:58.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:58.630+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:59.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:59.231+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:59.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:59.231+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:59.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:59.231+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:10:59.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:10:59.232+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:59.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:58.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:59.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:58.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:59.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:58.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:10:59.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:10:58.989+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:10:59.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:59.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:59.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:59.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:59.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:59.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:10:59.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:10:59.592+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:00.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:00.187+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:00.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:00.187+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:00.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:00.187+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:00.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:00.188+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:00.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:00.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:00.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:00.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:00.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:00.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:00.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:00.007+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:00.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:00.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:00.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:00.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:00.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:00.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:00.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:00.596+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:01.199 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:11:00.910+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 865 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:11:01.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:01.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:01.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:01.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:01.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:01.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:01.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:01.038+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:01.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:01.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:01.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:01.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:01.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:01.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:01.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:01.152+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:01.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:01.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:01.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:01.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:01.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:01.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:01.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:01.556+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:02.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:02.011+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:02.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:02.011+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:02.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:02.011+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:02.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:02.012+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:02.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:02.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:02.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:02.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:02.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:02.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:02.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:02.162+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:02.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:02.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:02.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:02.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:02.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:02.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:02.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:02.521+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:03.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:03.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:03.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:03.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:03.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:03.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:03.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:03.036+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:03.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:03.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:03.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:03.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:03.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:03.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:03.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:03.157+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:03.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:03.563+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:03.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:03.563+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:03.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:03.563+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:03.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:03.563+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:04.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:04.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:04.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:04.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:04.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:04.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:04.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:04.166+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:04.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:04.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:04.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:04.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:04.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:04.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:04.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:04.044+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:04.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:04.565+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:04.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:04.565+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:04.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:04.565+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:04.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:04.566+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:05.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:05.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:05.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:05.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:05.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:05.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:05.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:05.139+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:05.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:05.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:05.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:05.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:05.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:05.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:05.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:05.013+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:05.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:05.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:05.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:05.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:05.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:05.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:05.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:05.576+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:06.199 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:11:05.911+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 893 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:11:06.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:06.099+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:06.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:06.099+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:06.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:06.099+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:06.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:06.101+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:06.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:05.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:06.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:05.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:06.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:05.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:06.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:05.979+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:06.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:06.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:06.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:06.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:06.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:06.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:06.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:06.624+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:07.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:07.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:07.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:07.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:07.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:07.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:07.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:07.005+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:07.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:07.056+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:07.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:07.056+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:07.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:07.056+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:07.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:07.057+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:07.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:07.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:07.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:07.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:07.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:07.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:07.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:07.577+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:08.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:08.070+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:08.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:08.070+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:08.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:08.070+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:08.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:08.071+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:08.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:08.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:08.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:08.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:08.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:08.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:08.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:08.009+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:08.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:08.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:08.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:08.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:08.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:08.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:08.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:08.616+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:09.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:09.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:09.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:09.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:09.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:09.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:09.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:09.063+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:09.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:08.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:09.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:08.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:09.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:08.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:09.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:08.988+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:09.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:09.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:09.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:09.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:09.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:09.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:09.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:09.623+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:10.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:10.027+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:10.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:10.027+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:10.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:10.027+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:10.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:10.028+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:10.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:09.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:10.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:09.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:10.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:09.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:10.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:09.994+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:10.891 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:10.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:10.891 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:10.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:10.891 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:10.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:10.891 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:10.641+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:11.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:11:10.912+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 920 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:11:11.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:10.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:11.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:10.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:11.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:10.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:11.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:10.962+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:11.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:11.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:11.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:11.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:11.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:11.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:11.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:11.001+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:11.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:11.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:11.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:11.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:11.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:11.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:11.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:11.649+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:12.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:11.990+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:12.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:11.990+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:12.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:11.990+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:12.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:11.991+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:12.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:11.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:12.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:11.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:12.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:11.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:12.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:11.957+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:12.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:12.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:12.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:12.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:12.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:12.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:12.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:12.607+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:13.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:12.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:13.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:12.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:13.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:12.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:13.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:12.985+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:13.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:12.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:13.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:12.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:13.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:12.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:13.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:12.967+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:13.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:13.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:13.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:13.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:13.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:13.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:13.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:13.656+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:14.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:13.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:14.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:13.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:14.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:13.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:14.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:13.940+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:14.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:13.924+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:14.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:13.924+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:14.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:13.924+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:14.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:13.925+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:14.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:14.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:14.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:14.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:14.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:14.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:14.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:14.642+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:15.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:14.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:15.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:14.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:15.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:14.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:15.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:14.916+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:15.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:14.897+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:15.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:14.897+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:15.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:14.897+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:15.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:14.898+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:15.912 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:11:15.912+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 947 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:11:15.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:15.901+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:15.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:15.901+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:15.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:15.901+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:15.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:15.902+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:15.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:15.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:15.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:15.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:15.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:15.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:15.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:15.683+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:15.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:15.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:15.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:15.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:15.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:15.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:15.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:15.879+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:16.862 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:16.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:16.863 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:16.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:16.863 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:16.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:16.863 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:16.689+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:17.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:16.925+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:17.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:16.925+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:17.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:16.925+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:17.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:16.926+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:17.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:16.884+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:17.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:16.884+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:17.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:16.884+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:17.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:16.885+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:17.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:17.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:17.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:17.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:17.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:17.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:17.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:17.703+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:17.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:17.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:17.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:17.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:17.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:17.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:17.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:17.912+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:17.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:17.870+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:17.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:17.870+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:17.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:17.870+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:17.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:17.871+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:18.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:18.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:18.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:18.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:18.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:18.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:18.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:18.659+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:18.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:18.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:18.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:18.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:18.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:18.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:18.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:18.863+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:18.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:18.881+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:18.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:18.881+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:18.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:18.882+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:18.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:18.882+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:19.853 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:19.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:19.853 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:19.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:19.853 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:19.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:19.853 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:19.611+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:19.854 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:19.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:19.854 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:19.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:19.854 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:19.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:19.854 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:19.835+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:20.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:19.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:20.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:19.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:20.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:19.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:20.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:19.875+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:20.841 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:20.603+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:20.841 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:20.603+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:20.841 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:20.603+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:20.841 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:20.603+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:20.841 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:20.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:20.841 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:20.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:20.842 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:20.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:20.842 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:20.838+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:21.199 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:11:20.913+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 975 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:11:21.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:20.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:21.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:20.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:21.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:20.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:21.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:20.862+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:21.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:21.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:21.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:21.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:21.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:21.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:21.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:21.561+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:22.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:21.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:22.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:21.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:22.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:21.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:22.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:21.866+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:22.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:21.872+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:22.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:21.872+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:22.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:21.872+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:22.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:21.873+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:22.808 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:22.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:22.809 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:22.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:22.809 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:22.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:22.809 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:22.532+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:23.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:22.864+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:23.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:22.864+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:23.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:22.864+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:23.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:22.864+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:23.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:22.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:23.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:22.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:23.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:22.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:23.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:22.831+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:23.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:23.496+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:23.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:23.496+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:23.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:23.496+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:23.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:23.497+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:24.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:23.906+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:24.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:23.906+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:24.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:23.906+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:24.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:23.907+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:24.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:23.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:24.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:23.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:24.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:23.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:24.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:23.836+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:24.798 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:24.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:24.799 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:24.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:24.799 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:24.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:24.799 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:24.533+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:25.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:24.924+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:25.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:24.924+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:25.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:24.924+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:25.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:24.925+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:25.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:24.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:25.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:24.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:25.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:24.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:25.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:24.820+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:25.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:25.778+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:25.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:25.778+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:25.858 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:25.778+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:25.858 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:25.779+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:25.858 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:25.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:25.858 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:25.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:25.858 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:25.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:25.858 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:25.568+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:26.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:11:25.913+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1049 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:11:26.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:25.878+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:26.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:25.878+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:26.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:25.878+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:26.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:25.879+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:26.846 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:26.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:26.846 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:26.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:26.846 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:26.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:26.846 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:26.571+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:26.847 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:26.788+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:26.847 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:26.788+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:26.847 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:26.788+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:26.847 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:26.789+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:27.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:26.866+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:27.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:26.866+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:27.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:26.866+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:27.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:26.868+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:27.878 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:27.750+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:27.879 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:27.750+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:27.879 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:27.750+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:27.879 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:27.750+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:27.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:27.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:27.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:27.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:27.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:27.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:27.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:27.551+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:28.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:27.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:28.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:27.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:28.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:27.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:28.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:27.901+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:28.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:28.739+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:28.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:28.739+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:28.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:28.739+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:28.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:28.739+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:28.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:28.561+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:28.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:28.561+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:28.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:28.561+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:28.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:28.562+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:29.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:28.936+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:29.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:28.936+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:29.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:28.936+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:29.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:28.937+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:29.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:29.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:29.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:29.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:29.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:29.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:29.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:29.751+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:29.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:29.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:29.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:29.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:29.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:29.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:29.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:29.548+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:30.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:29.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:30.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:29.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:30.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:29.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:30.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:29.967+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:30.893 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:30.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:30.893 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:30.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:30.893 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:30.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:30.893 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:30.730+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:30.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:30.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:30.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:30.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:30.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:30.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:30.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:30.569+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:31.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:11:30.914+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1143 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:11:31.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:30.982+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:31.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:30.982+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:31.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:30.982+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:31.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:30.983+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:31.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:31.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:31.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:31.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:31.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:31.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:31.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:31.712+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:31.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:31.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:31.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:31.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:31.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:31.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:31.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:31.596+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:32.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:32.021+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:32.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:32.021+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:32.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:32.021+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:32.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:32.021+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:32.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:32.672+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:32.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:32.672+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:32.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:32.672+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:32.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:32.673+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:32.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:32.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:32.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:32.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:32.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:32.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:32.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:32.579+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:33.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:33.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:33.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:33.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:33.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:33.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:33.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:33.014+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:33.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:33.704+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:33.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:33.704+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:33.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:33.704+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:33.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:33.705+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:33.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:33.539+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:33.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:33.539+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:33.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:33.539+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:33.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:33.540+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:34.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:34.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:34.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:34.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:34.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:34.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:34.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:34.015+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:34.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:34.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:34.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:34.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:34.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:34.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:34.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:34.713+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:34.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:34.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:34.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:34.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:34.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:34.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:34.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:34.579+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:35.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:35.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:35.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:35.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:35.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:35.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:35.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:35.009+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:35.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:35.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:35.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:35.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:35.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:35.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:35.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:35.735+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:35.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:35.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:35.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:35.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:35.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:35.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:35.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:35.534+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:36.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:11:35.913+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1237 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:11:36.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:36.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:36.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:36.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:36.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:36.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:36.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:36.051+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:36.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:36.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:36.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:36.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:36.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:36.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:36.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:36.538+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:37.068 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:36.773+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:37.068 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:36.773+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:37.068 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:36.773+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:37.068 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:36.774+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:37.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:37.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:37.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:37.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:37.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:37.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:37.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:37.089+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:37.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:37.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:37.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:37.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:37.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:37.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:37.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:37.734+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:37.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:37.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:37.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:37.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:37.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:37.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:37.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:37.540+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:38.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:38.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:38.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:38.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:38.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:38.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:38.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:38.054+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:38.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:38.516+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:38.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:38.516+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:38.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:38.516+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:38.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:38.516+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:38.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:38.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:38.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:38.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:38.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:38.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:38.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:38.762+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:39.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:39.044+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:39.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:39.044+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:39.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:39.044+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:39.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:39.045+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:39.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:39.548+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:39.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:39.548+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:39.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:39.548+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:39.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:39.548+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:39.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:39.736+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:39.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:39.736+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:39.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:39.736+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:39.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:39.736+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:40.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:39.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:40.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:39.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:40.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:39.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:40.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:39.999+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:40.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:40.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:40.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:40.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:40.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:40.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:40.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:40.590+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:40.895 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:40.698+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:40.895 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:40.698+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:40.895 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:40.698+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:40.895 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:40.699+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:41.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:11:40.914+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1332 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:11:41.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:40.965+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:41.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:40.965+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:41.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:40.965+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:41.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:40.966+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:41.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:41.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:41.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:41.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:41.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:41.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:41.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:41.634+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:41.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:41.670+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:41.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:41.670+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:41.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:41.670+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:41.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:41.670+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:42.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:41.997+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:42.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:41.997+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:42.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:41.997+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:42.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:41.998+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:42.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:42.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:42.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:42.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:42.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:42.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:42.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:42.661+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:42.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:42.678+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:42.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:42.678+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:42.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:42.678+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:42.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:42.679+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:43.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:43.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:43.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:43.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:43.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:43.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:43.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:43.014+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:43.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:43.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:43.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:43.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:43.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:43.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:43.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:43.679+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:43.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:43.638+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:43.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:43.638+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:43.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:43.638+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:43.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:43.638+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:44.360 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:43.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:44.360 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:43.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:44.360 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:43.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:44.360 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:43.989+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:44.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:44.657+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:44.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:44.657+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:44.922 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:44.657+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:44.922 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:44.658+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:44.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:44.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:44.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:44.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:44.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:44.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:44.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:44.695+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:45.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:44.942+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:45.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:44.942+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:45.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:44.942+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:45.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:44.944+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:45.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:45.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:45.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:45.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:45.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:45.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:45.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:45.668+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:45.882 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:45.622+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:45.882 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:45.622+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:45.882 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:45.622+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:45.882 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:45.623+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:46.199 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:11:45.916+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1427 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:11:46.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:45.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:46.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:45.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:46.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:45.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:46.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:45.903+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:46.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:11:46.625+0000 7f770c428700 0 monclient(hunting): authenticate timed out after 300 2024-04-27T02:11:46.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:46.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:46.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:46.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:46.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:46.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:46.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:46.678+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:46.883 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:46.658+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:46.883 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:46.658+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:46.883 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:46.658+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:46.883 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:46.658+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:47.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:46.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:47.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:46.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:47.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:46.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:47.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:46.940+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:47.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:47.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:47.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:47.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:47.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:47.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:47.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:47.680+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:47.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:47.695+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:47.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:47.695+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:47.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:47.695+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:47.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:47.695+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:48.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:47.955+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:48.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:47.955+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:48.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:47.955+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:48.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:47.956+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:48.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:48.687+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:48.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:48.687+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:48.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:48.687+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:48.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:48.688+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:48.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:48.667+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:48.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:48.667+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:48.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:48.667+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:48.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:48.668+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:49.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:48.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:49.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:48.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:49.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:48.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:49.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:48.993+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:49.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:49.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:49.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:49.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:49.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:49.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:49.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:49.655+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:49.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:49.698+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:49.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:49.698+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:49.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:49.698+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:49.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:49.698+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:50.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:49.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:50.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:49.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:50.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:49.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:50.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:49.984+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:50.895 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:50.726+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:50.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:50.727+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:50.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:50.727+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:50.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:50.727+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:50.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:50.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:50.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:50.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:50.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:50.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:50.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:50.616+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:51.199 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:11:50.916+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1521 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:11:51.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:50.948+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:51.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:50.948+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:51.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:50.948+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:51.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:50.949+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:51.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:51.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:51.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:51.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:51.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:51.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:51.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:51.729+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:51.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:51.641+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:51.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:51.641+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:51.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:51.641+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:51.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:51.641+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:52.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:51.932+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:52.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:51.932+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:52.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:51.932+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:52.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:51.933+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:52.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:52.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:52.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:52.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:52.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:52.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:52.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:52.632+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:52.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:52.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:52.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:52.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:52.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:52.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:52.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:52.730+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:53.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:52.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:53.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:52.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:53.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:52.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:53.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:52.920+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:53.907 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:53.721+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:53.908 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:53.721+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:53.908 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:53.721+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:53.908 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:53.722+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:53.908 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:53.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:53.908 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:53.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:53.908 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:53.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:53.908 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:53.604+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:54.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:53.929+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:54.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:53.929+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:54.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:53.929+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:54.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:53.930+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:54.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:54.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:54.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:54.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:54.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:54.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:54.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:54.597+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:54.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:54.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:54.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:54.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:54.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:54.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:54.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:54.755+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:55.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:54.979+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:55.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:54.979+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:55.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:54.979+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:55.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:54.980+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:55.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:55.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:55.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:55.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:55.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:55.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:55.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:55.763+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:55.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:55.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:55.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:55.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:55.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:55.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:55.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:55.617+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:56.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:11:55.917+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1615 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:11:56.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:55.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:56.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:55.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:56.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:55.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:56.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:55.977+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:56.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:56.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:56.918 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:56.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:56.918 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:56.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:56.918 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:56.618+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:56.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:56.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:56.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:56.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:56.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:56.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:56.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:56.794+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:57.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:56.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:57.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:56.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:57.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:56.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:57.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:56.939+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:57.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:57.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:57.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:57.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:57.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:57.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:57.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:57.619+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:57.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:57.788+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:57.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:57.788+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:57.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:57.788+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:57.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:57.788+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:58.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:57.958+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:58.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:57.958+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:58.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:57.958+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:58.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:57.959+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:58.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:58.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:58.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:58.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:58.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:58.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:58.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:58.595+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:58.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:58.781+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:58.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:58.781+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:58.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:58.781+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:58.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:58.781+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:59.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:58.942+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:59.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:58.942+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:59.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:58.942+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:11:59.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:58.944+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:59.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:59.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:59.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:59.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:59.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:59.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:11:59.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:11:59.580+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:11:59.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:59.788+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:59.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:59.788+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:59.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:59.788+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:11:59.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:11:59.788+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:00.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:59.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:00.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:59.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:00.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:59.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:00.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:11:59.987+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:00.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:00.748+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:00.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:00.748+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:00.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:00.748+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:00.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:00.749+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:00.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:00.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:00.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:00.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:00.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:00.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:00.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:00.592+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:01.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:12:00.917+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1710 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:12:01.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:00.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:01.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:00.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:01.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:00.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:01.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:00.987+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:01.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:01.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:01.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:01.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:01.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:01.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:01.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:01.583+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:01.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:01.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:01.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:01.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:01.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:01.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:01.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:01.762+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:02.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:01.951+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:02.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:01.951+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:02.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:01.951+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:02.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:01.953+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:02.792 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:02.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:02.792 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:02.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:02.793 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:02.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:02.793 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:02.538+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:02.793 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:02.767+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:02.793 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:02.767+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:02.793 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:02.767+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:02.793 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:02.768+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:03.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:02.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:03.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:02.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:03.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:02.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:03.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:02.926+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:03.792 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:03.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:03.792 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:03.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:03.792 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:03.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:03.792 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:03.550+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:04.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:03.932+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:04.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:03.932+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:04.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:03.932+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:04.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:03.933+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:04.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:03.813+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:04.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:03.813+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:04.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:03.813+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:04.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:03.814+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:04.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:04.584+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:04.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:04.584+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:04.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:04.584+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:04.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:04.584+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:04.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:04.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:04.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:04.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:04.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:04.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:04.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:04.794+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:05.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:04.952+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:05.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:04.952+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:05.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:04.952+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:05.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:04.953+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:05.805 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:05.548+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:05.805 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:05.548+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:05.805 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:05.548+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:05.805 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:05.549+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:06.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:12:05.918+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1804 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:12:06.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:05.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:06.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:05.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:06.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:05.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:06.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:05.993+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:06.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:05.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:06.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:05.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:06.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:05.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:06.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:05.827+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:06.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:06.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:06.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:06.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:06.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:06.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:06.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:06.547+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:07.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:06.997+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:07.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:06.997+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:07.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:06.997+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:07.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:06.998+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:07.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:06.804+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:07.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:06.804+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:07.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:06.804+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:07.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:06.804+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:07.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:07.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:07.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:07.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:07.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:07.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:07.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:07.502+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:08.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:07.776+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:08.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:07.776+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:08.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:07.776+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:08.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:07.777+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:08.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:08.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:08.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:08.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:08.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:08.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:08.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:08.021+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:08.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:08.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:08.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:08.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:08.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:08.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:08.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:08.502+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:08.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:08.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:08.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:08.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:08.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:08.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:08.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:08.735+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:09.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:09.003+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:09.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:09.003+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:09.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:09.003+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:09.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:09.003+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:09.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:09.776+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:09.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:09.776+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:09.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:09.776+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:09.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:09.776+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:09.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:09.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:09.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:09.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:09.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:09.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:09.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:09.545+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:10.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:10.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:10.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:10.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:10.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:10.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:10.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:10.048+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:10.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:10.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:10.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:10.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:10.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:10.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:10.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:10.583+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:10.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:10.731+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:10.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:10.731+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:10.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:10.731+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:10.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:10.731+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:11.199 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:12:10.919+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1898 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:12:11.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:11.005+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:11.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:11.005+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:11.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:11.005+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:11.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:11.006+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:11.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:11.732+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:11.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:11.732+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:11.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:11.732+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:11.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:11.732+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:11.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:11.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:11.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:11.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:11.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:11.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:11.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:11.579+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:12.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:12.032+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:12.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:12.032+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:12.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:12.032+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:12.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:12.032+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:12.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:12.565+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:12.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:12.565+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:12.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:12.565+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:12.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:12.566+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:12.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:12.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:12.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:12.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:12.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:12.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:12.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:12.735+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:13.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:13.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:13.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:13.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:13.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:13.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:13.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:13.066+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:13.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:13.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:13.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:13.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:13.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:13.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:13.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:13.575+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:13.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:13.746+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:13.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:13.746+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:13.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:13.746+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:13.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:13.746+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:14.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:14.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:14.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:14.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:14.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:14.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:14.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:14.034+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:14.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:14.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:14.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:14.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:14.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:14.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:14.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:14.569+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:14.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:14.703+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:14.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:14.703+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:14.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:14.703+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:14.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:14.703+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:15.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:15.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:15.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:15.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:15.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:15.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:15.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:15.075+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:15.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:15.706+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:15.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:15.706+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:15.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:15.706+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:15.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:15.707+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:15.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:15.613+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:15.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:15.613+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:15.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:15.613+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:15.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:15.614+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:16.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:12:15.919+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 1992 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:12:16.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:16.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:16.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:16.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:16.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:16.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:16.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:16.094+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:16.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:16.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:16.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:16.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:16.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:16.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:16.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:16.593+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:16.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:16.694+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:16.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:16.694+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:16.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:16.694+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:16.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:16.695+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:17.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:17.079+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:17.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:17.079+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:17.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:17.079+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:17.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:17.080+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:17.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:17.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:17.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:17.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:17.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:17.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:17.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:17.556+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:17.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:17.655+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:17.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:17.655+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:17.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:17.655+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:17.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:17.656+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:18.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:18.096+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:18.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:18.096+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:18.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:18.096+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:18.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:18.097+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:18.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:18.557+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:18.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:18.557+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:18.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:18.557+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:18.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:18.558+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:18.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:18.630+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:18.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:18.630+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:18.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:18.630+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:18.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:18.630+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:19.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:19.059+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:19.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:19.059+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:19.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:19.059+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:19.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:19.061+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:19.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:19.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:19.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:19.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:19.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:19.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:19.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:19.595+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:19.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:19.620+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:19.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:19.620+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:19.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:19.620+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:19.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:19.620+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:20.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:20.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:20.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:20.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:20.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:20.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:20.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:20.063+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:20.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:20.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:20.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:20.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:20.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:20.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:20.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:20.564+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:20.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:20.607+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:20.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:20.607+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:20.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:20.607+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:20.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:20.608+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:21.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:12:20.920+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 2088 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:12:21.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:21.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:21.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:21.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:21.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:21.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:21.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:21.051+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:21.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:21.608+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:21.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:21.608+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:21.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:21.608+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:21.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:21.609+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:21.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:21.653+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:21.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:21.653+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:21.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:21.653+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:21.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:21.654+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:22.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:22.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:22.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:22.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:22.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:22.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:22.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:22.020+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:22.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:22.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:22.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:22.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:22.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:22.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:22.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:22.628+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:22.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:22.656+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:22.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:22.656+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:22.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:22.656+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:22.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:22.657+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:23.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:23.051+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:23.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:23.051+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:23.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:23.051+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:23.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:23.051+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:23.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:23.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:23.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:23.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:23.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:23.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:23.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:23.596+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:23.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:23.649+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:23.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:23.649+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:23.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:23.649+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:23.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:23.649+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:24.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:24.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:24.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:24.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:24.296 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:24.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:24.296 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:24.053+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:24.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:24.676+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:24.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:24.676+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:24.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:24.676+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:24.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:24.676+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:24.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:24.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:24.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:24.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:24.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:24.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:24.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:24.615+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:25.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:25.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:25.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:25.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:25.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:25.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:25.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:25.103+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:25.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:25.682+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:25.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:25.682+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:25.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:25.682+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:25.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:25.682+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:25.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:25.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:25.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:25.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:25.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:25.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:25.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:25.642+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:26.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:12:25.921+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 2183 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:12:26.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:26.114+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:26.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:26.114+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:26.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:26.114+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:26.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:26.115+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:26.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:26.687+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:26.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:26.687+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:26.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:26.687+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:26.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:26.688+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:26.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:26.647+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:26.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:26.647+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:26.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:26.647+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:26.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:26.648+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:27.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:27.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:27.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:27.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:27.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:27.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:27.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:27.091+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:27.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:27.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:27.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:27.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:27.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:27.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:27.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:27.694+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:27.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:27.670+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:27.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:27.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:27.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:27.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:27.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:27.671+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:28.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:28.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:28.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:28.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:28.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:28.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:28.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:28.135+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:28.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:28.660+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:28.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:28.660+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:28.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:28.660+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:28.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:28.661+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:28.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:28.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:28.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:28.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:28.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:28.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:28.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:28.639+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:29.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:29.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:29.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:29.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:29.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:29.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:29.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:29.090+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:29.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:29.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:29.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:29.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:29.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:29.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:29.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:29.674+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:29.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:29.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:29.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:29.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:29.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:29.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:29.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:29.711+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:30.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:30.106+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:30.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:30.106+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:30.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:30.106+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:30.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:30.108+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:30.948 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:12:30.921+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 2277 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:12:30.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:30.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:30.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:30.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:30.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:30.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:30.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:30.716+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:30.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:30.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:30.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:30.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:30.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:30.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:30.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:30.745+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:31.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:31.140+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:31.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:31.140+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:31.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:31.140+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:31.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:31.141+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:31.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:31.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:31.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:31.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:31.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:31.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:31.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:31.708+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:31.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:31.780+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:31.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:31.780+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:31.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:31.780+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:31.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:31.781+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:32.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:32.095+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:32.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:32.095+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:32.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:32.095+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:32.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:32.095+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:33.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:32.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:33.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:32.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:33.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:32.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:33.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:32.736+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:33.030 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:32.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:33.030 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:32.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:33.030 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:32.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:33.030 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:32.746+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:33.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:33.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:33.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:33.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:33.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:33.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:33.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:33.051+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:33.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:33.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:33.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:33.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:33.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:33.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:33.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:33.732+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:33.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:33.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:33.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:33.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:33.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:33.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:33.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:33.710+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:34.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:34.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:34.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:34.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:34.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:34.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:34.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:34.083+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:34.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:34.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:34.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:34.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:34.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:34.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:34.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:34.774+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:34.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:34.707+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:34.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:34.707+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:34.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:34.707+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:34.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:34.707+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:35.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:35.106+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:35.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:35.106+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:35.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:35.106+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:35.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:35.107+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:35.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:35.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:35.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:35.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:35.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:35.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:35.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:35.727+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:35.901 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:35.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:35.902 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:35.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:35.902 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:35.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:35.902 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:35.669+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:36.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:12:35.921+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 2371 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:12:36.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:36.098+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:36.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:36.098+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:36.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:36.098+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:36.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:36.099+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:36.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:36.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:36.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:36.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:36.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:36.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:36.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:36.710+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:36.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:36.709+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:36.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:36.709+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:36.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:36.709+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:36.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:36.710+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:37.355 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:37.067+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:37.355 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:37.067+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:37.355 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:37.067+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:37.355 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:37.067+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:37.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:37.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:37.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:37.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:37.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:37.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:37.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:37.665+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:37.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:37.745+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:37.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:37.745+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:37.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:37.745+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:37.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:37.746+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:38.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:38.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:38.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:38.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:38.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:38.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:38.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:38.077+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:38.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:38.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:38.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:38.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:38.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:38.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:38.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:38.785+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:38.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:38.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:38.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:38.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:38.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:38.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:38.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:38.684+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:39.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:39.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:39.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:39.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:39.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:39.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:39.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:39.077+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:39.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:39.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:39.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:39.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:39.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:39.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:39.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:39.652+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:39.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:39.831+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:39.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:39.831+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:39.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:39.831+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:39.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:39.832+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:40.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:40.064+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:40.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:40.064+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:40.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:40.064+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:40.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:40.064+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:40.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:40.654+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:40.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:40.654+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:40.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:40.654+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:40.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:40.654+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:40.903 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:40.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:40.903 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:40.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:40.903 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:40.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:40.903 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:40.852+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:41.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:12:40.922+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 2466 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:12:41.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:41.029+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:41.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:41.029+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:41.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:41.029+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:41.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:41.030+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:41.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:41.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:41.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:41.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:41.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:41.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:41.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:41.620+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:42.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:41.899+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:42.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:41.899+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:42.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:41.899+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:42.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:41.899+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:42.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:42.074+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:42.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:42.074+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:42.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:42.074+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:42.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:42.075+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:42.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:42.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:42.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:42.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:42.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:42.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:42.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:42.643+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:43.092 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:42.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:43.092 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:42.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:43.092 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:42.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:43.092 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:42.924+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:43.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:43.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:43.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:43.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:43.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:43.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:43.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:43.114+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:43.872 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:43.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:43.872 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:43.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:43.873 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:43.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:43.873 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:43.667+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:44.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:43.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:44.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:43.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:44.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:43.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:44.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:43.893+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:44.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:44.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:44.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:44.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:44.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:44.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:44.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:44.077+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:44.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:44.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:44.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:44.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:44.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:44.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:44.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:44.675+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:45.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:45.042+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:45.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:45.042+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:45.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:45.042+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:45.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:45.043+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:45.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:44.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:45.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:44.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:45.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:44.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:45.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:44.944+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:45.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:45.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:45.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:45.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:45.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:45.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:45.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:45.636+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:46.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:12:45.924+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 2560 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:12:46.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:46.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:46.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:46.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:46.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:46.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:46.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:46.057+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:46.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:45.902+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:46.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:45.902+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:46.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:45.902+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:46.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:45.902+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:46.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:46.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:46.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:46.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:46.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:46.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:46.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:46.663+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:46.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:46.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:46.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:46.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:46.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:46.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:46.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:46.862+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:47.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:47.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:47.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:47.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:47.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:47.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:47.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:47.015+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:47.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:47.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:47.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:47.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:47.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:47.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:47.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:47.626+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:48.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:48.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:48.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:48.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:48.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:48.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:48.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:48.034+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:48.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:47.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:48.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:47.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:48.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:47.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:48.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:47.868+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:48.872 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:48.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:48.872 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:48.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:48.872 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:48.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:48.872 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:48.822+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:48.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:48.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:48.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:48.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:48.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:48.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:48.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:48.594+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:49.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:49.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:49.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:49.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:49.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:49.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:49.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:49.023+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:49.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:49.786+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:49.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:49.786+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:49.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:49.786+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:49.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:49.787+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:49.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:49.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:49.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:49.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:49.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:49.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:49.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:49.581+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:50.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:49.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:50.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:49.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:50.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:49.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:50.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:49.990+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:50.903 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:50.750+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:50.903 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:50.750+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:50.903 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:50.750+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:50.904 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:50.750+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:50.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:50.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:50.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:50.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:50.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:50.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:50.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:50.551+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:51.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:12:50.924+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 2654 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:12:51.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:51.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:51.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:51.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:51.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:51.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:51.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:51.021+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:51.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:51.783+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:51.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:51.783+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:51.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:51.783+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:51.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:51.783+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:51.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:51.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:51.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:51.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:51.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:51.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:51.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:51.575+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:52.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:52.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:52.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:52.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:52.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:52.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:52.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:52.011+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:52.750 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:52.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:52.750 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:52.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:52.750 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:52.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:52.751 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:52.532+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:53.009 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:52.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:53.009 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:52.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:53.009 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:52.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:53.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:52.772+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:53.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:53.031+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:53.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:53.031+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:53.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:53.031+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:53.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:53.032+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:53.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:53.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:53.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:53.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:53.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:53.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:53.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:53.538+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:53.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:53.782+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:53.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:53.782+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:53.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:53.782+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:53.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:53.783+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:54.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:53.985+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:54.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:53.985+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:54.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:53.985+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:54.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:53.985+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:54.794 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:54.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:54.794 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:54.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:54.794 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:54.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:54.794 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:54.535+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:55.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:54.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:55.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:54.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:55.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:54.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:55.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:54.977+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:55.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:54.815+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:55.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:54.815+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:55.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:54.815+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:55.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:54.816+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:55.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:55.573+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:55.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:55.573+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:55.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:55.573+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:55.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:55.574+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:55.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:55.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:55.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:55.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:55.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:55.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:55.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:55.786+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:56.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:12:55.925+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 2748 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:12:56.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:55.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:56.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:55.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:56.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:55.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:56.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:55.984+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:56.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:56.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:56.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:56.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:56.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:56.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:56.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:56.563+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:56.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:56.776+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:56.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:56.776+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:56.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:56.776+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:56.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:56.777+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:57.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:56.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:57.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:56.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:57.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:56.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:57.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:56.975+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:57.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:57.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:57.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:57.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:57.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:57.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:57.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:57.735+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:57.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:57.600+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:57.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:57.600+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:57.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:57.600+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:57.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:57.601+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:58.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:58.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:58.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:58.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:58.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:58.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:58.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:58.001+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:58.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:58.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:58.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:58.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:58.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:58.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:58.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:58.591+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:58.935 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:58.783+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:58.935 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:58.783+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:58.935 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:58.783+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:58.936 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:58.783+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:59.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:58.956+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:59.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:58.956+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:59.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:58.956+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:12:59.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:58.957+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:59.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:59.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:59.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:59.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:59.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:59.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:12:59.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:12:59.560+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:12:59.902 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:59.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:59.902 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:59.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:59.902 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:59.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:12:59.902 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:12:59.784+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:00.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:59.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:00.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:59.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:00.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:59.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:00.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:12:59.923+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:00.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:00.585+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:00.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:00.585+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:00.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:00.585+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:00.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:00.586+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:00.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:00.761+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:00.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:00.761+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:00.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:00.761+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:00.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:00.762+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:01.160 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:13:00.926+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 2843 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:13:01.160 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:00.878+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:01.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:00.878+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:01.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:00.878+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:01.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:00.878+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:01.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:01.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:01.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:01.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:01.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:01.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:01.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:01.831+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:01.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:01.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:01.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:01.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:01.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:01.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:01.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:01.590+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:01.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:01.743+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:01.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:01.743+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:01.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:01.743+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:01.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:01.743+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:02.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:02.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:02.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:02.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:02.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:02.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:02.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:02.756+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:02.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:02.785+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:02.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:02.785+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:02.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:02.785+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:02.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:02.785+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:02.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:02.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:02.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:02.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:02.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:02.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:02.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:02.563+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:03.772 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:03.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:03.772 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:03.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:03.772 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:03.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:03.772 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:03.577+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:04.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:03.793+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:04.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:03.793+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:04.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:03.793+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:04.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:03.794+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:04.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:03.796+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:04.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:03.796+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:04.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:03.796+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:04.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:03.796+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:04.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:04.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:04.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:04.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:04.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:04.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:04.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:04.577+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:04.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:04.806+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:04.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:04.806+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:04.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:04.806+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:04.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:04.807+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:04.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:04.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:04.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:04.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:04.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:04.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:04.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:04.772+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:05.797 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:05.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:05.797 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:05.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:05.797 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:05.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:05.797 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:05.559+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:06.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:13:05.927+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 2938 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:13:06.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:05.821+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:06.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:05.821+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:06.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:05.821+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:06.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:05.822+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:06.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:05.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:06.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:05.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:06.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:05.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:06.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:05.819+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:06.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:06.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:06.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:06.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:06.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:06.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:06.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:06.547+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:07.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:06.803+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:07.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:06.803+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:07.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:06.803+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:07.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:06.804+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:07.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:06.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:07.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:06.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:07.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:06.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:07.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:06.843+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:07.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:07.548+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:07.791 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:07.548+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:07.791 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:07.548+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:07.791 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:07.549+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:08.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:07.812+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:08.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:07.812+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:08.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:07.812+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:08.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:07.813+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:08.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:07.855+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:08.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:07.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:08.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:07.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:08.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:07.856+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:08.765 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:08.518+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:08.765 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:08.518+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:08.765 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:08.518+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:08.765 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:08.519+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:09.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:08.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:09.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:08.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:09.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:08.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:09.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:08.858+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:09.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:08.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:09.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:08.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:09.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:08.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:09.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:08.787+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:09.795 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:09.493+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:09.795 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:09.493+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:09.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:09.493+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:09.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:09.494+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:10.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:09.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:10.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:09.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:10.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:09.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:10.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:09.817+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:10.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:09.829+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:10.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:09.829+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:10.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:09.829+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:10.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:09.830+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:10.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:10.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:10.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:10.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:10.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:10.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:10.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:10.459+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:11.030 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:13:10.927+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 3033 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:13:11.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:10.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:11.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:10.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:11.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:10.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:11.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:10.867+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:11.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:10.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:11.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:10.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:11.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:10.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:11.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:10.800+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:11.772 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:11.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:11.772 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:11.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:11.772 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:11.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:11.772 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:11.473+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:12.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:11.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:12.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:11.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:12.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:11.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:12.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:11.822+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:12.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:11.793+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:12.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:11.793+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:12.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:11.794+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:12.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:11.794+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:12.782 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:12.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:12.782 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:12.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:12.782 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:12.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:12.782 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:12.489+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:13.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:12.834+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:13.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:12.834+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:13.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:12.834+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:13.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:12.835+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:13.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:12.803+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:13.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:12.803+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:13.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:12.803+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:13.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:12.804+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:13.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:13.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:13.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:13.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:13.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:13.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:13.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:13.498+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:14.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:13.876+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:14.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:13.876+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:14.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:13.876+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:14.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:13.877+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:14.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:13.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:14.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:13.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:14.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:13.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:14.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:13.834+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:14.772 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:14.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:14.772 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:14.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:14.772 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:14.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:14.772 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:14.513+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:15.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:14.895+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:15.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:14.895+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:15.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:14.895+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:15.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:14.895+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:15.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:14.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:15.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:14.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:15.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:14.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:15.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:14.794+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:15.785 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:15.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:15.786 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:15.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:15.786 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:15.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:15.786 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:15.493+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:16.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:13:15.928+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 3127 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:13:16.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:15.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:16.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:15.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:16.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:15.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:16.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:15.923+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:16.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:15.807+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:16.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:15.807+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:16.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:15.807+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:16.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:15.807+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:16.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:16.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:16.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:16.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:16.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:16.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:16.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:16.492+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:17.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:16.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:17.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:16.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:17.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:16.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:17.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:16.927+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:17.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:16.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:17.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:16.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:17.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:16.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:17.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:16.856+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:17.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:17.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:17.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:17.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:17.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:17.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:17.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:17.476+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:18.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:17.879+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:18.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:17.879+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:18.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:17.879+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:18.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:17.880+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:18.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:17.859+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:18.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:17.859+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:18.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:17.859+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:18.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:17.859+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:18.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:18.462+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:18.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:18.462+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:18.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:18.462+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:18.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:18.462+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:19.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:18.908+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:19.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:18.908+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:19.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:18.908+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:19.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:18.908+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:19.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:18.886+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:19.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:18.886+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:19.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:18.886+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:19.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:18.887+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:19.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:19.432+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:19.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:19.432+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:19.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:19.432+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:19.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:19.433+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:20.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:19.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:20.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:19.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:20.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:19.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:20.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:19.886+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:20.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:19.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:20.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:19.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:20.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:19.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:20.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:19.883+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:20.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:20.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:20.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:20.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:20.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:20.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:20.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:20.410+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:21.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:13:20.929+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 3221 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:13:21.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:20.888+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:21.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:20.888+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:21.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:20.888+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:21.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:20.889+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:21.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:20.906+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:21.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:20.906+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:21.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:20.906+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:21.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:20.906+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:21.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:21.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:21.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:21.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:21.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:21.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:21.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:21.442+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:22.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:21.872+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:22.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:21.872+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:22.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:21.872+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:22.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:21.873+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:22.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:21.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:22.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:21.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:22.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:21.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:22.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:21.938+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:22.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:22.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:22.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:22.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:22.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:22.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:22.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:22.455+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:23.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:22.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:23.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:22.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:23.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:22.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:23.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:22.944+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:23.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:22.881+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:23.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:22.881+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:23.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:22.881+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:23.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:22.882+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:23.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:23.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:23.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:23.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:23.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:23.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:23.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:23.423+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:24.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:23.852+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:24.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:23.852+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:24.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:23.852+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:24.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:23.853+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:24.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:23.974+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:24.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:23.974+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:24.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:23.974+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:24.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:23.975+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:24.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:24.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:24.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:24.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:24.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:24.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:24.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:24.456+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:25.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:24.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:25.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:24.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:25.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:24.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:25.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:24.831+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:25.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:24.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:25.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:24.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:25.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:24.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:25.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:24.957+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:25.735 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:25.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:25.735 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:25.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:25.735 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:25.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:25.735 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:25.492+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:26.030 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:13:25.930+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 3314 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:13:26.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:25.799+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:26.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:25.799+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:26.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:25.799+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:26.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:25.800+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:26.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:25.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:26.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:25.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:26.032 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:25.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:26.032 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:25.912+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:26.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:26.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:26.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:26.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:26.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:26.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:26.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:26.527+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:27.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:26.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:27.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:26.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:27.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:26.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:27.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:26.864+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:27.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:26.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:27.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:26.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:27.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:26.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:27.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:26.839+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:27.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:27.552+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:27.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:27.552+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:27.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:27.552+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:27.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:27.552+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:28.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:27.859+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:28.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:27.859+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:28.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:27.859+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:28.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:27.860+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:28.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:27.858+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:28.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:27.858+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:28.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:27.858+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:28.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:27.859+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:28.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:28.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:28.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:28.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:28.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:28.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:28.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:28.556+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:29.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:28.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:29.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:28.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:29.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:28.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:29.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:28.903+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:29.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:28.811+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:29.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:28.811+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:29.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:28.811+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:29.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:28.811+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:29.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:29.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:29.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:29.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:29.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:29.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:29.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:29.549+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:29.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:29.775+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:29.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:29.775+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:29.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:29.776+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:29.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:29.776+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:30.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:29.919+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:30.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:29.919+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:30.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:29.919+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:30.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:29.920+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:30.909 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:30.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:30.909 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:30.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:30.909 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:30.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:30.909 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:30.546+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:30.909 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:30.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:30.910 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:30.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:30.910 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:30.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:30.910 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:30.743+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:31.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:13:30.930+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 3407 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:13:31.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:30.951+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:31.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:30.951+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:31.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:30.951+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:31.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:30.952+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:31.880 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:31.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:31.880 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:31.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:31.880 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:31.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:31.880 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:31.532+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:31.881 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:31.714+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:31.881 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:31.714+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:31.881 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:31.714+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:31.881 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:31.714+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:32.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:31.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:32.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:31.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:32.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:31.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:32.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:31.941+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:32.869 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:32.727+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:32.870 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:32.727+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:32.870 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:32.727+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:32.870 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:32.728+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:32.870 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:32.487+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:32.870 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:32.487+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:32.870 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:32.487+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:32.870 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:32.487+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:33.201 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:32.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:33.201 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:32.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:33.201 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:32.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:33.201 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:32.940+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:33.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:33.477+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:33.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:33.477+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:33.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:33.477+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:33.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:33.478+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:34.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:33.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:34.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:33.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:34.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:33.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:34.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:33.938+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:34.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:33.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:34.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:33.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:34.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:33.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:34.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:33.734+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:34.873 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:34.484+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:34.874 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:34.484+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:34.874 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:34.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:34.874 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:34.485+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:34.874 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:34.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:34.874 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:34.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:34.874 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:34.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:34.874 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:34.709+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:35.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:34.894+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:35.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:34.894+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:35.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:34.894+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:35.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:34.895+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:35.886 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:35.691+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:35.887 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:35.691+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:35.887 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:35.691+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:35.887 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:35.691+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:35.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:35.489+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:35.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:35.489+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:35.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:35.489+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:35.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:35.490+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:36.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:13:35.930+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 3501 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:13:36.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:35.907+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:36.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:35.907+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:36.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:35.907+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:36.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:35.908+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:36.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:36.447+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:36.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:36.447+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:36.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:36.447+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:36.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:36.448+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:36.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:36.650+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:36.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:36.650+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:36.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:36.650+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:36.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:36.651+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:37.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:36.957+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:37.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:36.957+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:37.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:36.957+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:37.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:36.958+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:37.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:37.697+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:37.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:37.697+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:37.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:37.697+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:37.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:37.698+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:37.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:37.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:37.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:37.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:37.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:37.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:37.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:37.473+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:38.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:37.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:38.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:37.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:38.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:37.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:38.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:37.975+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:38.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:38.453+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:38.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:38.453+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:38.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:38.453+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:38.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:38.454+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:38.990 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:38.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:38.990 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:38.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:38.990 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:38.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:38.990 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:38.730+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:39.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:39.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:39.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:39.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:39.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:39.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:39.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:39.011+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:39.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:39.460+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:39.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:39.460+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:39.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:39.460+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:39.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:39.460+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:40.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:39.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:40.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:39.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:40.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:39.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:40.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:39.966+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:40.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:39.737+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:40.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:39.737+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:40.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:39.737+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:40.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:39.738+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:40.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:40.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:40.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:40.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:40.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:40.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:40.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:40.464+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:41.034 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:13:40.931+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 3595 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:13:41.035 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:40.750+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:41.035 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:40.750+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:41.035 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:40.750+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:41.035 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:40.750+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:41.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:40.928+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:41.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:40.928+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:41.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:40.928+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:41.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:40.929+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:41.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:41.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:41.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:41.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:41.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:41.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:41.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:41.438+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:42.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:41.888+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:42.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:41.888+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:42.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:41.888+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:42.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:41.888+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:42.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:41.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:42.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:41.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:42.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:41.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:42.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:41.754+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:42.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:42.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:42.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:42.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:42.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:42.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:42.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:42.464+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:43.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:42.909+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:43.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:42.910+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:43.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:42.910+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:43.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:42.910+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:43.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:42.753+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:43.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:42.753+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:43.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:42.753+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:43.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:42.753+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:43.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:43.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:43.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:43.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:43.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:43.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:43.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:43.436+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:44.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:43.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:44.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:43.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:44.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:43.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:44.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:43.733+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:44.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:43.943+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:44.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:43.943+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:44.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:43.943+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:44.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:43.943+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:44.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:44.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:44.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:44.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:44.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:44.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:44.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:44.425+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:45.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:44.911+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:45.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:44.911+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:45.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:44.911+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:45.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:44.912+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:45.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:44.779+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:45.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:44.779+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:45.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:44.779+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:45.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:44.779+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:45.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:45.424+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:45.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:45.424+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:45.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:45.424+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:45.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:45.425+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:46.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:13:45.932+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 3689 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:13:46.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:45.867+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:46.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:45.867+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:46.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:45.867+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:46.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:45.868+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:46.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:45.816+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:46.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:45.816+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:46.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:45.816+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:46.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:45.817+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:46.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:46.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:46.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:46.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:46.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:46.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:46.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:46.475+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:47.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:46.892+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:47.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:46.892+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:47.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:46.892+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:47.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:46.894+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:47.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:46.789+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:47.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:46.789+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:47.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:46.789+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:47.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:46.790+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:47.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:47.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:47.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:47.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:47.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:47.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:47.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:47.487+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:48.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:47.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:48.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:47.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:48.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:47.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:48.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:47.888+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:48.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:47.810+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:48.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:47.810+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:48.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:47.810+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:48.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:47.810+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:48.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:48.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:48.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:48.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:48.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:48.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:48.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:48.468+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:49.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:48.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:49.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:48.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:49.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:48.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:49.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:48.839+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:49.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:48.811+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:49.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:48.811+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:49.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:48.811+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:49.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:48.811+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:49.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:49.517+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:49.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:49.517+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:49.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:49.517+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:49.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:49.518+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:50.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:49.843+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:50.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:49.843+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:50.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:49.843+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:50.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:49.844+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:50.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:49.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:50.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:49.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:50.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:49.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:50.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:49.812+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:50.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:50.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:50.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:50.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:50.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:50.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:50.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:50.482+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:51.033 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:13:50.933+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 3782 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:13:51.033 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:50.877+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:51.033 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:50.877+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:51.033 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:50.877+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:51.034 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:50.878+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:51.034 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:50.788+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:51.034 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:50.788+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:51.034 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:50.788+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:51.034 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:50.789+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:51.749 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:51.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:51.749 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:51.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:51.749 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:51.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:51.749 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:51.513+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:52.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:51.888+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:52.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:51.888+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:52.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:51.888+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:52.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:51.888+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:52.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:51.770+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:52.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:51.770+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:52.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:51.770+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:52.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:51.770+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:52.843 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:52.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:52.843 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:52.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:52.843 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:52.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:52.843 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:52.731+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:52.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:52.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:52.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:52.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:52.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:52.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:52.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:52.497+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:53.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:52.864+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:53.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:52.864+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:53.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:52.864+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:53.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:52.865+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:53.661 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:53.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:53.661 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:53.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:53.661 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:53.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:53.661 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:53.453+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:53.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:53.905+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:53.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:53.905+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:53.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:53.905+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:53.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:53.905+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:53.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:53.682+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:53.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:53.682+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:53.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:53.682+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:53.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:53.683+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:54.691 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:54.443+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:54.691 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:54.443+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:54.691 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:54.443+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:54.691 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:54.443+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:54.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:54.894+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:54.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:54.894+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:54.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:54.894+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:54.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:54.895+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:54.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:54.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:54.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:54.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:54.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:54.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:54.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:54.713+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:55.825 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:55.471+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:55.825 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:55.471+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:55.825 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:55.471+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:55.825 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:55.471+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:55.825 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:55.707+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:55.826 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:55.707+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:55.826 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:55.707+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:55.826 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:55.708+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:56.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:13:55.934+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 3876 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:13:56.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:55.846+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:56.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:55.846+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:56.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:55.846+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:56.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:55.847+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:56.842 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:56.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:56.842 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:56.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:56.842 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:56.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:56.842 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:56.489+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:56.843 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:56.671+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:56.843 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:56.671+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:56.843 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:56.671+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:56.843 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:56.672+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:57.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:56.864+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:57.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:56.864+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:57.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:56.864+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:57.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:56.864+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:57.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:57.626+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:57.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:57.626+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:57.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:57.626+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:57.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:57.627+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:57.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:57.441+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:57.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:57.441+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:57.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:57.441+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:57.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:57.442+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:58.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:57.904+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:58.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:57.904+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:58.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:57.904+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:58.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:57.905+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:58.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:58.584+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:58.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:58.584+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:58.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:58.584+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:58.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:58.584+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:58.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:58.451+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:58.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:58.451+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:58.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:58.451+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:58.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:58.452+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:59.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:58.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:59.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:58.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:59.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:58.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:13:59.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:58.886+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:59.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:59.406+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:59.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:59.406+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:59.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:59.406+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:13:59.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:13:59.407+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:13:59.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:59.625+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:59.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:59.625+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:59.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:59.625+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:13:59.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:13:59.625+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:00.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:59.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:00.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:59.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:00.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:59.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:00.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:13:59.932+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:00.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:00.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:00.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:00.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:00.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:00.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:00.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:00.421+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:00.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:00.620+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:00.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:00.620+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:00.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:00.620+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:00.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:00.621+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:01.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:14:00.935+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 3970 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:14:01.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:00.916+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:01.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:00.916+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:01.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:00.916+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:01.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:00.917+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:01.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:01.451+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:01.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:01.451+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:01.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:01.451+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:01.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:01.452+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:01.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:01.589+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:01.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:01.589+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:01.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:01.589+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:01.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:01.590+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:02.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:01.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:02.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:01.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:02.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:01.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:02.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:01.942+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:02.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:02.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:02.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:02.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:02.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:02.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:02.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:02.421+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:02.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:02.599+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:02.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:02.599+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:02.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:02.599+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:02.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:02.600+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:03.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:02.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:03.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:02.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:03.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:02.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:03.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:02.923+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:03.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:03.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:03.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:03.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:03.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:03.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:03.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:03.438+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:03.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:03.576+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:03.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:03.576+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:03.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:03.576+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:03.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:03.577+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:04.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:03.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:04.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:03.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:04.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:03.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:04.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:03.886+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:04.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:04.417+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:04.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:04.417+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:04.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:04.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:04.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:04.419+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:04.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:04.586+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:04.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:04.586+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:04.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:04.586+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:04.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:04.587+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:05.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:04.934+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:05.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:04.934+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:05.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:04.934+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:05.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:04.935+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:05.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:05.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:05.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:05.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:05.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:05.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:05.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:05.468+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:05.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:05.553+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:05.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:05.553+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:05.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:05.553+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:05.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:05.553+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:06.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:14:05.935+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 4064 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:14:06.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:05.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:06.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:05.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:06.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:05.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:06.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:05.969+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:06.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:06.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:06.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:06.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:06.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:06.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:06.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:06.465+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:06.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:06.517+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:06.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:06.517+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:06.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:06.517+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:06.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:06.518+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:07.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:06.944+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:07.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:06.944+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:07.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:06.944+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:07.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:06.944+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:07.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:07.487+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:07.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:07.487+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:07.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:07.487+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:07.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:07.488+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:07.943 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:07.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:07.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:07.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:07.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:07.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:07.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:07.529+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:08.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:07.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:08.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:07.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:08.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:07.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:08.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:07.965+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:08.744 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:08.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:08.744 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:08.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:08.744 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:08.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:08.744 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:08.506+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:08.744 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:08.514+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:08.744 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:08.514+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:08.744 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:08.514+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:08.745 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:08.514+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:09.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:08.916+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:09.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:08.916+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:09.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:08.916+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:09.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:08.916+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:09.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:09.459+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:09.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:09.459+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:09.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:09.459+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:09.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:09.459+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:09.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:09.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:09.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:09.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:09.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:09.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:09.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:09.529+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:10.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:09.889+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:10.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:09.889+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:10.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:09.889+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:10.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:09.890+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:10.851 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:10.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:10.851 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:10.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:10.851 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:10.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:10.851 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:10.579+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:10.852 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:10.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:10.852 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:10.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:10.852 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:10.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:10.852 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:10.503+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:11.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:14:10.936+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 4160 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:14:11.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:10.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:11.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:10.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:11.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:10.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:11.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:10.874+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:11.553 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:11.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:11.553 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:11.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:11.553 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:11.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:11.553 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:11.467+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:11.829 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:11.574+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:11.829 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:11.574+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:11.830 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:11.574+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:11.830 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:11.575+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:12.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:11.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:12.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:11.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:12.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:11.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:12.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:11.852+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:12.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:12.532+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:12.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:12.532+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:12.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:12.532+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:12.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:12.533+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:12.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:12.426+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:12.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:12.426+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:12.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:12.426+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:12.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:12.427+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:13.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:12.829+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:13.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:12.829+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:13.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:12.829+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:13.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:12.830+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:13.520 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:13.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:13.520 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:13.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:13.520 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:13.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:13.520 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:13.446+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:13.777 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:13.546+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:13.777 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:13.546+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:13.778 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:13.546+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:13.778 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:13.546+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:14.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:13.799+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:14.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:13.799+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:14.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:13.799+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:14.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:13.800+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:14.808 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:14.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:14.808 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:14.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:14.809 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:14.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:14.809 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:14.492+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:14.809 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:14.592+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:14.809 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:14.592+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:14.809 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:14.592+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:14.809 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:14.592+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:15.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:14.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:15.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:14.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:15.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:14.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:15.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:14.831+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:15.762 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:15.505+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:15.762 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:15.505+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:15.763 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:15.505+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:15.763 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:15.505+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:15.763 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:15.559+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:15.763 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:15.559+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:15.763 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:15.559+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:15.763 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:15.559+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:16.039 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:14:15.937+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 4258 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:14:16.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:15.784+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:16.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:15.784+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:16.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:15.784+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:16.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:15.785+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:16.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:16.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:16.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:16.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:16.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:16.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:16.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:16.520+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:16.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:16.531+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:16.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:16.531+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:16.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:16.531+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:16.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:16.531+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:17.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:16.817+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:17.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:16.817+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:17.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:16.817+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:17.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:16.818+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:17.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:17.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:17.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:17.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:17.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:17.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:17.815 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:17.546+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:17.815 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:17.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:17.815 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:17.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:17.815 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:17.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:17.815 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:17.568+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:18.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:17.836+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:18.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:17.836+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:18.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:17.836+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:18.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:17.837+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:18.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:18.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:18.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:18.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:18.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:18.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:18.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:18.547+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:18.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:18.551+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:18.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:18.551+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:18.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:18.551+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:18.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:18.551+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:19.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:18.839+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:19.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:18.839+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:19.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:18.839+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:19.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:18.840+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:19.794 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:19.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:19.794 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:19.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:19.794 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:19.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:19.794 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:19.556+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:19.794 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:19.536+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:19.794 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:19.536+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:19.795 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:19.536+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:19.795 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:19.537+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:20.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:19.815+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:20.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:19.815+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:20.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:19.815+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:20.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:19.816+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:20.795 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:20.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:20.795 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:20.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:20.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:20.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:20.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:20.543+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:20.796 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:20.533+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:20.796 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:20.533+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:20.796 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:20.533+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:20.797 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:20.533+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:21.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:14:20.938+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 4359 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:14:21.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:20.817+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:21.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:20.817+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:21.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:20.817+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:21.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:20.818+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:21.808 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:21.507+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:21.808 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:21.507+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:21.808 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:21.507+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:21.808 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:21.508+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:21.808 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:21.490+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:21.808 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:21.490+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:21.809 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:21.490+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:21.809 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:21.491+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:22.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:21.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:22.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:21.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:22.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:21.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:22.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:21.830+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:22.841 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:22.496+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:22.841 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:22.496+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:22.841 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:22.496+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:22.841 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:22.496+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:22.842 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:22.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:22.842 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:22.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:22.842 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:22.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:22.842 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:22.500+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:23.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:22.863+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:23.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:22.863+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:23.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:22.863+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:23.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:22.864+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:23.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:23.479+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:23.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:23.479+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:23.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:23.479+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:23.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:23.480+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:23.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:23.481+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:23.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:23.481+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:23.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:23.481+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:23.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:23.481+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:24.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:23.818+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:24.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:23.818+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:24.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:23.818+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:24.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:23.819+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:24.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:24.461+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:24.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:24.461+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:24.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:24.461+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:24.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:24.461+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:24.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:24.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:24.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:24.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:24.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:24.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:24.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:24.457+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:25.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:24.784+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:25.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:24.784+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:25.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:24.784+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:25.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:24.785+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:25.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:25.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:25.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:25.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:25.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:25.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:25.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:25.455+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:25.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:25.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:25.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:25.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:25.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:25.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:25.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:25.434+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:26.039 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:14:25.939+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 4458 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:14:26.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:25.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:26.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:25.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:26.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:25.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:26.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:25.740+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:26.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:26.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:26.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:26.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:26.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:26.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:26.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:26.412+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:26.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:26.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:26.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:26.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:26.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:26.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:26.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:26.397+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:27.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:26.747+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:27.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:26.747+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:27.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:26.747+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:27.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:26.748+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:27.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:27.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:27.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:27.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:27.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:27.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:27.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:27.404+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:27.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:27.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:27.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:27.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:27.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:27.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:27.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:27.426+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:28.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:27.773+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:28.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:27.773+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:28.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:27.773+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:28.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:27.773+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:28.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:28.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:28.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:28.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:28.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:28.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:28.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:28.433+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:28.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:28.442+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:28.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:28.442+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:28.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:28.442+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:28.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:28.443+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:29.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:28.787+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:29.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:28.787+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:29.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:28.787+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:29.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:28.788+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:29.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:29.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:29.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:29.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:29.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:29.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:29.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:29.426+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:29.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:29.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:29.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:29.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:29.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:29.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:29.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:29.395+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:30.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:29.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:30.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:29.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:30.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:29.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:30.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:29.825+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:30.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:30.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:30.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:30.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:30.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:30.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:30.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:30.397+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:30.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:30.421+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:30.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:30.421+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:30.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:30.421+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:30.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:30.421+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:31.039 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:14:30.939+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 4559 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:14:31.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:30.796+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:31.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:30.796+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:31.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:30.796+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:31.040 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:30.797+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:31.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:31.432+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:31.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:31.432+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:31.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:31.432+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:31.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:31.434+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:31.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:31.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:31.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:31.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:31.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:31.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:31.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:31.397+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:32.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:31.751+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:32.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:31.751+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:32.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:31.751+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:32.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:31.752+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:32.686 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:32.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:32.686 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:32.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:32.686 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:32.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:32.686 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:32.451+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:32.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:32.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:32.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:32.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:32.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:32.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:32.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:32.379+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:32.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:32.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:32.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:32.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:32.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:32.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:32.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:32.708+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:33.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:33.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:33.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:33.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:33.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:33.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:33.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:33.407+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:33.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:33.453+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:33.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:33.453+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:33.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:33.453+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:33.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:33.454+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:34.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:33.740+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:34.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:33.740+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:34.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:33.740+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:34.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:33.741+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:34.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:34.468+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:34.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:34.468+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:34.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:34.468+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:34.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:34.469+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:34.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:34.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:34.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:34.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:34.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:34.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:34.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:34.443+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:35.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:34.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:35.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:34.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:35.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:34.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:35.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:34.721+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:35.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:35.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:35.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:35.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:35.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:35.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:35.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:35.496+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:35.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:35.411+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:35.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:35.411+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:35.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:35.411+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:35.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:35.412+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:35.948 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:14:35.939+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 4653 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:14:35.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:35.715+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:35.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:35.715+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:35.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:35.715+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:35.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:35.716+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:36.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:36.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:36.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:36.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:36.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:36.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:36.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:36.523+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:36.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:36.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:36.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:36.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:36.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:36.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:36.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:36.382+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:36.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:36.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:36.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:36.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:36.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:36.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:36.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:36.712+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:37.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:37.477+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:37.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:37.477+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:37.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:37.477+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:37.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:37.477+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:37.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:37.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:37.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:37.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:37.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:37.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:37.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:37.408+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:37.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:37.713+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:37.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:37.713+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:37.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:37.713+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:37.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:37.713+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:38.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:38.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:38.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:38.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:38.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:38.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:38.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:38.410+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:38.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:38.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:38.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:38.489+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:38.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:38.489+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:38.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:38.489+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:39.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:38.756+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:39.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:38.756+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:39.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:38.756+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:39.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:38.757+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:39.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:39.427+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:39.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:39.427+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:39.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:39.427+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:39.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:39.427+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:39.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:39.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:39.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:39.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:39.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:39.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:39.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:39.521+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:40.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:39.777+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:40.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:39.777+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:40.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:39.777+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:40.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:39.777+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:40.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:40.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:40.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:40.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:40.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:40.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:40.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:40.438+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:40.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:40.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:40.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:40.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:40.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:40.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:40.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:40.476+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:41.042 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:14:40.940+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 4747 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:14:41.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:40.802+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:41.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:40.802+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:41.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:40.802+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:41.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:40.803+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:41.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:41.435+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:41.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:41.435+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:41.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:41.435+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:41.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:41.436+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:41.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:41.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:41.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:41.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:41.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:41.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:41.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:41.416+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:42.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:41.812+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:42.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:41.812+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:42.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:41.812+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:42.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:41.813+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:42.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:42.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:42.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:42.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:42.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:42.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:42.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:42.422+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:42.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:42.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:42.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:42.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:42.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:42.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:42.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:42.406+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:43.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:42.781+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:43.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:42.782+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:43.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:42.782+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:43.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:42.782+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:43.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:43.377+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:43.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:43.377+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:43.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:43.377+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:43.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:43.377+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:43.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:43.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:43.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:43.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:43.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:43.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:43.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:43.409+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:44.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:43.760+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:44.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:43.760+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:44.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:43.760+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:44.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:43.760+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:44.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:44.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:44.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:44.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:44.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:44.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:44.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:44.377+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:44.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:44.408+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:44.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:44.408+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:44.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:44.408+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:44.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:44.408+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:45.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:44.724+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:45.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:44.724+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:45.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:44.724+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:45.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:44.724+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:45.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:45.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:45.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:45.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:45.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:45.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:45.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:45.413+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:45.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:45.424+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:45.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:45.424+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:45.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:45.424+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:45.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:45.424+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:45.948 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:14:45.941+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 4842 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:14:45.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:45.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:45.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:45.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:45.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:45.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:45.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:45.720+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:46.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:46.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:46.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:46.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:46.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:46.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:46.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:46.441+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:46.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:46.431+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:46.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:46.431+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:46.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:46.431+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:46.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:46.431+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:46.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:46.704+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:46.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:46.704+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:46.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:46.704+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:46.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:46.706+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:47.662 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:47.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:47.662 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:47.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:47.662 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:47.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:47.662 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:47.383+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:47.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:47.491+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:47.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:47.491+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:47.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:47.491+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:47.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:47.491+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:47.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:47.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:47.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:47.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:47.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:47.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:47.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:47.685+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:48.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:48.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:48.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:48.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:48.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:48.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:48.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:48.421+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:48.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:48.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:48.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:48.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:48.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:48.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:48.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:48.668+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:48.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:48.471+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:48.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:48.471+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:48.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:48.471+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:48.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:48.471+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:49.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:49.655+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:49.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:49.655+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:49.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:49.655+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:49.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:49.655+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:49.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:49.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:49.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:49.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:49.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:49.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:49.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:49.460+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:49.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:49.431+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:49.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:49.431+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:49.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:49.431+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:49.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:49.432+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:50.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:50.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:50.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:50.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:50.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:50.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:50.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:50.621+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:50.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:50.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:50.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:50.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:50.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:50.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:50.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:50.473+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:50.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:50.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:50.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:50.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:50.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:50.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:50.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:50.428+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:51.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:14:50.943+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 4960 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:14:51.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:51.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:51.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:51.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:51.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:51.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:51.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:51.591+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:51.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:51.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:51.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:51.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:51.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:51.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:51.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:51.456+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:51.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:51.398+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:51.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:51.398+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:51.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:51.398+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:51.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:51.399+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:52.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:52.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:52.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:52.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:52.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:52.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:52.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:52.542+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:52.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:52.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:52.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:52.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:52.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:52.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:52.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:52.421+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:52.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:52.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:52.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:52.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:52.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:52.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:52.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:52.374+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:53.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:53.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:53.617 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:53.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:53.617 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:53.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:53.617 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:53.553+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:53.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:53.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:53.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:53.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:53.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:53.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:53.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:53.347+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:53.618 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:53.374+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:53.618 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:53.374+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:53.618 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:53.374+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:53.618 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:53.375+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:54.566 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:54.328+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:54.566 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:54.328+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:54.566 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:54.328+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:54.566 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:54.329+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:54.566 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:54.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:54.567 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:54.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:54.567 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:54.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:54.567 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:54.390+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:54.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:54.588+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:54.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:54.588+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:54.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:54.588+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:54.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:54.589+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:55.562 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:55.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:55.562 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:55.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:55.562 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:55.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:55.562 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:55.317+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:55.563 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:55.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:55.563 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:55.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:55.563 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:55.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:55.563 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:55.380+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:55.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:55.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:55.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:55.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:55.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:55.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:55.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:55.585+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:56.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:14:55.943+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 5067 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:14:56.523 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:56.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:56.523 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:56.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:56.523 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:56.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:56.523 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:56.418+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:56.524 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:56.285+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:56.524 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:56.285+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:56.524 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:56.285+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:56.524 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:56.285+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:56.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:56.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:56.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:56.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:56.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:56.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:56.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:56.546+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:57.530 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:57.244+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:57.530 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:57.244+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:57.530 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:57.244+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:57.530 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:57.245+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:57.530 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:57.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:57.530 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:57.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:57.530 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:57.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:57.531 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:57.445+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:57.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:57.551+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:57.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:57.551+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:57.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:57.551+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:57.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:57.552+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:58.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:58.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:58.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:58.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:58.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:58.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:58.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:58.219+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:58.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:58.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:58.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:58.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:58.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:58.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:58.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:58.418+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:58.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:58.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:58.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:58.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:58.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:58.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:58.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:58.542+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:59.560 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:59.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:59.561 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:59.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:59.561 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:59.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:14:59.561 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:14:59.381+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:59.561 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:59.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:59.561 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:59.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:59.561 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:59.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:14:59.561 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:14:59.259+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:14:59.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:59.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:59.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:59.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:59.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:59.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:14:59.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:14:59.583+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:00.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:00.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:00.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:00.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:00.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:00.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:00.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:00.384+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:00.574 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:00.308+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:00.574 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:00.308+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:00.574 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:00.308+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:00.574 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:00.309+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:00.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:00.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:00.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:00.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:00.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:00.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:00.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:00.596+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:01.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:15:00.944+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 5161 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:15:01.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:01.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:01.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:01.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:01.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:01.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:01.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:01.419+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:01.543 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:01.265+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:01.545 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:01.265+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:01.545 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:01.265+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:01.546 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:01.266+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:01.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:01.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:01.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:01.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:01.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:01.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:01.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:01.566+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:02.522 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:02.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:02.522 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:02.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:02.522 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:02.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:02.522 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:02.420+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:02.523 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:02.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:02.523 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:02.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:02.523 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:02.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:02.523 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:02.271+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:02.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:02.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:02.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:02.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:02.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:02.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:02.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:02.545+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:03.512 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:03.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:03.512 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:03.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:03.512 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:03.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:03.512 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:03.405+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:03.513 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:03.271+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:03.513 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:03.271+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:03.513 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:03.271+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:03.513 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:03.272+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:03.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:03.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:03.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:03.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:03.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:03.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:03.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:03.535+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:04.356 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:04.229+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:04.356 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:04.229+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:04.356 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:04.229+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:04.356 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:04.230+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:04.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:04.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:04.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:04.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:04.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:04.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:04.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:04.378+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:04.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:04.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:04.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:04.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:04.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:04.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:04.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:04.521+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:05.519 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:05.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:05.519 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:05.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:05.519 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:05.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:05.519 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:05.240+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:05.519 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:05.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:05.519 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:05.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:05.519 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:05.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:05.520 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:05.338+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:05.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:05.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:05.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:05.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:05.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:05.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:05.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:05.542+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:06.194 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:15:05.945+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 5254 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:15:06.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:06.216+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:06.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:06.216+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:06.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:06.216+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:06.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:06.217+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:06.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:06.304+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:06.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:06.304+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:06.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:06.304+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:06.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:06.304+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:06.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:06.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:06.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:06.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:06.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:06.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:06.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:06.538+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:07.516 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:07.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:07.517 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:07.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:07.517 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:07.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:07.517 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:07.353+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:07.517 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:07.220+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:07.517 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:07.220+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:07.517 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:07.220+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:07.517 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:07.221+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:07.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:07.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:07.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:07.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:07.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:07.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:07.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:07.539+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:08.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:08.174+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:08.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:08.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:08.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:08.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:08.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:08.175+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:08.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:08.345+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:08.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:08.345+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:08.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:08.345+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:08.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:08.345+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:08.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:08.555+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:08.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:08.555+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:08.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:08.555+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:08.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:08.556+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:09.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:09.213+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:09.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:09.213+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:09.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:09.214+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:09.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:09.214+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:09.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:09.317+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:09.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:09.317+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:09.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:09.317+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:09.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:09.318+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:09.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:09.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:09.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:09.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:09.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:09.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:09.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:09.558+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:10.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:10.213+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:10.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:10.213+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:10.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:10.213+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:10.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:10.214+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:10.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:10.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:10.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:10.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:10.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:10.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:10.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:10.367+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:10.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:10.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:10.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:10.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:10.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:10.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:10.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:10.515+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:11.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:15:10.946+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 5347 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:15:11.518 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:11.235+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:11.519 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:11.235+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:11.519 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:11.235+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:11.519 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:11.236+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:11.519 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:11.317+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:11.519 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:11.317+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:11.519 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:11.317+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:11.519 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:11.318+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:11.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:11.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:11.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:11.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:11.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:11.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:11.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:11.541+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:12.537 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:12.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:12.537 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:12.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:12.537 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:12.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:12.537 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:12.336+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:12.537 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:12.266+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:12.537 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:12.266+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:12.537 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:12.266+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:12.537 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:12.267+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:12.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:12.559+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:12.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:12.559+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:12.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:12.559+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:12.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:12.559+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:13.524 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:13.290+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:13.524 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:13.290+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:13.524 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:13.290+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:13.524 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:13.291+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:13.524 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:13.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:13.524 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:13.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:13.524 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:13.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:13.525 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:13.376+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:13.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:13.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:13.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:13.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:13.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:13.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:13.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:13.579+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:14.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:14.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:14.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:14.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:14.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:14.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:14.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:14.251+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:14.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:14.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:14.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:14.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:14.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:14.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:14.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:14.368+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:14.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:14.580+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:14.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:14.580+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:14.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:14.580+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:14.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:14.581+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:15.574 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:15.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:15.574 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:15.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:15.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:15.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:15.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:15.280+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:15.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:15.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:15.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:15.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:15.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:15.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:15.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:15.402+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:15.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:15.596+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:15.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:15.596+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:15.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:15.596+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:15.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:15.598+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:16.197 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:15:15.947+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 5440 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:15:16.529 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:16.284+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:16.529 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:16.284+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:16.529 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:16.284+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:16.530 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:16.285+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:16.530 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:16.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:16.530 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:16.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:16.530 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:16.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:16.530 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:16.353+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:16.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:16.551+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:16.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:16.551+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:16.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:16.551+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:16.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:16.552+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:17.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:17.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:17.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:17.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:17.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:17.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:17.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:17.351+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:17.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:17.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:17.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:17.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:17.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:17.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:17.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:17.506+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:17.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:17.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:17.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:17.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:17.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:17.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:17.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:17.306+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:18.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:18.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:18.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:18.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:18.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:18.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:18.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:18.399+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:18.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:18.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:18.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:18.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:18.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:18.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:18.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:18.526+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:18.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:18.326+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:18.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:18.326+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:18.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:18.326+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:18.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:18.327+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:19.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:19.355+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:19.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:19.355+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:19.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:19.355+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:19.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:19.355+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:19.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:19.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:19.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:19.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:19.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:19.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:19.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:19.512+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:19.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:19.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:19.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:19.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:19.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:19.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:19.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:19.292+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:20.514 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:20.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:20.514 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:20.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:20.514 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:20.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:20.514 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:20.320+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:20.514 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:20.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:20.514 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:20.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:20.515 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:20.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:20.515 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:20.281+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:20.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:20.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:20.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:20.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:20.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:20.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:20.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:20.537+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:21.197 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:15:20.947+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 5533 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:15:21.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:21.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:21.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:21.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:21.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:21.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:21.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:21.325+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:21.544 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:21.286+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:21.545 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:21.286+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:21.545 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:21.286+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:21.545 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:21.287+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:21.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:21.566+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:21.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:21.566+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:21.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:21.566+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:21.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:21.567+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:22.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:22.298+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:22.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:22.298+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:22.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:22.298+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:22.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:22.299+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:22.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:22.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:22.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:22.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:22.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:22.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:22.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:22.331+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:22.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:22.522+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:22.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:22.522+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:22.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:22.522+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:22.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:22.522+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:23.524 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:23.370+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:23.524 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:23.370+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:23.524 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:23.370+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:23.524 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:23.370+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:23.524 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:23.274+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:23.525 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:23.274+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:23.525 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:23.274+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:23.525 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:23.275+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:23.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:23.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:23.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:23.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:23.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:23.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:23.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:23.549+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:24.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:24.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:24.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:24.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:24.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:24.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:24.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:24.416+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:24.576 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:24.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:24.576 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:24.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:24.576 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:24.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:24.576 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:24.307+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:24.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:24.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:24.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:24.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:24.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:24.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:24.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:24.598+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:25.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:25.461+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:25.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:25.461+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:25.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:25.461+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:25.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:25.461+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:25.543 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:25.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:25.543 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:25.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:25.543 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:25.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:25.543 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:25.281+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:25.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:25.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:25.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:25.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:25.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:25.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:25.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:25.585+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:26.197 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:15:25.948+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 5626 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:15:26.552 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:26.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:26.552 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:26.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:26.552 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:26.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:26.552 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:26.414+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:26.553 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:26.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:26.553 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:26.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:26.553 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:26.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:26.553 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:26.240+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:26.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:26.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:26.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:26.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:26.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:26.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:26.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:26.575+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:27.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:27.463+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:27.584 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:27.463+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:27.584 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:27.463+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:27.584 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:27.464+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:27.584 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:27.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:27.584 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:27.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:27.584 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:27.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:27.584 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:27.250+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:27.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:27.605+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:27.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:27.605+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:27.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:27.605+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:27.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:27.606+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:28.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:28.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:28.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:28.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:28.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:28.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:28.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:28.439+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:28.576 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:28.249+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:28.576 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:28.249+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:28.576 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:28.249+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:28.576 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:28.249+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:28.935 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:28.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:28.935 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:28.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:28.935 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:28.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:28.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:28.598+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:29.527 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:29.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:29.527 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:29.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:29.528 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:29.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:29.528 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:29.444+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:29.528 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:29.232+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:29.528 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:29.232+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:29.528 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:29.232+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:29.528 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:29.232+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:29.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:29.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:29.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:29.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:29.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:29.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:29.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:29.550+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:30.559 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:30.248+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:30.559 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:30.248+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:30.559 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:30.248+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:30.559 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:30.249+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:30.560 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:30.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:30.560 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:30.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:30.560 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:30.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:30.560 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:30.489+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:30.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:30.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:30.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:30.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:30.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:30.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:30.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:30.582+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:31.197 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:15:30.949+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 5719 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:15:31.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:31.462+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:31.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:31.462+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:31.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:31.462+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:31.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:31.463+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:31.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:31.292+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:31.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:31.292+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:31.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:31.292+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:31.576 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:31.293+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:31.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:31.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:31.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:31.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:31.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:31.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:31.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:31.598+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:32.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:32.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:32.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:32.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:32.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:32.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:32.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:32.554+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:32.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:32.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:32.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:32.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:32.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:32.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:32.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:32.314+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:32.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:32.435+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:32.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:32.435+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:32.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:32.435+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:32.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:32.435+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:33.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:33.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:33.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:33.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:33.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:33.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:33.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:33.594+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:33.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:33.350+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:33.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:33.350+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:33.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:33.350+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:33.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:33.351+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:33.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:33.437+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:33.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:33.437+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:33.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:33.437+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:33.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:33.438+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:34.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:34.381+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:34.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:34.381+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:34.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:34.381+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:34.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:34.382+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:34.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:34.417+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:34.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:34.417+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:34.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:34.417+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:34.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:34.418+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:34.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:34.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:34.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:34.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:34.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:34.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:34.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:34.618+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:35.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:35.415+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:35.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:35.415+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:35.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:35.415+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:35.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:35.416+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:35.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:35.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:35.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:35.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:35.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:35.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:35.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:35.383+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:35.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:35.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:35.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:35.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:35.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:35.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:35.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:35.583+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:36.197 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:15:35.949+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 5813 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:15:36.602 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:36.414+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:36.602 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:36.414+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:36.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:36.414+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:36.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:36.414+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:36.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:36.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:36.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:36.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:36.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:36.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:36.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:36.354+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:36.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:36.623+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:36.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:36.623+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:36.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:36.623+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:36.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:36.624+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:37.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:37.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:37.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:37.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:37.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:37.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:37.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:37.609+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:37.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:37.386+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:37.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:37.386+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:37.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:37.386+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:37.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:37.387+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:37.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:37.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:37.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:37.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:37.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:37.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:37.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:37.393+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:38.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:38.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:38.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:38.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:38.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:38.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:38.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:38.562+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:38.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:38.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:38.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:38.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:38.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:38.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:38.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:38.404+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:38.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:38.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:38.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:38.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:38.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:38.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:38.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:38.402+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:39.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:39.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:39.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:39.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:39.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:39.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:39.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:39.564+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:39.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:39.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:39.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:39.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:39.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:39.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:39.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:39.422+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:39.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:39.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:39.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:39.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:39.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:39.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:39.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:39.373+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:40.564 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:40.430+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:40.564 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:40.430+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:40.565 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:40.430+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:40.565 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:40.430+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:40.565 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:40.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:40.565 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:40.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:40.565 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:40.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:40.565 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:40.329+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:40.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:40.585+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:40.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:40.585+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:40.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:40.585+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:40.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:40.586+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:41.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:15:40.950+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 5906 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:15:41.606 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:41.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:41.606 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:41.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:41.606 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:41.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:41.609 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:41.344+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:41.610 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:41.441+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:41.610 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:41.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:41.610 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:41.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:41.610 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:41.442+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:41.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:41.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:41.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:41.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:41.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:41.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:41.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:41.628+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:42.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:42.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:42.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:42.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:42.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:42.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:42.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:42.313+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:42.627 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:42.422+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:42.627 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:42.422+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:42.627 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:42.422+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:42.627 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:42.423+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:42.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:42.644+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:42.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:42.644+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:42.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:42.644+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:42.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:42.645+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:43.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:43.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:43.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:43.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:43.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:43.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:43.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:43.467+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:43.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:43.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:43.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:43.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:43.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:43.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:43.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:43.302+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:43.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:43.674+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:43.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:43.674+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:43.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:43.674+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:43.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:43.675+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:44.646 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:44.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:44.646 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:44.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:44.646 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:44.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:44.646 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:44.498+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:44.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:44.327+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:44.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:44.327+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:44.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:44.327+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:44.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:44.327+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:44.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:44.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:44.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:44.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:44.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:44.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:44.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:44.668+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:45.628 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:45.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:45.629 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:45.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:45.629 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:45.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:45.629 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:45.492+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:45.629 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:45.321+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:45.629 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:45.321+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:45.629 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:45.321+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:45.629 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:45.321+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:45.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:45.650+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:45.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:45.650+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:45.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:45.650+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:45.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:45.651+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:46.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:15:45.951+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 5999 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:15:46.600 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:46.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:46.601 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:46.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:46.601 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:46.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:46.601 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:46.457+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:46.601 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:46.342+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:46.601 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:46.342+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:46.601 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:46.342+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:46.602 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:46.342+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:46.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:46.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:46.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:46.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:46.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:46.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:46.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:46.623+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:47.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:47.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:47.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:47.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:47.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:47.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:47.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:47.475+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:47.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:47.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:47.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:47.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:47.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:47.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:47.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:47.328+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:47.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:47.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:47.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:47.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:47.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:47.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:47.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:47.667+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:48.599 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:48.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:48.600 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:48.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:48.600 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:48.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:48.600 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:48.457+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:48.600 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:48.349+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:48.600 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:48.349+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:48.600 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:48.349+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:48.600 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:48.350+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:48.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:48.621+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:48.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:48.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:48.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:48.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:48.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:48.622+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:49.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:49.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:49.420 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:49.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:49.420 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:49.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:49.420 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:49.374+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:49.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:49.635+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:49.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:49.635+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:49.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:49.635+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:49.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:49.636+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:49.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:49.441+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:49.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:49.441+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:49.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:49.441+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:49.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:49.443+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:50.384 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:50.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:50.384 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:50.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:50.384 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:50.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:50.384 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:50.356+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:50.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:50.406+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:50.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:50.406+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:50.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:50.406+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:50.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:50.407+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:50.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:50.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:50.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:50.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:50.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:50.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:50.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:50.640+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:51.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:15:50.953+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 6092 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:15:51.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:51.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:51.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:51.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:51.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:51.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:51.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:51.621+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:51.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:51.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:51.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:51.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:51.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:51.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:51.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:51.373+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:51.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:51.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:51.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:51.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:51.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:51.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:51.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:51.389+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:52.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:52.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:52.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:52.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:52.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:52.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:52.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:52.394+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:52.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:52.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:52.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:52.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:52.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:52.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:52.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:52.599+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:52.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:52.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:52.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:52.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:52.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:52.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:52.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:52.376+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:53.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:53.355+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:53.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:53.355+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:53.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:53.355+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:53.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:53.355+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:53.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:53.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:53.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:53.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:53.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:53.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:53.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:53.424+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:53.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:53.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:53.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:53.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:53.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:53.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:53.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:53.608+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:54.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:54.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:54.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:54.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:54.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:54.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:54.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:54.366+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:54.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:54.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:54.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:54.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:54.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:54.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:54.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:54.387+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:54.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:54.639+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:54.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:54.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:54.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:54.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:54.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:54.640+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:55.598 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:55.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:55.598 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:55.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:55.599 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:55.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:55.599 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:55.412+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:55.599 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:55.355+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:55.599 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:55.355+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:55.599 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:55.355+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:55.599 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:55.355+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:55.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:55.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:55.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:55.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:55.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:55.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:55.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:55.621+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:56.197 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:15:55.953+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 6185 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:15:56.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:56.424+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:56.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:56.424+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:56.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:56.424+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:56.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:56.424+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:56.631 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:56.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:56.631 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:56.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:56.631 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:56.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:56.631 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:56.332+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:56.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:56.652+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:56.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:56.652+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:56.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:56.652+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:56.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:56.653+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:57.663 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:57.412+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:57.663 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:57.412+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:57.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:57.412+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:57.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:57.413+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:57.664 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:57.346+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:57.664 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:57.346+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:57.664 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:57.346+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:57.664 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:57.347+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:57.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:57.685+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:57.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:57.685+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:57.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:57.685+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:57.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:57.686+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:58.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:58.377+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:58.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:58.377+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:58.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:58.377+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:58.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:58.378+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:58.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:58.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:58.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:58.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:58.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:58.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:58.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:58.316+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:58.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:58.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:58.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:58.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:58.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:58.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:58.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:58.719+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:59.678 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:59.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:59.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:59.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:59.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:59.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:15:59.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:15:59.424+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:59.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:59.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:59.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:59.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:59.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:59.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:15:59.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:15:59.347+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:15:59.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:59.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:59.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:59.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:59.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:59.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:15:59.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:15:59.701+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:00.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:00.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:00.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:00.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:00.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:00.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:00.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:00.471+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:00.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:00.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:00.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:00.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:00.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:00.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:00.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:00.396+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:01.056 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:16:00.954+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 6278 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:16:01.056 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:00.741+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:01.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:00.741+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:01.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:00.741+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:01.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:00.742+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:01.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:01.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:01.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:01.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:01.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:01.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:01.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:01.521+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:01.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:01.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:01.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:01.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:01.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:01.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:01.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:01.424+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:02.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:01.788+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:02.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:01.788+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:02.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:01.788+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:02.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:01.789+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:02.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:02.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:02.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:02.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:02.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:02.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:02.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:02.568+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:02.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:02.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:02.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:02.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:02.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:02.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:02.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:02.467+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:03.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:02.817+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:03.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:02.817+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:03.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:02.817+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:03.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:02.818+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:03.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:03.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:03.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:03.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:03.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:03.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:03.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:03.593+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:03.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:03.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:03.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:03.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:03.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:03.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:03.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:03.437+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:04.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:03.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:04.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:03.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:04.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:03.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:04.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:03.861+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:04.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:04.454+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:04.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:04.454+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:04.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:04.454+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:04.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:04.454+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:04.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:04.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:04.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:04.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:04.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:04.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:04.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:04.596+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:05.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:04.882+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:05.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:04.882+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:05.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:04.882+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:05.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:04.883+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:05.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:05.601+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:05.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:05.601+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:05.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:05.601+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:05.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:05.602+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:05.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:05.454+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:05.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:05.454+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:05.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:05.454+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:05.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:05.454+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:06.197 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:16:05.955+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 6368 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:16:06.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:05.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:06.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:05.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:06.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:05.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:06.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:05.925+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:06.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:06.566+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:06.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:06.566+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:06.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:06.566+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:06.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:06.566+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:06.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:06.442+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:06.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:06.442+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:06.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:06.442+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:06.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:06.442+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:07.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:06.884+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:07.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:06.884+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:07.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:06.884+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:07.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:06.885+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:07.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:07.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:07.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:07.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:07.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:07.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:07.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:07.570+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:07.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:07.429+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:07.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:07.429+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:07.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:07.429+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:07.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:07.430+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:08.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:07.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:08.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:07.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:08.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:07.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:08.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:07.838+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:08.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:08.552+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:08.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:08.552+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:08.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:08.552+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:08.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:08.553+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:08.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:08.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:08.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:08.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:08.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:08.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:08.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:08.466+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:09.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:08.811+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:09.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:08.811+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:09.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:08.811+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:09.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:08.812+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:09.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:09.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:09.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:09.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:09.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:09.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:09.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:09.550+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:09.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:09.439+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:09.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:09.439+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:09.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:09.439+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:09.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:09.439+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:10.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:09.792+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:10.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:09.792+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:10.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:09.792+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:10.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:09.794+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:10.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:10.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:10.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:10.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:10.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:10.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:10.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:10.596+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:10.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:10.463+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:10.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:10.463+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:10.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:10.463+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:10.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:10.463+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:11.056 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:16:10.956+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 6463 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:16:11.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:10.773+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:11.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:10.773+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:11.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:10.773+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:11.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:10.774+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:11.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:11.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:11.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:11.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:11.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:11.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:11.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:11.623+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:11.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:11.733+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:11.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:11.733+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:11.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:11.733+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:11.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:11.734+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:11.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:11.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:11.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:11.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:11.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:11.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:11.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:11.505+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:12.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:12.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:12.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:12.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:12.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:12.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:12.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:12.632+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:12.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:12.702+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:12.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:12.702+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:12.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:12.702+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:12.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:12.703+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:12.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:12.468+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:12.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:12.468+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:12.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:12.468+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:12.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:12.469+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:13.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:13.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:13.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:13.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:13.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:13.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:13.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:13.610+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:13.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:13.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:13.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:13.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:13.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:13.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:13.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:13.424+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:13.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:13.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:13.996 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:13.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:13.996 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:13.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:13.997 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:13.719+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:14.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:14.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:14.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:14.626+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:14.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:14.626+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:14.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:14.626+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:14.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:14.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:14.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:14.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:14.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:14.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:14.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:14.428+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:14.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:14.764+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:14.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:14.764+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:14.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:14.764+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:14.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:14.765+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:15.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:15.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:15.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:15.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:15.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:15.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:15.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:15.624+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:15.765 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:15.472+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:15.765 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:15.472+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:15.765 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:15.472+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:15.765 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:15.472+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:16.061 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:16:15.957+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 6555 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:16:16.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:15.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:16.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:15.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:16.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:15.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:16.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:15.788+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:16.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:16.603+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:16.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:16.603+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:16.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:16.603+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:16.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:16.604+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:16.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:16.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:16.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:16.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:16.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:16.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:16.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:16.430+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:17.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:16.761+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:17.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:16.761+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:17.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:16.761+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:17.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:16.761+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:17.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:17.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:17.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:17.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:17.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:17.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:17.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:17.406+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:17.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:17.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:17.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:17.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:17.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:17.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:17.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:17.584+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:18.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:17.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:18.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:17.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:18.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:17.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:18.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:17.786+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:18.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:18.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:18.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:18.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:18.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:18.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:18.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:18.442+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:18.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:18.573+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:18.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:18.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:18.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:18.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:18.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:18.574+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:19.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:18.816+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:19.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:18.816+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:19.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:18.816+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:19.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:18.817+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:19.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:19.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:19.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:19.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:19.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:19.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:19.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:19.528+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:19.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:19.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:19.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:19.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:19.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:19.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:19.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:19.405+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:20.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:19.849+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:20.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:19.849+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:20.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:19.849+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:20.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:19.850+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:20.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:20.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:20.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:20.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:20.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:20.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:20.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:20.487+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:20.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:20.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:20.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:20.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:20.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:20.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:20.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:20.368+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:21.062 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:16:20.958+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 6649 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:16:21.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:20.828+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:21.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:20.828+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:21.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:20.828+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:21.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:20.829+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:21.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:21.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:21.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:21.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:21.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:21.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:21.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:21.354+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:21.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:21.514+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:21.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:21.514+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:21.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:21.514+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:21.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:21.516+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:22.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:21.784+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:22.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:21.784+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:22.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:21.784+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:22.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:21.785+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:22.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:22.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:22.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:22.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:22.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:22.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:22.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:22.494+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:22.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:22.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:22.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:22.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:22.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:22.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:22.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:22.353+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:23.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:22.762+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:23.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:22.762+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:23.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:22.762+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:23.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:22.762+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:23.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:23.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:23.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:23.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:23.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:23.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:23.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:23.369+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:23.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:23.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:23.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:23.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:23.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:23.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:23.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:23.449+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:24.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:23.736+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:24.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:23.736+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:24.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:23.736+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:24.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:23.737+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:24.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:24.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:24.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:24.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:24.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:24.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:24.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:24.348+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:24.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:24.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:24.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:24.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:24.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:24.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:24.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:24.457+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:25.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:24.767+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:25.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:24.767+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:25.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:24.767+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:25.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:24.768+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:25.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:25.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:25.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:25.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:25.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:25.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:25.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:25.473+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:25.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:25.346+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:25.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:25.346+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:25.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:25.346+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:25.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:25.346+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:26.061 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:16:25.959+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 6743 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:16:26.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:25.767+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:26.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:25.767+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:26.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:25.767+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:26.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:25.767+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:26.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:26.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:26.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:26.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:26.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:26.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:26.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:26.326+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:26.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:26.489+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:26.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:26.489+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:26.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:26.489+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:26.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:26.490+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:27.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:26.759+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:27.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:26.759+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:27.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:26.759+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:27.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:26.760+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:27.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:27.280+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:27.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:27.280+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:27.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:27.280+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:27.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:27.281+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:27.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:27.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:27.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:27.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:27.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:27.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:27.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:27.500+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:27.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:27.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:27.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:27.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:27.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:27.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:27.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:27.721+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:28.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:28.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:28.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:28.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:28.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:28.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:28.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:28.458+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:28.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:28.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:28.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:28.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:28.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:28.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:28.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:28.326+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:29.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:28.748+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:29.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:28.748+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:29.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:28.748+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:29.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:28.749+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:29.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:29.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:29.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:29.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:29.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:29.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:29.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:29.284+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:29.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:29.415+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:29.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:29.415+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:29.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:29.415+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:29.686 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:29.416+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:29.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:29.706+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:29.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:29.706+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:29.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:29.706+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:29.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:29.708+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:30.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:30.301+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:30.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:30.301+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:30.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:30.301+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:30.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:30.301+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:30.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:30.426+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:30.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:30.426+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:30.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:30.426+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:30.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:30.427+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:31.063 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:16:30.960+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 6837 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:16:31.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:30.744+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:31.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:30.744+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:31.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:30.744+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:31.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:30.744+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:31.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:31.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:31.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:31.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:31.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:31.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:31.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:31.342+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:31.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:31.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:31.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:31.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:31.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:31.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:31.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:31.456+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:31.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:31.705+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:31.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:31.705+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:31.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:31.705+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:31.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:31.707+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:32.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:32.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:32.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:32.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:32.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:32.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:32.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:32.369+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:32.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:32.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:32.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:32.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:32.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:32.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:32.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:32.442+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:33.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:32.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:33.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:32.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:33.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:32.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:33.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:32.740+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:33.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:33.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:33.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:33.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:33.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:33.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:33.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:33.486+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:33.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:33.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:33.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:33.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:33.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:33.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:33.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:33.344+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:34.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:33.735+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:34.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:33.735+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:34.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:33.735+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:34.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:33.736+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:34.672 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:34.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:34.672 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:34.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:34.672 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:34.453+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:34.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:34.453+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:34.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:34.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:34.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:34.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:34.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:34.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:34.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:34.377+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:34.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:34.694+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:34.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:34.694+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:34.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:34.694+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:34.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:34.695+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:35.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:35.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:35.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:35.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:35.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:35.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:35.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:35.424+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:35.680 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:35.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:35.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:35.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:35.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:35.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:35.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:35.421+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:35.939 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:35.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:35.939 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:35.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:35.939 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:35.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:35.939 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:35.702+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:36.197 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:16:35.960+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 6931 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:16:36.662 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:36.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:36.662 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:36.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:36.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:36.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:36.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:36.379+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:36.663 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:36.410+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:36.663 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:36.410+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:36.663 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:36.410+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:36.663 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:36.412+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:36.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:36.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:36.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:36.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:36.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:36.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:36.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:36.684+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:37.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:37.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:37.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:37.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:37.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:37.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:37.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:37.446+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:37.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:37.411+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:37.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:37.411+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:37.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:37.411+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:37.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:37.412+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:37.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:37.689+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:37.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:37.689+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:37.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:37.689+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:37.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:37.690+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:38.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:38.439+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:38.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:38.439+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:38.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:38.439+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:38.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:38.440+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:38.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:38.477+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:38.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:38.477+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:38.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:38.477+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:38.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:38.478+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:38.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:38.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:38.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:38.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:38.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:38.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:38.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:38.708+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:39.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:39.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:39.672 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:39.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:39.672 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:39.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:39.672 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:39.475+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:39.672 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:39.434+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:39.672 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:39.434+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:39.672 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:39.434+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:39.672 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:39.434+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:39.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:39.692+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:39.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:39.692+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:39.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:39.692+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:39.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:39.693+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:40.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:40.496+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:40.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:40.496+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:40.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:40.496+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:40.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:40.497+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:40.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:40.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:40.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:40.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:40.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:40.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:40.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:40.446+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:41.063 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:16:40.961+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 7025 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:16:41.063 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:40.738+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:41.063 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:40.738+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:41.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:40.738+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:41.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:40.739+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:41.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:41.468+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.370321+0000 front 2024-04-27T02:06:15.370312+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:41.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:41.468+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.370341+0000 front 2024-04-27T02:06:15.370234+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:41.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:41.468+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.370259+0000 front 2024-04-27T02:06:15.370279+0000 (oldest deadline 2024-04-27T02:06:41.270170+0000) 2024-04-27T02:16:41.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:41.469+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:41.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:41.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:41.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:41.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:41.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:41.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:41.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:41.474+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:42.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:41.766+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:42.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:41.766+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:42.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:41.766+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:42.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:41.766+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:42.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:42.451+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:42.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:42.478+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:42.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:42.478+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:42.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:42.478+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:42.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:42.478+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:43.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:42.760+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:43.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:42.760+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:43.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:42.760+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:43.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:42.760+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:43.742 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:43.492+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:43.743 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:43.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:43.743 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:43.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:43.743 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:43.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:43.743 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:43.526+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:44.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:43.764+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:44.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:43.764+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:44.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:43.764+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:44.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:43.765+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:44.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:44.539+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:44.789 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:44.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:44.789 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:44.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:44.789 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:44.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:44.789 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:44.500+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:45.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:44.811+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:45.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:44.811+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:45.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:44.811+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:45.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:44.811+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:45.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:45.541+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:45.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:45.536+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:45.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:45.536+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:45.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:45.536+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:45.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:45.536+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:46.197 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:16:45.962+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 7119 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:16:46.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:45.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:46.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:45.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:46.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:45.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:46.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:45.851+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:46.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:16:46.624+0000 7f770c428700 0 monclient(hunting): authenticate timed out after 300 2024-04-27T02:16:46.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:46.580+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:46.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:46.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:46.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:46.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:46.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:46.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:46.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:46.500+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:46.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:46.840+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:46.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:46.840+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:46.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:46.840+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:46.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:46.840+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:47.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:47.586+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:47.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:47.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:47.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:47.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:47.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:47.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:47.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:47.457+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:48.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:47.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:48.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:47.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:48.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:47.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:48.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:47.888+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:48.847 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:48.565+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:48.847 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:48.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:48.847 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:48.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:48.847 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:48.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:48.847 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:48.494+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:49.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:48.869+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:49.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:48.869+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:49.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:48.869+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:49.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:48.870+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:49.868 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:49.547+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:49.869 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:49.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:49.869 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:49.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:49.869 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:49.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:49.869 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:49.538+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:50.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:49.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:50.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:49.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:50.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:49.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:50.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:49.892+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:50.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:50.548+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:50.849 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:50.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:50.849 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:50.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:50.849 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:50.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:50.849 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:50.526+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:51.197 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:16:50.964+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 7213 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:16:51.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:50.871+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:51.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:50.871+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:51.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:50.871+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:51.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:50.872+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:51.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:51.507+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:51.877 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:51.560+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:51.877 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:51.560+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:51.877 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:51.560+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:51.877 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:51.561+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:52.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:51.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:52.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:51.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:52.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:51.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:52.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:51.900+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:52.873 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:52.529+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:52.874 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:52.593+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:15.718298+0000 front 2024-04-27T02:06:15.718368+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:52.874 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:52.593+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:15.718283+0000 front 2024-04-27T02:06:15.718348+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:52.874 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:52.593+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:15.718387+0000 front 2024-04-27T02:06:15.718499+0000 (oldest deadline 2024-04-27T02:06:41.618373+0000) 2024-04-27T02:16:52.874 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:52.593+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:53.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:52.896+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:53.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:52.896+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:53.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:52.896+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:53.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:52.897+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:53.862 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:53.496+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:53.862 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:53.608+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:54.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:53.884+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 since back 2024-04-27T02:06:20.651756+0000 front 2024-04-27T02:06:20.651906+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:54.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:53.884+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 since back 2024-04-27T02:06:20.651775+0000 front 2024-04-27T02:06:20.651865+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:54.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:53.884+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 since back 2024-04-27T02:06:20.651840+0000 front 2024-04-27T02:06:20.651850+0000 (oldest deadline 2024-04-27T02:06:42.951619+0000) 2024-04-27T02:16:54.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:53.885+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:54.861 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:54.643+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:54.862 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:54.523+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:55.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:54.884+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:55.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:55.487+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:55.889 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:55.602+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:56.197 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:16:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:16:55.965+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 7307 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:16:56.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:55.911+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:56.852 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:56.507+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:56.852 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:56.633+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:57.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:56.874+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:57.871 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:57.498+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:57.871 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:57.657+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:58.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:57.893+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:58.850 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:58.647+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:58.851 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:58.517+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:59.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:58.872+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:59.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:16:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:16:59.508+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:16:59.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:16:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:16:59.629+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:00.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:16:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:16:59.900+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:00.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:00.505+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:00.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:00.650+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:01.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:00.906+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:01.197 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:17:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:17:00.966+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 7402 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:17:01.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:01.457+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:01.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:01.634+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:02.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:01.898+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:02.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:02.677+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:02.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:02.441+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:03.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:02.894+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:03.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:03.438+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:03.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:03.651+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:04.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:03.918+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:04.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:04.419+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:04.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:04.871+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:04.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:04.689+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:05.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:05.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:05.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:05.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:05.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:05.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:05.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:05.449+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:05.947 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:17:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:17:05.967+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 7497 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:17:05.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:05.886+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:05.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:05.718+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:06.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:06.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:06.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:06.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:06.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:06.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:06.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:06.447+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:07.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:06.901+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:07.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:06.732+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:07.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:07.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:07.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:07.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:07.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:07.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:07.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:07.450+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:08.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:07.947+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:08.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:07.757+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:08.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:08.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:08.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:08.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:08.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:08.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:08.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:08.425+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:09.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:08.950+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:09.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:08.741+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:09.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:09.392+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:09.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:09.392+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:09.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:09.392+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:09.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:09.392+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:09.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:09.695+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:10.330 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:09.975+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:10.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:10.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:10.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:10.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:10.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:10.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:10.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:10.353+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:10.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:10.653+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:11.197 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:17:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:17:10.968+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 7592 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:17:11.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:10.987+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:11.647 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:11.380+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:11.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:11.380+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:11.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:11.380+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:11.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:11.381+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:11.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:11.670+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:12.329 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:11.988+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:12.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:12.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:12.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:12.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:12.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:12.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:12.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:12.353+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:12.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:12.690+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:13.372 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:13.008+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:13.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:13.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:13.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:13.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:13.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:13.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:13.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:13.396+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:13.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:13.702+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:14.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:13.960+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:14.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:14.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:14.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:14.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:14.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:14.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:14.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:14.359+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:14.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:14.701+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:14.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:14.951+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:15.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:15.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:15.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:15.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:15.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:15.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:15.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:15.393+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:15.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:15.717+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:16.197 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:17:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:17:15.969+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 7687 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:17:16.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:15.975+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:16.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:16.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:16.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:16.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:16.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:16.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:16.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:16.441+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:17.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:16.963+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:17.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:16.746+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:17.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:16.746+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:17.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:16.746+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:17.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:16.746+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:17.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:17.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:17.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:17.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:17.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:17.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:17.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:17.449+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:18.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:17.791+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:18.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:17.791+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:18.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:17.791+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:18.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:17.792+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:18.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:17.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:18.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:17.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:18.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:17.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:18.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:17.982+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:18.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:18.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:18.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:18.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:18.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:18.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:18.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:18.441+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:19.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:18.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:19.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:18.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:19.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:18.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:19.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:18.822+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:19.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:19.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:19.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:19.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:19.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:19.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:19.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:19.013+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:19.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:19.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:19.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:19.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:19.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:19.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:19.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:19.491+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:20.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:19.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:20.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:19.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:20.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:19.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:20.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:19.844+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:20.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:20.057+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:20.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:20.057+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:20.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:20.057+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:20.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:20.057+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:20.826 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:20.518+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:20.826 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:20.518+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:20.826 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:20.518+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:20.826 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:20.519+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:21.197 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:17:20.970+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 7782 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:17:21.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:20.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:21.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:20.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:21.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:20.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:21.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:20.849+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:21.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:21.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:21.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:21.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:21.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:21.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:21.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:21.065+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:21.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:21.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:21.797 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:21.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:21.797 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:21.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:21.797 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:21.524+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:22.091 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:21.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:22.091 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:21.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:22.091 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:21.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:22.091 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:21.819+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:22.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:22.114+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:22.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:22.114+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:22.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:22.114+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:22.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:22.115+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:22.762 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:22.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:22.762 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:22.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:22.762 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:22.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:22.762 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:22.522+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:23.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:22.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:23.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:22.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:23.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:22.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:23.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:22.785+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:23.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:23.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:23.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:23.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:23.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:23.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:23.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:23.162+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:23.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:23.541+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:23.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:23.541+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:23.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:23.541+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:23.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:23.542+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:24.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:23.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:24.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:23.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:24.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:23.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:24.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:23.813+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:24.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:24.171+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:24.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:24.171+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:24.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:24.171+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:24.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:24.172+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:24.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:24.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:24.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:24.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:24.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:24.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:24.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:24.559+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:24.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:24.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:24.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:24.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:24.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:24.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:24.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:24.794+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:25.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:25.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:25.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:25.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:25.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:25.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:25.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:25.146+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:25.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:25.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:25.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:25.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:25.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:25.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:25.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:25.555+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:26.074 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:17:25.971+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 7877 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:17:26.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:25.837+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:26.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:25.837+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:26.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:25.837+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:26.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:25.837+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:26.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:26.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:26.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:26.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:26.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:26.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:26.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:26.171+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:26.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:26.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:26.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:26.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:26.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:26.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:26.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:26.554+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:27.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:26.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:27.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:26.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:27.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:26.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:27.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:26.885+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:27.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:27.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:27.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:27.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:27.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:27.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:27.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:27.191+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:27.851 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:27.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:27.851 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:27.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:27.851 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:27.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:27.851 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:27.568+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:28.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:27.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:28.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:27.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:28.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:27.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:28.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:27.874+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:28.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:28.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:28.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:28.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:28.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:28.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:28.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:28.191+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:28.832 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:28.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:28.832 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:28.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:28.832 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:28.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:28.832 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:28.535+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:29.155 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:28.854+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:29.155 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:28.854+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:29.155 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:28.854+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:29.155 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:28.855+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:29.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:29.178+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:29.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:29.178+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:29.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:29.178+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:29.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:29.179+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:29.785 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:29.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:29.785 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:29.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:29.785 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:29.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:29.786 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:29.538+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:30.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:29.808+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:30.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:29.808+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:30.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:29.808+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:30.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:29.808+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:30.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:30.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:30.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:30.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:30.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:30.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:30.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:30.134+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:30.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:30.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:30.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:30.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:30.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:30.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:30.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:30.559+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:30.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:30.808+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:30.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:30.808+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:30.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:30.808+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:30.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:30.808+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:31.447 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:17:30.972+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 7972 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:17:31.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:31.140+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:31.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:31.140+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:31.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:31.140+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:31.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:31.141+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:31.787 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:31.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:31.788 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:31.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:31.788 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:31.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:31.788 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:31.541+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:32.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:31.810+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:32.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:31.810+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:32.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:31.810+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:32.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:31.811+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:32.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:32.188+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:32.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:32.188+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:32.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:32.188+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:32.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:32.189+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:32.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:32.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:32.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:32.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:32.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:32.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:32.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:32.538+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:33.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:32.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:33.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:32.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:33.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:32.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:33.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:32.824+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:33.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:33.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:33.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:33.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:33.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:33.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:33.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:33.166+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:33.810 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:33.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:33.810 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:33.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:33.810 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:33.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:33.810 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:33.524+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:34.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:33.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:34.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:33.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:34.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:33.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:34.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:33.832+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:34.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:34.137+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:34.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:34.137+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:34.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:34.137+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:34.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:34.138+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:34.818 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:34.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:34.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:34.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:34.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:34.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:34.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:34.524+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:35.118 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:34.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:35.119 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:34.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:35.119 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:34.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:35.119 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:34.841+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:35.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:35.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:35.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:35.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:35.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:35.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:35.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:35.146+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:35.868 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:35.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:35.869 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:35.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:35.869 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:35.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:35.869 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:35.521+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:36.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:17:35.972+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 8067 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:17:36.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:35.890+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:36.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:35.890+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:36.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:35.890+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:36.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:35.890+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:36.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:36.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:36.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:36.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:36.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:36.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:36.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:36.191+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:36.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:36.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:36.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:36.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:36.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:36.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:36.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:36.536+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:37.188 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:36.933+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:37.188 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:36.933+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:37.188 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:36.933+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:37.188 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:36.933+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:37.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:37.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:37.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:37.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:37.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:37.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:37.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:37.210+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:37.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:37.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:37.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:37.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:37.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:37.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:37.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:37.568+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:38.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:37.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:38.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:37.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:38.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:37.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:38.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:37.953+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:38.577 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:38.228+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:38.577 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:38.228+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:38.577 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:38.228+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:38.577 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:38.229+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:38.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:38.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:38.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:38.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:38.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:38.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:38.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:38.599+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:39.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:39.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:39.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:39.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:39.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:39.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:39.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:39.191+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:39.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:38.970+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:39.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:38.970+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:39.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:38.970+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:39.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:38.970+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:39.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:39.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:39.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:39.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:39.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:39.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:39.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:39.643+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:40.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:40.166+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:40.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:40.166+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:40.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:40.166+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:40.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:40.167+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:40.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:39.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:40.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:39.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:40.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:39.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:40.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:39.955+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:40.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:40.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:40.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:40.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:40.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:40.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:40.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:40.654+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:41.197 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:17:40.973+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 8162 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:17:41.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:40.967+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:41.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:40.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:41.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:40.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:41.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:40.968+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:41.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:41.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:41.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:41.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:41.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:41.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:41.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:41.147+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:41.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:41.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:41.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:41.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:41.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:41.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:41.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:41.685+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:42.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:41.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:42.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:41.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:42.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:41.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:42.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:41.973+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:42.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:42.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:42.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:42.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:42.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:42.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:42.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:42.121+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:42.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:42.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:42.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:42.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:42.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:42.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:42.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:42.641+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:43.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:43.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:43.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:43.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:43.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:43.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:43.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:43.013+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:43.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:43.162+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:43.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:43.162+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:43.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:43.162+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:43.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:43.163+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:43.782 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:43.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:43.782 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:43.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:43.782 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:43.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:43.783 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:43.651+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:44.094 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:44.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:44.095 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:44.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:44.095 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:44.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:44.095 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:44.031+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:44.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:44.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:44.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:44.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:44.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:44.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:44.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:44.117+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:44.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:44.684+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:44.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:44.684+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:44.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:44.684+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:44.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:44.685+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:45.215 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:45.144+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:45.215 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:45.144+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:45.215 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:45.144+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:45.215 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:45.145+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:45.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:44.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:45.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:44.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:45.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:44.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:45.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:44.985+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:45.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:45.718+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:45.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:45.718+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:45.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:45.718+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:45.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:45.718+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:45.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:45.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:45.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:45.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:45.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:45.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:45.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:45.943+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:46.344 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:17:45.975+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 8256 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:17:46.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:46.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:46.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:46.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:46.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:46.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:46.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:46.165+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:46.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:46.700+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:46.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:46.701+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:46.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:46.701+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:46.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:46.701+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:47.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:46.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:47.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:46.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:47.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:46.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:47.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:46.974+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:47.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:47.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:47.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:47.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:47.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:47.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:47.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:47.160+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:47.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:47.656+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:47.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:47.656+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:47.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:47.656+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:47.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:47.657+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:48.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:48.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:48.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:48.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:48.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:48.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:48.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:48.205+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:48.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:47.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:48.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:47.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:48.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:47.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:48.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:47.963+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:48.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:48.665+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:48.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:48.665+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:48.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:48.665+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:48.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:48.665+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:49.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:48.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:49.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:48.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:49.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:48.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:49.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:48.934+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:49.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:49.245+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:49.690 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:49.245+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:49.690 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:49.245+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:49.690 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:49.246+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:49.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:49.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:49.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:49.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:49.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:49.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:49.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:49.885+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:49.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:49.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:49.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:49.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:49.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:49.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:49.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:49.713+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:50.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:50.238+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:50.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:50.238+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:50.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:50.238+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:50.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:50.239+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:50.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:17:50.977+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 8350 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:17:50.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:50.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:50.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:50.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:50.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:50.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:50.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:50.740+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:50.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:50.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:50.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:50.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:50.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:50.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:50.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:50.905+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:51.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:51.273+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:51.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:51.273+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:51.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:51.273+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:51.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:51.273+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:52.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:51.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:52.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:51.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:52.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:51.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:52.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:51.764+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:52.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:51.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:52.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:51.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:52.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:51.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:52.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:51.880+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:52.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:52.243+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:52.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:52.243+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:52.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:52.243+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:52.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:52.243+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:52.999 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:52.730+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:52.999 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:52.730+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:52.999 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:52.730+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:52.999 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:52.731+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:52.999 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:52.858+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:53.000 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:52.858+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:53.000 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:52.858+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:53.000 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:52.858+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:53.533 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:53.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:53.533 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:53.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:53.533 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:53.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:53.533 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:53.278+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:53.864 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:53.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:53.864 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:53.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:53.864 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:53.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:53.865 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:53.687+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:54.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:53.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:54.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:53.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:54.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:53.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:54.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:53.887+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:54.630 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:54.246+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:54.630 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:54.246+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:54.630 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:54.246+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:54.630 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:54.247+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:54.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:54.838+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:54.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:54.838+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:54.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:54.838+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:54.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:54.838+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:54.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:54.652+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:54.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:54.652+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:54.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:54.652+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:54.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:54.653+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:55.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:55.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:55.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:55.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:55.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:55.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:55.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:55.243+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:55.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:55.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:55.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:55.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:55.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:55.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:55.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:55.870+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:55.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:55.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:55.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:55.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:55.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:55.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:55.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:55.693+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:56.234 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:17:55.978+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 8446 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:17:56.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:56.257+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:56.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:56.257+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:56.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:56.257+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:56.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:56.258+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:56.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:56.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:56.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:56.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:56.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:56.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:56.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:56.692+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:56.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:56.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:56.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:56.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:56.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:56.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:56.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:56.832+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:57.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:57.286+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:57.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:57.286+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:57.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:57.286+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:57.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:57.287+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:57.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:57.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:57.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:57.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:57.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:57.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:57.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:57.681+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:57.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:57.879+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:57.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:57.879+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:57.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:57.879+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:57.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:57.880+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:58.649 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:58.246+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:58.650 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:58.246+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:58.650 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:58.246+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:58.650 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:58.247+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:58.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:58.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:58.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:58.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:58.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:58.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:58.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:58.673+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:58.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:58.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:58.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:58.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:58.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:58.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:58.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:58.900+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:59.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:59.240+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:59.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:59.240+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:59.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:59.240+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:17:59.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:17:59.241+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:59.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:59.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:59.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:59.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:59.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:59.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:17:59.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:17:59.716+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:17:59.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:59.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:59.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:59.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:59.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:59.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:17:59.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:17:59.866+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:00.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:00.290+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:00.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:00.290+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:00.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:00.290+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:00.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:00.291+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:00.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:00.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:00.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:00.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:00.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:00.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:00.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:00.819+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:00.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:00.700+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:00.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:00.700+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:00.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:00.700+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:00.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:00.700+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:01.233 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:18:00.979+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 8540 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:18:01.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:01.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:01.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:01.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:01.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:01.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:01.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:01.257+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:02.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:01.749+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:02.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:01.749+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:02.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:01.749+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:02.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:01.750+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:02.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:01.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:02.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:01.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:02.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:01.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:02.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:01.812+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:02.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:02.294+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:02.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:02.294+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:02.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:02.294+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:02.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:02.295+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:03.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:02.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:03.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:02.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:03.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:02.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:03.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:02.727+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:03.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:02.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:03.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:02.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:03.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:02.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:03.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:02.793+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:03.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:03.307+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:03.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:03.307+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:03.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:03.307+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:03.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:03.308+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:04.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:03.765+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:04.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:03.765+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:04.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:03.765+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:04.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:03.765+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:04.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:03.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:04.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:03.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:04.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:03.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:04.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:03.755+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:04.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:04.268+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:04.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:04.268+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:04.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:04.268+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:04.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:04.269+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:04.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:04.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:04.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:04.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:04.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:04.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:04.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:04.803+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:04.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:04.708+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:04.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:04.708+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:04.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:04.708+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:04.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:04.709+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:05.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:05.292+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:05.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:05.292+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:05.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:05.292+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:05.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:05.293+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:05.968 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:18:05.980+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 8634 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:18:05.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:05.739+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:05.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:05.739+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:05.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:05.739+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:05.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:05.739+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:05.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:05.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:05.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:05.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:05.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:05.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:05.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:05.798+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:06.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:06.266+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:06.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:06.266+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:06.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:06.266+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:06.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:06.267+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:07.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:06.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:07.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:06.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:07.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:06.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:07.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:06.777+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:07.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:06.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:07.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:06.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:07.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:06.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:07.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:06.761+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:07.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:07.228+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:07.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:07.228+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:07.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:07.228+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:07.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:07.229+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:08.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:07.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:08.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:07.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:08.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:07.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:08.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:07.794+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:08.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:07.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:08.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:07.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:08.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:07.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:08.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:07.793+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:08.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:08.235+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:08.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:08.235+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:08.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:08.235+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:08.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:08.236+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:09.178 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:08.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:09.179 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:08.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:09.179 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:08.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:09.179 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:08.828+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:09.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:08.789+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:09.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:08.789+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:09.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:08.789+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:09.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:08.789+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:09.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:09.201+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:09.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:09.201+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:09.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:09.201+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:09.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:09.202+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:10.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:09.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:10.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:09.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:10.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:09.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:10.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:09.867+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:10.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:09.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:10.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:09.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:10.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:09.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:10.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:09.788+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:10.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:10.173+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:10.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:10.173+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:10.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:10.173+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:10.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:10.174+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:10.980 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:18:10.981+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 8728 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:18:10.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:10.857+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:10.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:10.857+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:10.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:10.857+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:10.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:10.858+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:10.981 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:10.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:10.981 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:10.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:10.981 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:10.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:10.981 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:10.742+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:11.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:11.207+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:11.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:11.207+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:11.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:11.207+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:11.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:11.208+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:12.174 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:11.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:12.174 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:11.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:12.174 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:11.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:12.174 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:11.875+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:12.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:11.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:12.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:11.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:12.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:11.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:12.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:11.725+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:12.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:12.195+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:12.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:12.195+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:12.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:12.195+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:12.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:12.196+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:13.163 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:12.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:13.164 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:12.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:13.164 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:12.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:13.164 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:12.870+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:13.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:12.767+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:13.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:12.767+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:13.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:12.767+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:13.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:12.767+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:13.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:13.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:13.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:13.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:13.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:13.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:13.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:13.187+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:13.810 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:13.786+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:13.810 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:13.786+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:13.810 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:13.786+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:13.811 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:13.786+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:14.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:13.832+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:14.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:13.832+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:14.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:13.832+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:14.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:13.833+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:14.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:14.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:14.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:14.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:14.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:14.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:14.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:14.154+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:14.839 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:14.769+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:14.839 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:14.769+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:14.839 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:14.769+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:14.839 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:14.769+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:15.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:14.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:15.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:14.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:15.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:14.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:15.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:14.863+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:15.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:15.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:15.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:15.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:15.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:15.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:15.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:15.136+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:15.792 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:15.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:15.792 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:15.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:15.792 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:15.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:15.793 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:15.734+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:16.084 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:18:15.983+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 8822 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:18:16.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:15.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:16.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:15.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:16.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:15.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:16.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:15.816+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:16.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:16.152+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:16.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:16.152+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:16.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:16.152+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:16.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:16.153+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:16.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:16.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:16.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:16.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:16.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:16.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:16.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:16.811+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:16.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:16.686+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:16.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:16.686+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:16.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:16.686+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:16.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:16.687+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:17.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:17.177+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:17.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:17.177+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:17.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:17.177+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:17.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:17.178+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:17.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:17.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:17.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:17.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:17.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:17.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:17.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:17.851+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:17.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:17.717+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:17.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:17.717+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:17.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:17.717+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:17.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:17.718+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:18.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:18.137+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:18.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:18.137+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:18.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:18.137+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:18.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:18.138+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:18.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:18.863+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:18.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:18.863+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:18.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:18.863+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:18.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:18.863+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:18.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:18.678+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:18.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:18.678+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:18.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:18.678+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:18.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:18.678+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:19.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:19.168+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:19.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:19.168+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:19.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:19.168+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:19.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:19.169+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:19.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:19.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:19.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:19.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:19.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:19.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:19.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:19.822+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:19.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:19.716+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:19.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:19.716+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:19.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:19.716+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:19.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:19.716+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:20.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:20.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:20.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:20.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:20.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:20.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:20.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:20.138+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:20.816 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:20.759+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:20.817 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:20.759+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:20.817 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:20.759+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:20.817 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:20.760+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:21.067 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:18:20.984+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 8916 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:18:21.068 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:20.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:21.068 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:20.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:21.068 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:20.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:21.068 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:20.840+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:21.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:21.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:21.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:21.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:21.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:21.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:21.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:21.091+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:22.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:21.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:22.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:21.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:22.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:21.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:22.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:21.811+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:22.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:21.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:22.061 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:21.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:22.061 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:21.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:22.061 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:21.778+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:22.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:22.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:22.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:22.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:22.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:22.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:22.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:22.083+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:23.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:22.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:23.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:22.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:23.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:22.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:23.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:22.784+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:23.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:22.828+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:23.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:22.828+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:23.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:22.828+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:23.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:22.829+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:23.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:23.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:23.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:23.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:23.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:23.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:23.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:23.094+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:23.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:23.778+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:23.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:23.778+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:23.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:23.778+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:23.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:23.778+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:23.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:23.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:23.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:23.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:23.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:23.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:23.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:23.806+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:24.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:24.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:24.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:24.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:24.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:24.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:24.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:24.056+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:25.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:24.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:25.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:24.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:25.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:24.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:25.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:24.782+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:25.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:24.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:25.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:24.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:25.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:24.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:25.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:24.824+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:25.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:25.099+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:25.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:25.099+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:25.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:25.099+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:25.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:25.100+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:25.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:25.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:25.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:25.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:25.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:25.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:25.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:25.755+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:25.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:25.858+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:25.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:25.859+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:25.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:25.859+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:25.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:25.859+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:26.344 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:18:25.985+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9009 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:18:26.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:26.110+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:26.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:26.110+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:26.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:26.110+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:26.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:26.111+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:27.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:26.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:27.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:26.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:27.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:26.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:27.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:26.749+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:27.136 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:26.825+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:27.136 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:26.825+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:27.136 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:26.825+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:27.136 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:26.825+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:27.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:27.158+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:27.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:27.158+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:27.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:27.158+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:27.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:27.159+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:28.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:27.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:28.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:27.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:28.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:27.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:28.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:27.862+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:28.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:27.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:28.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:27.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:28.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:27.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:28.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:27.727+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:28.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:28.168+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:28.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:28.168+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:28.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:28.168+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:28.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:28.169+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:29.181 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:28.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:29.181 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:28.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:29.181 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:28.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:29.181 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:28.849+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:29.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:28.774+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:29.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:28.774+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:29.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:28.774+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:29.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:28.775+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:29.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:29.203+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:29.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:29.203+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:29.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:29.203+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:29.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:29.205+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:30.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:29.825+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:30.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:29.825+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:30.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:29.825+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:30.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:29.826+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:30.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:29.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:30.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:29.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:30.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:29.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:30.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:29.735+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:30.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:30.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:30.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:30.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:30.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:30.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:30.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:30.166+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:30.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:30.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:30.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:30.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:30.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:30.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:30.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:30.834+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:30.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:30.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:30.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:30.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:30.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:30.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:30.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:30.741+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:31.447 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:18:30.986+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9103 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:18:31.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:31.174+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:31.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:31.174+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:31.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:31.174+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:31.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:31.175+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:32.155 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:31.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:32.155 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:31.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:32.155 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:31.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:32.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:31.844+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:32.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:31.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:32.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:31.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:32.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:31.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:32.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:31.762+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:32.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:32.178+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:32.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:32.178+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:32.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:32.178+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:32.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:32.179+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:33.184 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:32.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:33.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:32.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:33.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:32.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:33.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:32.738+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:33.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:32.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:33.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:32.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:33.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:32.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:33.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:32.829+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:33.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:33.207+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:33.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:33.207+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:33.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:33.207+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:33.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:33.208+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:33.827 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:33.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:33.827 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:33.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:33.827 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:33.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:33.827 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:33.787+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:34.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:33.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:34.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:33.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:34.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:33.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:34.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:33.850+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:34.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:34.241+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:34.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:34.241+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:34.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:34.241+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:34.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:34.241+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:35.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:34.803+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:35.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:34.803+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:35.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:34.803+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:35.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:34.803+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:35.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:34.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:35.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:34.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:35.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:34.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:35.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:34.799+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:35.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:35.283+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:35.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:35.283+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:35.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:35.283+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:35.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:35.284+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:35.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:35.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:35.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:35.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:35.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:35.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:35.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:35.760+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:35.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:35.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:35.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:35.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:35.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:35.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:35.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:35.826+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:36.236 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:18:35.986+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9196 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:18:36.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:36.258+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:36.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:36.258+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:36.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:36.258+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:36.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:36.259+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:37.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:36.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:37.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:36.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:37.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:36.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:37.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:36.751+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:37.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:36.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:37.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:36.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:37.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:36.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:37.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:36.871+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:37.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:37.223+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:37.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:37.223+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:37.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:37.223+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:37.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:37.224+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:37.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:37.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:37.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:37.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:37.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:37.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:37.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:37.708+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:37.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:37.920+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:37.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:37.920+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:37.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:37.920+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:37.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:37.920+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:38.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:38.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:38.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:38.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:38.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:38.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:38.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:38.253+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:38.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:38.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:38.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:38.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:38.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:38.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:38.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:38.887+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:38.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:38.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:38.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:38.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:38.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:38.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:38.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:38.676+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:39.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:39.235+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:39.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:39.235+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:39.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:39.235+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:39.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:39.237+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:39.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:39.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:39.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:39.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:39.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:39.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:39.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:39.910+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:39.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:39.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:39.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:39.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:39.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:39.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:39.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:39.670+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:40.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:40.254+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:40.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:40.254+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:40.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:40.254+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:40.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:40.254+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:40.886 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:40.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:40.886 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:40.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:40.886 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:40.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:40.886 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:40.632+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:41.197 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:18:40.988+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9290 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:18:41.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:40.907+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:41.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:40.907+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:41.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:40.907+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:41.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:40.908+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:41.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:41.248+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:41.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:41.248+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:41.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:41.248+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:41.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:41.249+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:41.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:41.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:41.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:41.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:41.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:41.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:41.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:41.638+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:41.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:41.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:41.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:41.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:41.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:41.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:41.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:41.863+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:42.637 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:42.296+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:42.637 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:42.296+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:42.637 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:42.296+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:42.637 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:42.297+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:42.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:42.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:42.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:42.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:42.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:42.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:42.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:42.895+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:42.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:42.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:42.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:42.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:42.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:42.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:42.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:42.660+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:43.535 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:43.286+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:43.535 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:43.286+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:43.535 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:43.286+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:43.535 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:43.287+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:43.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:43.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:43.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:43.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:43.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:43.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:43.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:43.658+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:44.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:43.904+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:44.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:43.904+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:44.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:43.904+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:44.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:43.904+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:44.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:44.259+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:44.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:44.259+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:44.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:44.259+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:44.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:44.260+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:44.918 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:44.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:44.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:44.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:44.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:44.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:44.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:44.633+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:45.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:44.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:45.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:44.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:45.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:44.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:45.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:44.941+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:45.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:45.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:45.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:45.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:45.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:45.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:45.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:45.211+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:45.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:45.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:45.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:45.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:45.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:45.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:45.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:45.661+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:46.183 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:18:45.989+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9380 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:18:46.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:45.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:46.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:45.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:46.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:45.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:46.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:45.916+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:46.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:46.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:46.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:46.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:46.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:46.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:46.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:46.206+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:46.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:46.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:46.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:46.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:46.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:46.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:46.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:46.623+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:47.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:46.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:47.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:46.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:47.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:46.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:47.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:46.947+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:47.634 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:47.245+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:47.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:47.245+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:47.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:47.245+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:47.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:47.246+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:47.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:47.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:47.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:47.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:47.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:47.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:47.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:47.901+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:47.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:47.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:47.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:47.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:47.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:47.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:47.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:47.658+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:48.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:48.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:48.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:48.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:48.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:48.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:48.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:48.253+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:48.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:48.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:48.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:48.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:48.908 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:48.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:48.908 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:48.665+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:49.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:48.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:49.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:48.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:49.184 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:48.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:49.184 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:48.931+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:49.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:49.206+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:49.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:49.206+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:49.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:49.206+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:49.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:49.207+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:49.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:49.907+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:49.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:49.907+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:49.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:49.907+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:49.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:49.908+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:49.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:49.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:49.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:49.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:49.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:49.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:49.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:49.702+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:50.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:50.167+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:50.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:50.167+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:50.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:50.167+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:50.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:50.168+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:50.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:50.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:50.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:50.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:50.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:50.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:50.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:50.895+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:50.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:50.736+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:50.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:50.736+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:50.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:50.736+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:50.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:50.736+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:51.367 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:18:50.991+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9471 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:18:51.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:51.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:51.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:51.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:51.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:51.177+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:51.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:51.178+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:52.125 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:51.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:52.126 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:51.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:52.126 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:51.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:52.126 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:51.723+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:52.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:51.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:52.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:51.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:52.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:51.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:52.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:51.869+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:52.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:52.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:52.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:52.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:52.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:52.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:52.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:52.149+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:52.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:52.718+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:52.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:52.718+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:52.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:52.718+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:52.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:52.719+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:52.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:52.906+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:52.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:52.906+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:52.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:52.906+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:52.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:52.907+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:53.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:53.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:53.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:53.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:53.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:53.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:53.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:53.195+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:53.918 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:53.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:53.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:53.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:53.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:53.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:53.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:53.695+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:54.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:54.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:54.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:54.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:54.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:54.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:54.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:54.186+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:54.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:53.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:54.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:53.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:54.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:53.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:54.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:53.941+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:55.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:54.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:55.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:54.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:55.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:54.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:55.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:54.720+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:55.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:54.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:55.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:54.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:55.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:54.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:55.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:54.944+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:55.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:55.182+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:55.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:55.182+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:55.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:55.182+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:55.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:55.183+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:55.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:55.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:55.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:55.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:55.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:55.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:55.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:55.732+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:56.344 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:18:55.992+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9555 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:18:56.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:56.149+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:56.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:56.149+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:56.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:56.149+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:56.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:56.150+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:56.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:55.989+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:56.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:55.989+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:56.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:55.989+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:56.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:55.990+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:57.109 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:56.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:57.109 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:56.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:57.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:56.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:57.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:56.761+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:57.110 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:56.972+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:57.110 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:56.972+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:57.110 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:56.972+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:57.110 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:56.972+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:57.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:57.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:57.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:57.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:57.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:57.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:57.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:57.133+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:57.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:57.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:57.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:57.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:57.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:57.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:57.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:57.748+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:58.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:57.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:58.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:57.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:58.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:57.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:58.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:57.991+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:58.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:58.155+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:58.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:58.155+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:58.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:58.155+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:58.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:58.156+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:58.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:58.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:58.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:58.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:58.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:58.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:58.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:58.716+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:59.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:58.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:59.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:58.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:59.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:58.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:59.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:58.995+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:59.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:59.152+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:59.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:59.152+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:59.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:59.152+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:18:59.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:18:59.153+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:59.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:59.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:59.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:59.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:59.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:59.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:18:59.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:18:59.719+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:18:59.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:59.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:59.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:59.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:59.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:59.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:18:59.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:18:59.965+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:00.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:00.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:00.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:00.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:00.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:00.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:00.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:00.194+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:00.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:00.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:00.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:00.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:00.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:00.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:00.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:00.713+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:01.197 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:19:00.993+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9639 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:19:01.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:01.205+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:01.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:01.205+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:01.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:01.205+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:01.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:01.206+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:01.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:00.967+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:01.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:00.967+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:01.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:00.967+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:01.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:00.967+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:01.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:01.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:01.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:01.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:01.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:01.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:01.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:01.674+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:02.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:01.938+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:02.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:01.938+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:02.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:01.938+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:02.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:01.938+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:02.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:02.195+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:02.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:02.195+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:02.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:02.195+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:02.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:02.196+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:02.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:02.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:02.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:02.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:02.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:02.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:02.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:02.634+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:03.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:02.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:03.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:02.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:03.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:02.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:03.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:02.910+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:03.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:03.230+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:03.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:03.230+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:03.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:03.230+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:03.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:03.230+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:03.885 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:03.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:03.886 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:03.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:03.886 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:03.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:03.886 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:03.593+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:04.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:03.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:04.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:03.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:04.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:03.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:04.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:03.909+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:04.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:04.214+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:04.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:04.214+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:04.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:04.214+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:04.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:04.215+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:04.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:04.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:04.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:04.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:04.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:04.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:04.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:04.551+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:05.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:04.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:05.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:04.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:05.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:04.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:05.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:04.872+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:05.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:05.202+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:05.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:05.202+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:05.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:05.202+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:05.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:05.202+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:05.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:05.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:05.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:05.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:05.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:05.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:05.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:05.575+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:05.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:05.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:05.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:05.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:05.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:05.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:05.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:05.823+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:06.343 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:19:05.994+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9726 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:19:06.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:06.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:06.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:06.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:06.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:06.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:06.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:06.162+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:06.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:06.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:06.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:06.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:06.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:06.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:06.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:06.560+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:07.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:06.854+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:07.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:06.854+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:07.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:06.854+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:07.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:06.854+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:07.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:07.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:07.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:07.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:07.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:07.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:07.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:07.147+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:07.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:07.608+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:07.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:07.608+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:07.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:07.608+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:07.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:07.608+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:08.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:07.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:08.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:07.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:08.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:07.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:08.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:07.902+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:08.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:08.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:08.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:08.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:08.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:08.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:08.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:08.176+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:08.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:08.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:08.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:08.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:08.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:08.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:08.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:08.639+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:09.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:08.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:09.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:08.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:09.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:08.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:09.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:08.901+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:09.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:09.139+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:09.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:09.139+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:09.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:09.139+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:09.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:09.139+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:09.864 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:09.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:09.865 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:09.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:09.865 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:09.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:09.865 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:09.631+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:10.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:10.107+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:10.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:10.107+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:10.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:10.107+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:10.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:10.108+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:10.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:09.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:10.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:09.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:10.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:09.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:10.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:09.888+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:10.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:10.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:10.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:10.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:10.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:10.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:10.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:10.658+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:11.197 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:19:10.996+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9814 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:19:11.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:11.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:11.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:11.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:11.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:11.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:11.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:11.109+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:11.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:10.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:11.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:10.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:11.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:10.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:11.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:10.912+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:11.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:11.690+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:11.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:11.690+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:11.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:11.690+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:11.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:11.690+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:11.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:11.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:11.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:11.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:11.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:11.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:11.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:11.884+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:12.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:12.074+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:12.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:12.074+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:12.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:12.074+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:12.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:12.075+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:13.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:12.837+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:13.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:12.837+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:13.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:12.837+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:13.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:12.837+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:13.055 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:12.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:13.055 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:12.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:13.055 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:12.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:13.055 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:12.730+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:13.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:13.078+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:13.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:13.078+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:13.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:13.078+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:13.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:13.079+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:13.792 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:13.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:13.792 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:13.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:13.792 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:13.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:13.793 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:13.761+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:14.078 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:13.815+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:14.079 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:13.815+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:14.079 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:13.815+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:14.079 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:13.816+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:14.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:14.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:14.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:14.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:14.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:14.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:14.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:14.103+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:15.037 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:14.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:15.037 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:14.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:15.037 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:14.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:15.037 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:14.778+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:15.037 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:14.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:15.037 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:14.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:15.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:14.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:15.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:14.736+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:15.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:15.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:15.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:15.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:15.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:15.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:15.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:15.061+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:15.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:15.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:15.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:15.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:15.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:15.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:15.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:15.744+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:15.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:15.781+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:15.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:15.781+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:15.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:15.781+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:15.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:15.782+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:16.344 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:19:15.997+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9903 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:19:16.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:16.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:16.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:16.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:16.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:16.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:16.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:16.051+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:17.027 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:16.757+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:17.027 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:16.758+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:17.027 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:16.758+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:17.027 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:16.758+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:17.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:16.789+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:17.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:16.789+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:17.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:16.789+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:17.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:16.790+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:17.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:17.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:17.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:17.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:17.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:17.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:17.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:17.050+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:18.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:18.006+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:18.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:18.006+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:18.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:18.006+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:18.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:18.007+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:18.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:17.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:18.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:17.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:18.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:17.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:18.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:17.835+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:18.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:17.763+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:18.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:17.763+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:18.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:17.763+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:18.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:17.764+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:18.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:18.857+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:18.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:18.857+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:18.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:18.857+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:18.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:18.858+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:18.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:18.726+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:18.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:18.726+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:18.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:18.726+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:18.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:18.727+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:19.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:18.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:19.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:18.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:19.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:18.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:19.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:18.982+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:19.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:19.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:19.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:19.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:19.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:19.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:19.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:19.831+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:19.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:19.713+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:19.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:19.713+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:19.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:19.713+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:19.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:19.714+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:20.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:20.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:20.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:20.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:20.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:20.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:20.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:20.009+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:20.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:20.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:20.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:20.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:20.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:20.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:20.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:20.787+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:20.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:20.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:20.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:20.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:20.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:20.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:20.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:20.713+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:21.447 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:19:20.998+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 9997 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:19:21.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:21.042+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:21.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:21.042+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:21.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:21.042+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:21.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:21.044+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:22.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:21.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:22.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:21.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:22.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:21.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:22.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:21.757+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:22.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:21.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:22.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:21.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:22.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:21.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:22.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:21.830+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:22.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:22.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:22.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:22.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:22.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:22.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:22.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:22.082+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:22.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:22.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:22.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:22.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:22.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:22.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:22.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:22.712+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:22.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:22.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:22.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:22.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:22.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:22.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:22.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:22.807+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:23.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:23.069+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:23.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:23.069+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:23.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:23.069+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:23.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:23.070+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:23.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:23.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:23.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:23.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:23.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:23.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:23.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:23.795+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:23.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:23.682+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:23.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:23.682+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:23.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:23.682+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:23.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:23.683+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:24.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:24.107+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:24.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:24.107+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:24.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:24.107+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:24.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:24.108+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:24.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:24.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:24.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:24.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:24.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:24.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:24.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:24.787+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:24.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:24.690+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:24.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:24.690+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:24.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:24.690+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:24.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:24.690+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:25.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:25.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:25.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:25.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:25.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:25.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:25.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:25.138+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:25.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:25.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:25.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:25.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:25.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:25.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:25.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:25.748+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:25.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:25.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:25.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:25.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:25.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:25.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:25.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:25.669+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:26.343 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:19:25.999+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 10091 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:19:26.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:26.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:26.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:26.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:26.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:26.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:26.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:26.161+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:26.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:26.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:26.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:26.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:26.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:26.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:26.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:26.765+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:26.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:26.654+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:26.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:26.654+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:26.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:26.654+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:26.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:26.655+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:27.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:27.124+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:27.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:27.124+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:27.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:27.124+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:27.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:27.125+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:27.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:27.626+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:27.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:27.626+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:27.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:27.626+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:27.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:27.626+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:27.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:27.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:27.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:27.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:27.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:27.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:27.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:27.776+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:28.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:28.115+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:28.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:28.115+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:28.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:28.115+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:28.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:28.115+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:28.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:28.601+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:28.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:28.601+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:28.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:28.601+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:28.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:28.602+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:29.051 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:28.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:29.051 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:28.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:29.051 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:28.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:29.051 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:28.745+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:29.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:29.074+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:29.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:29.074+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:29.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:29.074+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:29.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:29.075+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:29.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:29.595+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:29.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:29.595+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:29.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:29.595+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:29.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:29.595+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:29.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:29.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:29.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:29.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:29.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:29.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:29.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:29.714+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:30.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:30.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:30.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:30.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:30.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:30.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:30.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:30.037+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:30.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:30.590+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:30.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:30.590+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:30.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:30.590+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:30.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:30.591+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:30.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:30.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:30.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:30.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:30.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:30.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:30.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:30.759+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:31.447 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:19:31.001+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 10185 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:19:31.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:31.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:31.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:31.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:31.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:31.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:31.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:31.012+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:31.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:31.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:31.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:31.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:31.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:31.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:31.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:31.754+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:31.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:31.594+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:31.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:31.594+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:31.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:31.594+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:31.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:31.595+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:32.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:31.991+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:32.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:31.991+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:32.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:31.991+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:32.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:31.993+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:32.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:32.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:32.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:32.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:32.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:32.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:32.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:32.760+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:32.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:32.583+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:32.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:32.583+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:32.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:32.583+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:32.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:32.583+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:33.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:32.991+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:33.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:32.991+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:33.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:32.991+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:33.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:32.993+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:33.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:33.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:33.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:33.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:33.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:33.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:33.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:33.725+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:33.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:33.606+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:33.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:33.606+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:33.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:33.606+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:33.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:33.606+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:34.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:33.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:34.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:33.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:34.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:33.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:34.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:33.997+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:34.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:34.570+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:34.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:34.570+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:34.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:34.570+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:34.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:34.570+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:34.995 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:34.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:34.995 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:34.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:34.995 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:34.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:34.995 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:34.723+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:35.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:35.018+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:35.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:35.018+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:35.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:35.018+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:35.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:35.019+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:35.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:35.730+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:35.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:35.730+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:35.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:35.730+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:35.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:35.731+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:35.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:35.578+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:35.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:35.578+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:35.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:35.578+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:35.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:35.578+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:36.257 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:19:36.001+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 10279 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:19:36.258 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:36.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:36.258 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:36.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:36.258 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:36.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:36.258 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:36.010+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:36.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:36.541+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:36.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:36.541+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:36.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:36.541+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:36.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:36.542+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:36.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:36.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:36.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:36.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:36.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:36.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:36.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:36.722+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:37.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:36.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:37.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:36.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:37.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:36.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:37.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:36.984+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:37.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:37.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:37.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:37.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:37.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:37.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:37.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:37.712+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:37.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:37.550+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:37.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:37.550+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:37.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:37.550+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:37.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:37.550+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:38.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:38.030+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:38.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:38.030+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:38.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:38.030+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:38.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:38.031+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:38.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:38.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:38.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:38.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:38.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:38.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:38.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:38.670+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:38.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:38.578+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:38.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:38.578+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:38.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:38.578+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:38.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:38.578+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:39.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:39.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:39.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:39.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:39.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:39.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:39.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:39.017+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:39.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:39.572+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:39.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:39.572+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:39.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:39.572+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:39.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:39.573+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:39.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:39.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:39.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:39.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:39.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:39.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:39.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:39.706+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:40.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:40.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:40.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:40.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:40.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:40.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:40.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:40.054+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:40.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:40.580+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:40.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:40.580+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:40.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:40.580+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:40.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:40.581+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:40.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:40.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:40.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:40.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:40.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:40.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:40.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:40.678+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:41.309 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:19:41.002+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 10373 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:19:41.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:41.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:41.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:41.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:41.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:41.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:41.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:41.050+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:41.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:41.561+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:41.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:41.561+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:41.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:41.561+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:41.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:41.561+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:41.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:41.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:41.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:41.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:41.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:41.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:41.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:41.633+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:42.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:42.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:42.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:42.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:42.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:42.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:42.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:42.022+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:42.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:42.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:42.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:42.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:42.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:42.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:42.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:42.621+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:42.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:42.566+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:42.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:42.566+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:42.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:42.566+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:42.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:42.566+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:43.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:43.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:43.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:43.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:43.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:43.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:43.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:43.039+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:43.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:43.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:43.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:43.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:43.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:43.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:43.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:43.668+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:43.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:43.576+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:43.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:43.577+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:43.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:43.577+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:43.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:43.577+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:44.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:44.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:44.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:44.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:44.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:44.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:44.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:44.048+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:44.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:44.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:44.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:44.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:44.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:44.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:44.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:44.679+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:44.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:44.597+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:44.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:44.597+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:44.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:44.597+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:44.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:44.597+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:45.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:45.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:45.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:45.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:45.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:45.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:45.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:45.092+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:45.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:45.586+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:45.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:45.586+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:45.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:45.586+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:45.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:45.586+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:45.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:45.675+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:45.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:45.675+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:45.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:45.675+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:45.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:45.675+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:46.343 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:19:46.003+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 10467 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:19:46.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:46.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:46.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:46.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:46.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:46.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:46.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:46.142+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:46.609 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:46.542+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:46.609 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:46.542+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:46.609 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:46.542+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:46.609 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:46.542+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:46.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:46.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:46.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:46.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:46.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:46.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:46.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:46.632+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:47.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:47.189+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:47.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:47.189+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:47.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:47.189+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:47.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:47.189+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:47.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:47.561+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:47.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:47.561+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:47.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:47.561+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:47.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:47.561+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:47.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:47.635+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:47.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:47.635+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:47.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:47.635+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:47.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:47.636+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:48.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:48.207+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:48.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:48.207+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:48.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:48.207+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:48.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:48.208+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:48.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:48.585+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:48.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:48.585+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:48.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:48.585+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:48.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:48.586+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:48.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:48.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:48.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:48.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:48.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:48.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:48.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:48.674+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:49.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:49.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:49.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:49.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:49.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:49.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:49.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:49.185+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:49.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:49.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:49.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:49.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:49.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:49.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:49.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:49.641+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:49.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:49.582+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:49.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:49.582+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:49.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:49.582+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:49.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:49.582+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:50.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:50.191+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:50.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:50.191+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:50.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:50.191+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:50.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:50.192+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:50.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:50.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:50.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:50.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:50.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:50.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:50.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:50.621+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:50.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:50.618+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:50.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:50.618+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:50.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:50.618+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:50.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:50.618+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:51.401 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:19:51.006+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 10561 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:19:51.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:51.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:51.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:51.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:51.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:51.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:51.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:51.184+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:51.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:51.611+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:51.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:51.611+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:51.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:51.611+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:51.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:51.612+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:51.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:51.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:51.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:51.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:51.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:51.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:51.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:51.596+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:52.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:52.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:52.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:52.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:52.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:52.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:52.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:52.136+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:52.806 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:52.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:52.807 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:52.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:52.807 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:52.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:52.807 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:52.647+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:52.807 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:52.565+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:52.807 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:52.565+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:52.807 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:52.565+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:52.807 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:52.565+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:53.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:53.160+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:53.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:53.160+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:53.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:53.160+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:53.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:53.161+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:53.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:53.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:53.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:53.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:53.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:53.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:53.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:53.524+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:53.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:53.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:53.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:53.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:53.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:53.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:53.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:53.693+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:54.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:54.169+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:54.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:54.169+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:54.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:54.169+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:54.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:54.171+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:54.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:54.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:54.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:54.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:54.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:54.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:54.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:54.714+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:54.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:54.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:54.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:54.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:54.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:54.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:54.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:54.540+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:55.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:55.134+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:55.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:55.134+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:55.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:55.134+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:55.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:55.136+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:55.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:55.675+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:55.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:55.675+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:55.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:55.675+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:55.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:55.676+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:55.879 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:55.521+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:55.879 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:55.521+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:55.879 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:55.521+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:55.880 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:55.522+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:56.128 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:19:56.007+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 10655 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:19:56.128 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:56.151+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:56.128 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:56.151+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:56.128 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:56.151+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:56.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:56.152+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:56.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:56.482+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:56.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:56.482+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:56.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:56.482+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:56.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:56.483+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:56.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:56.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:56.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:56.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:56.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:56.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:56.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:56.693+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:57.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:57.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:57.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:57.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:57.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:57.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:57.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:57.117+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:57.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:57.519+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:57.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:57.519+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:57.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:57.519+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:57.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:57.520+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:57.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:57.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:57.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:57.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:57.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:57.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:57.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:57.659+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:58.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:58.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:58.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:58.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:58.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:58.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:58.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:58.161+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:58.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:58.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:58.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:58.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:58.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:58.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:58.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:58.683+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:58.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:58.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:58.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:58.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:58.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:58.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:58.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:58.494+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:59.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:59.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:59.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:59.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:59.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:59.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:19:59.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:19:59.187+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:59.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:59.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:59.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:59.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:59.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:59.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:19:59.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:19:59.690+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:19:59.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:59.498+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:59.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:59.498+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:59.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:59.498+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:19:59.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:19:59.498+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:00.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:00.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:00.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:00.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:00.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:00.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:00.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:00.182+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:00.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:00.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:00.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:00.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:00.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:00.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:00.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:00.699+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:00.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:00.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:00.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:00.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:00.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:00.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:00.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:00.471+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:01.446 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:20:01.008+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 10749 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:20:01.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:01.177+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:01.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:01.177+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:01.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:01.177+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:01.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:01.178+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:01.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:01.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:01.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:01.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:01.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:01.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:01.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:01.662+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:01.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:01.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:01.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:01.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:01.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:01.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:01.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:01.508+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:02.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:02.185+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:02.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:02.185+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:02.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:02.185+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:02.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:02.186+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:02.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:02.542+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:02.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:02.542+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:02.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:02.542+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:02.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:02.543+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:02.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:02.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:02.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:02.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:02.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:02.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:02.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:02.638+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:03.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:03.163+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:03.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:03.163+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:03.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:03.163+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:03.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:03.164+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:03.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:03.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:03.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:03.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:03.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:03.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:03.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:03.628+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:03.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:03.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:03.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:03.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:03.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:03.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:03.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:03.537+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:04.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:04.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:04.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:04.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:04.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:04.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:04.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:04.142+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:04.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:04.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:04.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:04.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:04.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:04.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:04.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:04.669+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:04.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:04.517+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:04.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:04.517+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:04.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:04.517+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:04.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:04.517+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:05.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:05.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:05.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:05.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:05.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:05.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:05.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:05.104+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:05.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:05.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:05.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:05.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:05.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:05.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:05.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:05.662+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:05.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:05.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:05.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:05.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:05.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:05.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:05.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:05.539+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:06.344 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:20:06.010+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 10843 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:20:06.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:06.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:06.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:06.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:06.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:06.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:06.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:06.138+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:06.602 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:06.552+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:06.602 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:06.552+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:06.602 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:06.552+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:06.602 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:06.552+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:06.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:06.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:06.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:06.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:06.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:06.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:06.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:06.626+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:07.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:07.169+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:07.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:07.169+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:07.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:07.169+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:07.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:07.170+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:07.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:07.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:07.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:07.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:07.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:07.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:07.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:07.598+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:07.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:07.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:07.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:07.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:07.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:07.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:07.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:07.505+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:08.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:08.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:08.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:08.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:08.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:08.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:08.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:08.176+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:08.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:08.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:08.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:08.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:08.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:08.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:08.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:08.563+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:08.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:08.520+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:08.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:08.520+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:08.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:08.520+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:08.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:08.520+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:09.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:09.139+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:09.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:09.139+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:09.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:09.139+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:09.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:09.140+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:09.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:09.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:09.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:09.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:09.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:09.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:09.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:09.524+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:09.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:09.516+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:09.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:09.516+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:09.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:09.516+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:09.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:09.517+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:10.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:10.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:10.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:10.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:10.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:10.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:10.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:10.120+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:10.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:10.489+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:10.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:10.489+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:10.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:10.489+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:10.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:10.489+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:10.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:10.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:10.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:10.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:10.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:10.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:10.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:10.499+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:11.446 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:20:11.011+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 10937 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:20:11.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:11.160+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:11.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:11.160+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:11.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:11.160+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:11.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:11.161+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:11.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:11.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:11.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:11.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:11.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:11.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:11.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:11.499+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:11.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:11.536+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:11.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:11.536+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:11.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:11.536+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:11.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:11.537+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:12.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:12.178+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:12.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:12.178+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:12.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:12.178+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:12.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:12.179+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:12.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:12.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:12.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:12.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:12.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:12.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:12.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:12.568+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:12.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:12.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:12.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:12.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:12.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:12.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:12.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:12.545+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:13.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:13.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:13.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:13.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:13.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:13.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:13.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:13.185+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:13.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:13.548+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:13.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:13.548+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:13.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:13.548+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:13.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:13.549+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:13.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:13.540+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:13.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:13.540+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:13.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:13.541+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:13.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:13.541+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:14.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:14.168+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:14.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:14.168+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:14.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:14.168+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:14.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:14.168+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:14.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:14.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:14.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:14.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:14.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:14.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:14.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:14.528+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:14.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:14.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:14.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:14.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:14.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:14.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:14.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:14.530+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:15.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:15.171+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:15.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:15.171+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:15.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:15.171+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:15.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:15.172+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:15.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:15.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:15.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:15.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:15.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:15.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:15.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:15.563+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:15.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:15.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:15.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:15.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:15.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:15.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:15.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:15.550+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:16.344 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:20:16.012+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 11031 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:20:16.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:16.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:16.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:16.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:16.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:16.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:16.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:16.139+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:16.641 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:16.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:16.641 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:16.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:16.641 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:16.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:16.641 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:16.571+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:16.641 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:16.544+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:16.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:16.544+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:16.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:16.544+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:16.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:16.545+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:17.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:17.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:17.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:17.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:17.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:17.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:17.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:17.172+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:17.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:17.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:17.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:17.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:17.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:17.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:17.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:17.538+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:17.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:17.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:17.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:17.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:17.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:17.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:17.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:17.500+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:18.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:18.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:18.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:18.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:18.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:18.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:18.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:18.176+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:18.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:18.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:18.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:18.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:18.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:18.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:18.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:18.560+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:18.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:18.471+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:18.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:18.471+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:18.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:18.471+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:18.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:18.472+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:19.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:19.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:19.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:19.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:19.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:19.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:19.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:19.213+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:19.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:19.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:19.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:19.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:19.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:19.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:19.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:19.428+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:19.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:19.529+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:19.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:19.529+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:19.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:19.529+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:19.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:19.530+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:20.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:20.244+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:20.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:20.244+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:20.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:20.244+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:20.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:20.245+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:20.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:20.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:20.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:20.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:20.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:20.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:20.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:20.533+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:20.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:20.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:20.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:20.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:20.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:20.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:20.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:20.447+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:21.271 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:20:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:20:21.014+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 11125 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:20:21.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:21.481+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:21.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:21.481+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:21.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:21.481+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:21.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:21.482+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:21.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:21.294+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:21.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:21.295+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:21.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:21.295+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:21.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:21.295+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:21.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:21.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:21.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:21.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:21.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:21.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:21.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:21.532+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:22.535 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:22.527+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:22.535 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:22.527+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:22.535 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:22.527+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:22.535 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:22.528+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:22.535 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:22.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:22.535 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:22.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:22.535 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:22.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:22.535 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:22.290+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:22.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:22.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:22.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:22.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:22.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:22.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:22.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:22.559+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:23.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:23.501+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:23.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:23.501+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:23.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:23.501+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:23.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:23.501+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:23.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:23.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:23.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:23.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:23.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:23.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:23.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:23.324+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:23.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:23.525+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:23.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:23.525+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:23.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:23.525+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:23.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:23.525+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:24.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:24.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:24.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:24.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:24.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:24.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:24.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:24.530+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:24.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:24.302+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:24.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:24.302+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:24.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:24.302+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:24.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:24.302+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:24.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:24.548+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:24.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:24.548+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:24.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:24.548+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:24.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:24.549+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:25.553 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:25.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:25.553 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:25.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:25.553 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:25.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:25.553 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:25.522+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:25.553 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:25.298+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:25.553 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:25.298+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:25.553 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:25.298+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:25.554 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:25.298+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:25.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:25.576+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:25.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:25.576+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:25.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:25.576+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:25.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:25.576+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:26.270 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:20:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:20:26.015+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 11219 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:20:26.520 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:26.293+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:26.520 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:26.293+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:26.520 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:26.293+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:26.520 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:26.294+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:26.521 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:26.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:26.521 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:26.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:26.521 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:26.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:26.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:26.544+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:26.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:26.559+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:26.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:26.559+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:26.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:26.559+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:26.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:26.559+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:27.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:27.301+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:27.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:27.301+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:27.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:27.301+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:27.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:27.302+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:27.562 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:27.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:27.563 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:27.514+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:27.563 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:27.514+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:27.563 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:27.515+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:27.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:27.585+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:27.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:27.585+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:27.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:27.585+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:27.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:27.585+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:28.529 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:28.299+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:28.530 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:28.299+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:28.530 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:28.299+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:28.530 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:28.300+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:28.530 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:28.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:28.530 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:28.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:28.530 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:28.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:28.530 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:28.475+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:28.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:28.552+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:28.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:28.552+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:28.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:28.552+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:28.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:28.553+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:29.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:29.343+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:29.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:29.343+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:29.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:29.343+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:29.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:29.344+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:29.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:29.525+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:29.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:29.525+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:29.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:29.525+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:29.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:29.525+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:29.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:29.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:29.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:29.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:29.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:29.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:29.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:29.437+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:30.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:30.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:30.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:30.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:30.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:30.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:30.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:30.369+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:30.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:30.510+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:30.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:30.510+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:30.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:30.510+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:30.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:30.510+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:30.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:30.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:30.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:30.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:30.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:30.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:30.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:30.426+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:31.298 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:20:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:20:31.016+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 11313 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:20:31.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:31.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:31.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:31.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:31.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:31.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:31.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:31.322+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:31.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:31.394+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:31.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:31.394+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:31.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:31.394+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:31.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:31.395+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:31.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:31.554+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:31.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:31.554+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:31.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:31.554+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:31.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:31.555+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:32.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:32.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:32.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:32.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:32.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:32.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:32.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:32.437+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:32.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:32.525+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:32.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:32.525+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:32.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:32.525+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:32.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:32.525+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:32.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:32.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:32.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:32.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:32.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:32.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:32.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:32.368+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:33.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:33.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:33.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:33.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:33.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:33.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:33.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:33.523+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:33.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:33.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:33.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:33.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:33.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:33.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:33.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:33.451+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:33.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:33.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:33.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:33.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:33.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:33.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:33.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:33.337+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:34.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:34.538+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:34.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:34.538+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:34.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:34.538+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:34.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:34.539+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:34.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:34.451+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:34.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:34.451+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:34.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:34.451+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:34.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:34.452+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:34.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:34.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:34.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:34.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:34.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:34.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:34.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:34.364+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:35.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:35.569+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:35.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:35.569+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:35.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:35.569+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:35.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:35.570+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:35.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:35.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:35.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:35.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:35.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:35.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:35.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:35.498+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:35.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:35.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:35.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:35.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:35.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:35.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:35.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:35.334+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:36.117 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:20:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:20:36.016+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 11407 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:20:36.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:36.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:36.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:36.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:36.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:36.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:36.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:36.350+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:36.825 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:36.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:36.826 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:36.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:36.826 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:36.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:36.826 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:36.544+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:36.826 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:36.552+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:36.826 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:36.552+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:36.826 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:36.552+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:36.826 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:36.553+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:37.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:37.531+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:37.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:37.532+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:37.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:37.532+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:37.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:37.532+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:37.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:37.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:37.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:37.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:37.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:37.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:37.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:37.323+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:37.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:37.516+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:37.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:37.516+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:37.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:37.516+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:37.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:37.517+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:38.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:38.561+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:38.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:38.561+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:38.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:38.561+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:38.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:38.562+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:38.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:38.328+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:38.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:38.328+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:38.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:38.328+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:38.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:38.329+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:38.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:38.504+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:38.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:38.504+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:38.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:38.504+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:38.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:38.505+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:39.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:39.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:39.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:39.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:39.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:39.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:39.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:39.307+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:39.583 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:39.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:39.583 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:39.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:39.583 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:39.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:39.583 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:39.498+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:39.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:39.604+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:39.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:39.604+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:39.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:39.604+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:39.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:39.605+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:40.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:40.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:40.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:40.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:40.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:40.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:40.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:40.301+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:40.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:40.577+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:40.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:40.577+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:40.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:40.577+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:40.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:40.577+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:40.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:40.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:40.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:40.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:40.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:40.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:40.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:40.491+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:41.300 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:20:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:20:41.017+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 11501 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:20:41.599 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:41.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:41.599 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:41.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:41.599 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:41.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:41.599 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:41.486+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:41.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:41.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:41.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:41.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:41.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:41.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:41.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:41.324+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:41.871 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:41.621+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:41.871 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:41.621+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:41.871 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:41.621+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:41.871 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:41.622+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:42.563 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:42.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:42.563 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:42.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:42.563 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:42.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:42.563 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:42.336+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:42.563 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:42.447+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:42.564 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:42.447+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:42.564 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:42.447+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:42.564 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:42.448+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:42.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:42.585+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:42.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:42.585+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:42.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:42.585+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:42.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:42.586+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:43.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:43.417+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:43.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:43.417+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:43.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:43.417+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:43.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:43.417+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:43.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:43.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:43.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:43.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:43.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:43.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:43.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:43.322+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:43.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:43.636+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:43.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:43.636+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:43.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:43.636+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:43.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:43.636+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:44.633 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:44.426+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:44.633 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:44.426+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:44.633 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:44.426+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:44.633 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:44.426+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:44.633 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:44.311+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:44.634 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:44.311+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:44.634 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:44.311+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:44.634 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:44.312+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:44.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:44.656+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:44.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:44.656+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:44.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:44.656+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:44.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:44.656+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:45.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:45.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:45.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:45.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:45.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:45.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:45.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:45.306+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:45.587 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:45.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:45.587 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:45.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:45.587 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:45.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:45.587 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:45.423+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:45.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:45.610+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:45.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:45.610+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:45.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:45.610+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:45.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:45.610+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:46.299 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:20:46.019+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 11595 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:20:46.551 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:46.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:46.551 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:46.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:46.552 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:46.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:46.552 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:46.323+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:46.552 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:46.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:46.552 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:46.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:46.552 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:46.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:46.552 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:46.435+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:46.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:46.574+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:46.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:46.574+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:46.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:46.574+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:46.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:46.575+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:47.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:47.351+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:47.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:47.351+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:47.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:47.351+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:47.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:47.353+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:47.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:47.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:47.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:47.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:47.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:47.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:47.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:47.447+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:47.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:47.562+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:47.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:47.562+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:47.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:47.562+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:47.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:47.563+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:48.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:48.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:48.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:48.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:48.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:48.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:48.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:48.388+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:48.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:48.430+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:48.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:48.430+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:48.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:48.430+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:48.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:48.430+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:48.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:48.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:48.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:48.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:48.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:48.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:48.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:48.528+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:49.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:49.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:49.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:49.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:49.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:49.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:49.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:49.438+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:49.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:49.451+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:49.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:49.451+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:49.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:49.451+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:49.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:49.451+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:49.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:49.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:49.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:49.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:49.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:49.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:49.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:49.530+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:50.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:50.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:50.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:50.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:50.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:50.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:50.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:50.429+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:50.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:50.447+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:50.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:50.447+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:50.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:50.447+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:50.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:50.448+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:50.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:50.527+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:50.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:50.527+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:50.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:50.527+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:50.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:50.527+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:51.408 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:20:51.021+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 11689 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:20:51.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:51.517+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:51.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:51.517+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:51.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:51.517+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:51.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:51.518+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:51.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:51.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:51.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:51.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:51.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:51.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:51.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:51.432+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:51.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:51.487+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:51.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:51.487+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:51.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:51.487+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:51.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:51.488+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:52.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:52.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:52.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:52.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:52.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:52.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:52.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:52.483+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:52.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:52.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:52.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:52.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:52.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:52.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:52.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:52.481+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:52.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:52.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:52.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:52.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:52.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:52.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:52.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:52.446+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:53.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:53.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:53.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:53.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:53.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:53.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:53.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:53.468+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:53.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:53.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:53.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:53.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:53.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:53.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:53.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:53.474+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:53.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:53.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:53.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:53.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:53.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:53.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:53.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:53.461+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:54.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:54.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:54.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:54.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:54.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:54.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:54.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:54.478+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:54.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:54.429+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:54.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:54.429+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:54.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:54.429+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:54.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:54.429+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:54.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:54.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:54.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:54.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:54.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:54.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:54.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:54.447+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:55.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:55.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:55.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:55.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:55.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:55.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:55.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:55.465+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:55.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:55.419+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:55.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:55.419+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:55.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:55.419+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:55.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:55.420+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:55.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:55.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:55.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:55.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:55.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:55.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:55.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:55.417+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:56.343 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:20:56.022+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 11784 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:20:56.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:56.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:56.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:56.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:56.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:56.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:56.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:56.396+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:56.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:56.499+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:56.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:56.499+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:56.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:56.499+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:56.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:56.500+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:56.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:56.384+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:56.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:56.384+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:56.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:56.384+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:56.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:56.385+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:57.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:57.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:57.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:57.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:57.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:57.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:57.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:57.514+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:57.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:57.343+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:57.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:57.343+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:57.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:57.343+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:57.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:57.344+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:57.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:57.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:57.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:57.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:57.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:57.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:57.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:57.348+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:58.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:58.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:58.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:58.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:58.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:58.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:58.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:58.394+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:58.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:58.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:58.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:58.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:58.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:58.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:58.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:58.478+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:58.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:58.382+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:58.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:58.382+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:58.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:58.382+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:58.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:58.382+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:59.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:59.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:59.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:59.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:59.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:59.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:20:59.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:20:59.500+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:59.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:59.338+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:59.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:59.338+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:59.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:59.338+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:20:59.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:20:59.338+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:20:59.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:59.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:59.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:59.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:59.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:59.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:20:59.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:20:59.406+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:00.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:00.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:00.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:00.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:00.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:00.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:00.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:00.384+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:00.390 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:00.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:00.390 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:00.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:00.391 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:00.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:00.391 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:00.352+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:00.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:00.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:00.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:00.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:00.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:00.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:00.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:00.521+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:01.323 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:21:01.024+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 11878 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:21:01.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:01.346+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:01.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:01.346+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:01.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:01.346+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:01.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:01.347+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:01.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:01.515+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:01.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:01.515+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:01.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:01.515+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:01.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:01.516+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:01.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:01.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:01.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:01.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:01.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:01.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:01.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:01.406+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:02.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:02.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:02.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:02.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:02.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:02.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:02.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:02.406+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:02.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:02.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:02.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:02.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:02.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:02.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:02.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:02.501+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:02.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:02.378+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:02.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:02.378+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:02.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:02.378+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:02.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:02.379+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:03.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:03.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:03.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:03.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:03.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:03.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:03.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:03.515+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:03.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:03.346+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:03.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:03.346+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:03.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:03.346+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:03.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:03.347+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:03.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:03.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:03.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:03.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:03.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:03.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:03.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:03.444+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:04.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:04.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:04.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:04.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:04.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:04.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:04.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:04.477+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:04.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:04.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:04.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:04.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:04.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:04.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:04.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:04.358+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:04.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:04.492+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:04.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:04.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:04.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:04.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:04.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:04.493+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:05.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:05.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:05.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:05.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:05.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:05.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:05.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:05.508+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:05.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:05.457+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:05.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:05.457+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:05.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:05.457+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:05.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:05.457+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:05.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:05.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:05.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:05.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:05.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:05.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:05.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:05.311+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:06.344 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:21:06.025+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 11972 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:21:06.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:06.267+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:06.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:06.267+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:06.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:06.267+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:06.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:06.268+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:06.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:06.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:06.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:06.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:06.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:06.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:06.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:06.496+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:06.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:06.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:06.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:06.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:06.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:06.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:06.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:06.468+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:07.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:07.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:07.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:07.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:07.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:07.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:07.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:07.317+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:07.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:07.480+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:07.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:07.480+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:07.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:07.480+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:07.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:07.481+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:07.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:07.478+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:07.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:07.478+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:07.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:07.478+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:07.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:07.478+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:08.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:08.284+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:08.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:08.284+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:08.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:08.284+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:08.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:08.285+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:08.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:08.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:08.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:08.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:08.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:08.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:08.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:08.452+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:08.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:08.464+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:08.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:08.464+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:08.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:08.464+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:08.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:08.464+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:09.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:09.268+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:09.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:09.268+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:09.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:09.268+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:09.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:09.268+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:09.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:09.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:09.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:09.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:09.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:09.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:09.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:09.446+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:09.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:09.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:09.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:09.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:09.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:09.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:09.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:09.484+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:10.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:10.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:10.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:10.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:10.415 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:10.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:10.415 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:10.274+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:10.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:10.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:10.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:10.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:10.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:10.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:10.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:10.454+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:10.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:10.437+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:10.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:10.437+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:10.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:10.437+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:10.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:10.438+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:11.436 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:21:11.026+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 12066 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:21:11.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:11.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:11.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:11.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:11.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:11.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:11.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:11.271+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:11.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:11.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:11.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:11.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:11.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:11.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:11.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:11.460+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:11.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:11.482+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:11.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:11.482+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:11.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:11.482+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:11.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:11.482+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:12.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:12.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:12.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:12.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:12.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:12.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:12.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:12.281+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:12.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:12.484+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:12.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:12.484+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:12.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:12.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:12.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:12.486+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:12.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:12.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:12.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:12.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:12.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:12.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:12.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:12.500+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:13.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:13.452+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:13.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:13.452+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:13.545 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:13.452+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:13.545 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:13.453+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:13.545 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:13.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:13.545 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:13.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:13.545 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:13.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:13.545 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:13.460+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:13.545 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:13.237+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:13.545 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:13.237+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:13.546 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:13.237+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:13.546 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:13.238+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:14.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:14.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:14.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:14.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:14.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:14.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:14.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:14.449+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:14.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:14.205+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:14.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:14.205+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:14.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:14.205+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:14.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:14.206+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:14.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:14.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:14.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:14.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:14.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:14.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:14.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:14.482+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:15.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:15.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:15.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:15.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:15.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:15.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:15.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:15.175+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:15.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:15.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:15.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:15.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:15.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:15.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:15.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:15.488+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:15.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:15.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:15.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:15.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:15.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:15.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:15.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:15.451+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:16.343 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:21:16.028+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 12160 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:21:16.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:16.148+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:16.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:16.148+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:16.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:16.148+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:16.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:16.149+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:16.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:16.496+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:16.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:16.496+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:16.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:16.496+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:16.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:16.497+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:16.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:16.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:16.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:16.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:16.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:16.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:16.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:16.407+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:17.382 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:17.125+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:17.382 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:17.125+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:17.382 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:17.125+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:17.382 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:17.125+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:17.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:17.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:17.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:17.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:17.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:17.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:17.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:17.407+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:17.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:17.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:17.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:17.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:17.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:17.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:17.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:17.509+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:18.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:18.135+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:18.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:18.135+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:18.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:18.135+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:18.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:18.135+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:18.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:18.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:18.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:18.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:18.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:18.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:18.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:18.359+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:18.822 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:18.520+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:18.822 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:18.520+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:18.822 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:18.520+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:18.822 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:18.520+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:19.316 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:19.089+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:19.316 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:19.089+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:19.316 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:19.089+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:19.316 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:19.090+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:19.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:19.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:19.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:19.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:19.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:19.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:19.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:19.340+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:19.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:19.514+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:19.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:19.514+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:19.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:19.514+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:19.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:19.514+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:20.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:20.045+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:20.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:20.045+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:20.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:20.045+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:20.367 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:20.046+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:20.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:20.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:20.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:20.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:20.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:20.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:20.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:20.390+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:20.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:20.534+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:20.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:20.534+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:20.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:20.534+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:20.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:20.535+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:21.346 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:21:21.029+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 12254 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:21:21.346 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:21.072+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:21.346 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:21.072+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:21.346 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:21.072+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:21.346 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:21.073+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:21.638 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:21.554+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:21.638 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:21.554+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:21.639 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:21.554+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:21.639 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:21.555+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:21.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:21.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:21.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:21.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:21.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:21.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:21.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:21.371+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:22.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:22.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:22.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:22.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:22.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:22.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:22.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:22.085+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:22.577 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:22.394+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:22.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:22.394+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:22.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:22.394+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:22.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:22.395+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:22.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:22.601+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:22.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:22.601+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:22.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:22.601+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:22.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:22.601+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:23.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:23.085+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:23.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:23.085+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:23.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:23.086+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:23.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:23.086+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:23.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:23.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:23.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:23.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:23.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:23.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:23.604 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:23.355+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:23.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:23.627+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:23.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:23.627+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:23.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:23.627+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:23.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:23.627+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:24.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:24.046+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:24.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:24.046+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:24.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:24.046+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:24.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:24.048+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:24.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:24.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:24.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:24.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:24.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:24.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:24.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:24.400+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:24.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:24.616+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:24.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:24.616+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:24.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:24.616+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:24.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:24.617+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:25.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:25.011+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:25.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:25.011+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:25.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:25.011+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:25.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:25.012+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:25.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:25.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:25.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:25.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:25.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:25.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:25.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:25.369+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:25.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:25.615+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:25.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:25.615+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:25.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:25.615+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:25.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:25.615+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:26.330 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:21:26.031+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 12348 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:21:26.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:26.002+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:26.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:26.002+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:26.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:26.002+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:26.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:26.003+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:26.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:26.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:26.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:26.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:26.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:26.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:26.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:26.354+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:26.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:26.592+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:26.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:26.592+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:26.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:26.592+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:26.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:26.593+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:27.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:26.976+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:27.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:26.976+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:27.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:26.976+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:27.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:26.977+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:27.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:27.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:27.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:27.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:27.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:27.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:27.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:27.358+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:27.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:27.548+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:27.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:27.548+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:27.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:27.548+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:27.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:27.548+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:28.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:28.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:28.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:28.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:28.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:28.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:28.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:28.001+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:28.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:28.407+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:28.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:28.407+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:28.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:28.407+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:28.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:28.408+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:28.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:28.576+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:28.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:28.576+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:28.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:28.576+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:28.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:28.576+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:29.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:29.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:29.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:29.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:29.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:29.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:29.356 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:29.027+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:29.638 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:29.594+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:29.638 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:29.594+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:29.638 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:29.594+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:29.639 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:29.594+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:29.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:29.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:29.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:29.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:29.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:29.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:29.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:29.380+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:30.402 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:30.046+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:30.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:30.046+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:30.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:30.046+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:30.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:30.047+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:30.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:30.598+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:30.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:30.598+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:30.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:30.598+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:30.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:30.598+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:30.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:30.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:30.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:30.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:30.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:30.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:30.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:30.427+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:31.379 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:21:31.032+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 12442 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:21:31.379 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:31.088+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:31.379 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:31.088+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:31.379 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:31.088+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:31.379 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:31.089+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:31.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:31.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:31.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:31.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:31.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:31.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:31.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:31.403+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:31.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:31.611+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:31.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:31.611+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:31.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:31.611+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:31.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:31.612+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:32.429 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:32.104+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:32.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:32.104+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:32.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:32.104+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:32.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:32.105+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:32.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:32.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:32.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:32.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:32.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:32.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:32.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:32.454+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:32.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:32.640+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:32.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:32.640+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:32.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:32.640+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:32.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:32.640+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:33.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:33.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:33.442 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:33.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:33.442 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:33.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:33.442 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:33.092+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:33.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:33.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:33.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:33.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:33.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:33.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:33.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:33.466+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:33.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:33.617+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:33.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:33.617+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:33.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:33.617+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:33.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:33.618+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:34.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:34.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:34.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:34.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:34.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:34.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:34.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:34.063+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:34.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:34.574+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:34.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:34.574+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:34.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:34.574+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:34.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:34.575+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:34.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:34.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:34.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:34.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:34.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:34.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:34.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:34.426+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:35.392 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:35.108+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:35.392 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:35.108+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:35.392 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:35.108+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:35.392 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:35.109+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:35.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:35.553+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:35.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:35.553+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:35.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:35.553+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:35.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:35.553+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:35.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:35.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:35.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:35.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:35.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:35.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:35.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:35.416+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:36.343 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:21:36.032+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 12533 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:21:36.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:36.132+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:36.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:36.132+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:36.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:36.132+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:36.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:36.133+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:36.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:36.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:36.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:36.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:36.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:36.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:36.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:36.390+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:36.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:36.587+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:36.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:36.587+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:36.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:36.587+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:36.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:36.587+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:37.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:37.392+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:37.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:37.392+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:37.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:37.392+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:37.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:37.393+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:37.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:37.154+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:37.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:37.154+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:37.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:37.154+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:37.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:37.154+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:37.830 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:37.574+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:37.830 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:37.574+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:37.830 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:37.574+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:37.830 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:37.574+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:38.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:38.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:38.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:38.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:38.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:38.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:38.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:38.386+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:38.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:38.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:38.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:38.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:38.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:38.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:38.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:38.189+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:38.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:38.609+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:38.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:38.609+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:38.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:38.609+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:38.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:38.610+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:39.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:39.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:39.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:39.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:39.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:39.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:39.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:39.429+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:39.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:39.213+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:39.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:39.213+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:39.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:39.213+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:39.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:39.214+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:39.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:39.572+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:39.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:39.572+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:39.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:39.572+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:39.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:39.572+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:40.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:40.196+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:40.399 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:40.196+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:40.399 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:40.196+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:40.399 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:40.197+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:40.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:40.421+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:40.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:40.421+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:40.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:40.421+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:40.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:40.422+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:40.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:40.569+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:40.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:40.569+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:40.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:40.569+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:40.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:40.569+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:41.377 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:21:41.034+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 12628 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:21:41.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:41.182+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:41.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:41.182+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:41.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:41.182+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:41.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:41.182+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:41.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:41.583+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:41.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:41.583+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:41.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:41.583+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:41.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:41.583+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:41.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:41.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:41.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:41.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:41.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:41.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:41.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:41.401+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:42.400 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:42.149+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:42.400 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:42.149+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:42.400 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:42.149+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:42.400 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:42.150+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:42.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:42.547+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:42.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:42.547+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:42.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:42.547+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:42.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:42.548+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:42.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:42.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:42.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:42.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:42.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:42.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:42.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:42.437+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:43.382 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:43.121+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:43.382 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:43.121+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:43.382 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:43.121+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:43.382 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:43.121+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:43.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:43.570+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:43.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:43.570+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:43.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:43.570+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:43.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:43.571+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:43.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:43.404+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:43.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:43.404+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:43.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:43.404+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:43.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:43.405+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:44.387 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:44.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:44.387 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:44.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:44.387 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:44.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:44.387 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:44.122+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:44.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:44.611+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:44.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:44.611+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:44.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:44.611+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:44.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:44.612+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:44.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:44.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:44.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:44.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:44.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:44.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:44.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:44.411+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:45.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:45.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:45.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:45.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:45.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:45.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:45.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:45.146+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:45.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:45.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:45.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:45.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:45.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:45.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:45.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:45.370+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:45.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:45.602+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:45.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:45.602+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:45.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:45.602+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:45.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:45.602+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:46.343 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:21:46.036+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 12723 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:21:46.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:46.101+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:46.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:46.101+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:46.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:46.101+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:46.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:46.102+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:46.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:21:46.625+0000 7f770c428700 0 monclient(hunting): authenticate timed out after 300 2024-04-27T02:21:46.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:46.564+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:46.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:46.564+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:46.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:46.564+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:46.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:46.564+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:46.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:46.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:46.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:46.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:46.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:46.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:46.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:46.401+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:47.440 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:47.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:47.440 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:47.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:47.441 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:47.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:47.441 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:47.354+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:47.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:47.108+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:47.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:47.108+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:47.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:47.108+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:47.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:47.108+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:47.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:47.556+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:47.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:47.556+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:47.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:47.556+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:47.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:47.557+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:48.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:48.120+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:48.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:48.120+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:48.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:48.120+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:48.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:48.120+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:48.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:48.351+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:48.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:48.351+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:48.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:48.351+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:48.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:48.351+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:48.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:48.559+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:48.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:48.559+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:48.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:48.559+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:48.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:48.560+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:49.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:49.105+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:49.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:49.105+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:49.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:49.105+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:49.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:49.106+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:49.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:49.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:49.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:49.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:49.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:49.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:49.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:49.313+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:49.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:49.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:49.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:49.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:49.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:49.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:49.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:49.537+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:50.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:50.147+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:50.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:50.147+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:50.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:50.147+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:50.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:50.148+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:50.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:50.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:50.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:50.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:50.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:50.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:50.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:50.347+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:50.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:50.514+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:50.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:50.514+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:50.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:50.514+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:50.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:50.514+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:51.352 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:21:51.037+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 12818 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:21:51.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:51.183+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:51.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:51.183+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:51.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:51.183+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:51.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:51.184+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:51.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:51.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:51.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:51.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:51.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:51.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:51.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:51.540+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:51.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:51.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:51.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:51.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:51.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:51.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:51.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:51.376+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:52.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:52.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:52.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:52.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:52.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:52.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:52.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:52.161+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:52.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:52.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:52.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:52.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:52.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:52.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:52.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:52.364+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:52.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:52.574+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:52.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:52.574+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:52.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:52.574+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:52.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:52.574+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:53.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:53.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:53.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:53.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:53.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:53.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:53.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:53.370+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:53.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:53.178+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:53.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:53.178+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:53.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:53.178+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:53.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:53.179+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:53.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:53.587+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:53.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:53.587+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:53.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:53.587+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:53.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:53.587+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:54.380 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:54.149+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:54.380 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:54.149+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:54.380 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:54.149+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:54.380 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:54.149+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:54.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:54.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:54.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:54.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:54.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:54.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:54.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:54.404+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:54.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:54.599+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:54.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:54.599+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:54.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:54.599+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:54.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:54.600+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:55.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:55.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:55.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:55.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:55.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:55.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:55.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:55.147+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:55.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:55.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:55.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:55.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:55.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:55.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:55.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:55.402+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:55.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:55.560+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:55.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:55.560+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:55.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:55.560+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:55.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:55.560+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:56.343 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:21:56.039+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 12913 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:21:56.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:56.148+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:56.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:56.148+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:56.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:56.148+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:56.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:56.148+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:56.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:56.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:56.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:56.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:56.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:56.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:56.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:56.425+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:56.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:56.546+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:56.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:56.546+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:56.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:56.546+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:56.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:56.546+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:57.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:57.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:57.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:57.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:57.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:57.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:57.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:57.379+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:57.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:57.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:57.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:57.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:57.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:57.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:57.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:57.135+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:57.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:57.593+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:57.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:57.593+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:57.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:57.593+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:57.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:57.594+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:58.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:58.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:58.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:58.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:58.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:58.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:58.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:58.379+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:58.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:58.140+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:58.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:58.140+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:58.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:58.140+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:58.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:58.140+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:58.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:58.608+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:58.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:58.608+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:58.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:58.608+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:58.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:58.608+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:59.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:59.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:59.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:59.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:59.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:59.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:21:59.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:21:59.339+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:59.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:59.109+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:59.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:59.109+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:59.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:59.109+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:21:59.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:21:59.110+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:21:59.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:59.563+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:59.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:59.563+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:59.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:59.563+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:21:59.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:21:59.564+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:00.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:00.296+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:00.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:00.296+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:00.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:00.296+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:00.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:00.297+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:00.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:00.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:00.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:00.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:00.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:00.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:00.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:00.147+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:00.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:00.563+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:00.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:00.563+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:00.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:00.563+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:00.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:00.563+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:01.446 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:22:01.040+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 13008 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:22:01.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:01.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:01.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:01.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:01.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:01.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:01.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:01.128+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:01.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:01.276+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:01.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:01.276+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:01.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:01.276+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:01.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:01.277+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:01.829 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:01.570+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:01.830 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:01.570+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:01.830 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:01.570+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:01.830 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:01.571+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:02.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:02.082+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:02.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:02.082+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:02.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:02.082+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:02.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:02.082+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:02.498 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:02.271+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:02.498 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:02.271+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:02.498 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:02.271+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:02.498 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:02.271+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:02.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:02.521+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:02.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:02.521+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:02.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:02.521+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:02.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:02.522+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:03.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:03.273+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:03.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:03.273+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:03.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:03.273+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:03.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:03.274+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:03.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:03.077+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:03.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:03.077+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:03.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:03.077+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:03.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:03.077+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:03.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:03.507+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:03.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:03.507+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:03.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:03.507+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:03.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:03.507+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:04.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:04.250+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:04.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:04.250+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:04.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:04.250+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:04.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:04.251+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:04.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:04.110+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:04.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:04.110+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:04.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:04.110+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:04.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:04.111+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:04.902 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:04.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:04.902 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:04.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:04.902 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:04.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:04.902 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:04.484+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:05.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:05.158+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:05.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:05.158+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:05.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:05.158+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:05.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:05.158+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:05.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:05.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:05.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:05.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:05.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:05.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:05.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:05.282+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:05.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:05.509+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:05.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:05.509+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:05.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:05.509+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:05.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:05.509+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:06.343 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:22:06.041+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 13103 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:22:06.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:06.281+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:06.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:06.281+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:06.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:06.281+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:06.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:06.282+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:06.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:06.137+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:06.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:06.137+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:06.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:06.137+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:06.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:06.138+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:06.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:06.555+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:06.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:06.555+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:06.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:06.555+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:06.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:06.556+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:07.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:07.167+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:07.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:07.167+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:07.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:07.167+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:07.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:07.168+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:07.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:07.302+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:07.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:07.302+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:07.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:07.302+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:07.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:07.303+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:07.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:07.597+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:07.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:07.597+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:07.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:07.597+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:07.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:07.598+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:08.258 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:08.217+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:08.258 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:08.217+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:08.259 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:08.217+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:08.259 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:08.218+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:08.565 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:08.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:08.565 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:08.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:08.566 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:08.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:08.566 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:08.282+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:08.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:08.589+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:08.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:08.589+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:08.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:08.589+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:08.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:08.589+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:09.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:09.297+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:09.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:09.297+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:09.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:09.297+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:09.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:09.297+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:09.587 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:09.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:09.587 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:09.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:09.587 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:09.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:09.587 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:09.240+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:09.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:09.610+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:09.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:09.610+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:09.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:09.610+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:09.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:09.610+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:10.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:10.245+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:10.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:10.245+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:10.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:10.245+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:10.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:10.245+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:10.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:10.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:10.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:10.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:10.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:10.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:10.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:10.340+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:10.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:10.594+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:10.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:10.594+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:10.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:10.594+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:10.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:10.594+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:11.322 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:22:11.043+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 13198 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:22:11.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:11.241+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:11.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:11.241+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:11.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:11.241+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:11.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:11.241+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:11.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:11.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:11.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:11.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:11.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:11.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:11.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:11.346+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:11.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:11.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:11.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:11.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:11.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:11.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:11.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:11.575+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:12.543 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:12.310+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:12.543 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:12.310+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:12.543 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:12.310+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:12.543 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:12.311+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:12.543 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:12.283+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:12.543 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:12.283+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:12.544 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:12.283+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:12.544 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:12.284+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:12.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:12.566+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:12.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:12.566+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:12.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:12.566+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:12.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:12.566+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:13.548 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:13.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:13.549 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:13.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:13.549 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:13.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:13.549 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:13.358+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:13.549 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:13.304+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:13.549 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:13.304+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:13.549 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:13.304+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:13.549 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:13.305+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:13.550 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:13.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:13.550 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:13.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:13.550 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:13.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:13.550 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:13.527+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:14.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:14.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:14.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:14.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:14.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:14.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:14.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:14.336+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:14.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:14.287+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:14.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:14.287+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:14.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:14.287+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:14.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:14.288+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:14.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:14.502+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:14.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:14.502+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:14.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:14.502+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:14.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:14.503+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:15.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:15.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:15.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:15.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:15.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:15.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:15.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:15.308+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:15.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:15.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:15.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:15.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:15.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:15.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:15.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:15.459+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:15.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:15.374+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:15.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:15.374+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:15.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:15.374+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:15.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:15.376+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:16.275 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:22:16.044+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 13293 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:22:16.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:16.469+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:16.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:16.469+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:16.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:16.469+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:16.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:16.469+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:16.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:16.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:16.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:16.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:16.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:16.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:16.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:16.336+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:16.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:16.299+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:16.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:16.299+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:16.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:16.299+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:16.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:16.300+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:17.441 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:17.294+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:17.441 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:17.294+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:17.441 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:17.294+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:17.442 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:17.295+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:17.442 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:17.328+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:17.442 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:17.328+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:17.442 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:17.328+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:17.442 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:17.329+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:17.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:17.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:17.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:17.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:17.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:17.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:17.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:17.465+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:18.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:18.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:18.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:18.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:18.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:18.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:18.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:18.504+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:18.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:18.311+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:18.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:18.311+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:18.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:18.311+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:18.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:18.313+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:18.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:18.294+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:18.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:18.295+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:18.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:18.295+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:18.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:18.295+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:19.522 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:19.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:19.522 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:19.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:19.522 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:19.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:19.523 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:19.305+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:19.523 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:19.292+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:19.523 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:19.292+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:19.523 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:19.292+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:19.523 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:19.292+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:19.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:19.546+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:19.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:19.546+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:19.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:19.546+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:19.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:19.546+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:20.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:20.306+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:20.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:20.306+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:20.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:20.306+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:20.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:20.307+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:20.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:20.267+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:20.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:20.267+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:20.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:20.267+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:20.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:20.268+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:20.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:20.582+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:20.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:20.582+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:20.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:20.582+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:20.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:20.583+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:21.275 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:22:21.046+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 13388 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:22:21.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:21.299+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:21.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:21.299+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:21.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:21.299+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:21.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:21.299+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:21.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:21.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:21.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:21.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:21.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:21.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:21.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:21.348+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:21.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:21.592+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:21.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:21.592+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:21.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:21.592+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:21.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:21.593+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:22.552 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:22.290+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:22.552 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:22.290+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:22.552 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:22.290+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:22.552 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:22.291+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:22.553 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:22.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:22.553 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:22.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:22.553 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:22.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:22.553 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:22.370+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:22.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:22.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:22.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:22.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:22.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:22.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:22.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:22.576+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:23.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:23.333+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:23.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:23.333+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:23.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:23.333+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:23.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:23.335+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:23.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:23.412+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:23.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:23.412+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:23.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:23.412+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:23.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:23.413+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:23.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:23.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:23.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:23.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:23.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:23.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:23.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:23.569+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:24.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:24.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:24.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:24.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:24.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:24.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:24.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:24.581+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:24.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:24.365+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:24.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:24.365+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:24.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:24.365+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:24.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:24.366+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:24.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:24.427+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:24.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:24.427+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:24.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:24.427+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:24.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:24.429+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:25.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:25.628+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:25.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:25.628+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:25.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:25.628+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:25.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:25.629+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:25.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:25.390+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:25.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:25.390+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:25.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:25.390+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:25.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:25.390+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:25.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:25.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:25.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:25.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:25.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:25.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:25.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:25.399+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:26.343 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:22:26.047+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 13483 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:22:26.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:26.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:26.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:26.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:26.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:26.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:26.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:26.378+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:26.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:26.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:26.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:26.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:26.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:26.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:26.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:26.447+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:26.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:26.614+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:26.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:26.614+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:26.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:26.614+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:26.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:26.614+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:27.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:27.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:27.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:27.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:27.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:27.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:27.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:27.420+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:27.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:27.479+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:27.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:27.479+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:27.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:27.479+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:27.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:27.480+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:27.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:27.580+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:27.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:27.580+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:27.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:27.580+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:27.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:27.580+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:28.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:28.409+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:28.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:28.409+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:28.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:28.409+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:28.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:28.409+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:28.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:28.588+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:28.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:28.588+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:28.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:28.588+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:28.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:28.589+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:28.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:28.525+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:28.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:28.525+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:28.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:28.525+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:28.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:28.525+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:29.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:29.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:29.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:29.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:29.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:29.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:29.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:29.420+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:29.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:29.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:29.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:29.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:29.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:29.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:29.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:29.501+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:29.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:29.600+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:29.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:29.600+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:29.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:29.600+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:29.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:29.600+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:30.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:30.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:30.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:30.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:30.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:30.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:30.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:30.414+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:30.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:30.595+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:30.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:30.595+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:30.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:30.595+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:30.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:30.595+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:30.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:30.536+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:30.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:30.536+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:30.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:30.536+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:30.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:30.537+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:31.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:22:31.048+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 13578 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:22:31.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:31.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:31.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:31.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:31.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:31.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:31.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:31.443+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:31.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:31.590+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:31.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:31.590+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:31.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:31.590+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:31.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:31.590+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:31.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:31.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:31.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:31.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:31.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:31.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:31.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:31.520+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:32.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:32.567+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:32.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:32.567+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:32.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:32.567+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:32.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:32.567+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:32.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:32.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:32.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:32.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:32.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:32.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:32.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:32.427+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:32.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:32.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:32.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:32.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:32.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:32.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:32.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:32.501+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:33.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:33.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:33.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:33.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:33.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:33.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:33.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:33.467+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:33.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:33.607+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:33.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:33.607+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:33.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:33.607+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:33.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:33.607+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:33.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:33.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:33.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:33.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:33.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:33.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:33.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:33.544+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:34.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:34.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:34.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:34.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:34.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:34.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:34.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:34.432+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:34.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:34.570+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:34.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:34.570+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:34.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:34.570+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:34.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:34.571+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:34.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:34.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:34.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:34.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:34.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:34.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:34.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:34.514+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:35.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:35.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:35.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:35.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:35.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:35.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:35.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:35.522+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:35.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:35.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:35.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:35.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:35.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:35.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:35.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:35.415+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:35.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:35.551+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:35.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:35.551+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:35.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:35.551+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:35.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:35.552+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:36.344 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:22:36.049+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 13673 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:22:36.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:36.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:36.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:36.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:36.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:36.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:36.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:36.572+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:36.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:36.551+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:36.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:36.552+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:36.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:36.552+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:36.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:36.552+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:36.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:36.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:36.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:36.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:36.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:36.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:36.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:36.406+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:37.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:37.398+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:37.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:37.398+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:37.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:37.398+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:37.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:37.399+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:37.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:37.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:37.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:37.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:37.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:37.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:37.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:37.576+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:37.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:37.560+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:37.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:37.560+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:37.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:37.560+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:37.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:37.560+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:38.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:38.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:38.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:38.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:38.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:38.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:38.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:38.401+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:38.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:38.588+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:38.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:38.588+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:38.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:38.588+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:38.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:38.589+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:38.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:38.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:38.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:38.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:38.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:38.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:38.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:38.537+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:39.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:39.522+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:39.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:39.522+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:39.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:39.522+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:39.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:39.522+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:39.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:39.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:39.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:39.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:39.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:39.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:39.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:39.437+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:39.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:39.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:39.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:39.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:39.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:39.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:39.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:39.635+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:40.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:40.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:40.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:40.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:40.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:40.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:40.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:40.473+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:40.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:40.490+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:40.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:40.490+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:40.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:40.490+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:40.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:40.490+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:40.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:40.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:40.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:40.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:40.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:40.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:40.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:40.664+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:41.413 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:22:41.050+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 13768 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:22:41.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:41.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:41.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:41.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:41.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:41.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:41.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:41.625+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:41.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:41.536+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:41.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:41.536+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:41.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:41.536+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:41.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:41.536+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:41.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:41.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:41.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:41.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:41.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:41.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:41.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:41.437+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:42.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:42.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:42.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:42.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:42.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:42.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:42.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:42.577+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:42.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:42.533+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:42.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:42.533+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:42.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:42.533+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:42.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:42.533+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:42.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:42.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:42.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:42.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:42.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:42.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:42.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:42.449+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:43.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:43.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:43.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:43.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:43.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:43.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:43.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:43.575+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:43.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:43.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:43.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:43.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:43.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:43.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:43.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:43.453+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:43.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:43.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:43.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:43.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:43.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:43.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:43.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:43.508+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:44.522 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:44.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:44.522 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:44.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:44.522 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:44.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:44.522 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:44.544+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:44.523 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:44.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:44.523 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:44.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:44.523 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:44.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:44.523 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:44.500+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:44.523 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:44.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:44.523 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:44.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:44.523 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:44.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:44.523 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:44.479+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:45.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:45.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:45.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:45.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:45.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:45.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:45.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:45.512+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:45.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:45.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:45.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:45.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:45.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:45.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:45.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:45.568+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:45.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:45.520+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:45.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:45.520+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:45.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:45.520+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:45.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:45.521+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:46.342 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:22:46.053+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 13863 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:22:46.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:46.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:46.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:46.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:46.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:46.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:46.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:46.491+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:46.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:46.588+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:46.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:46.588+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:46.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:46.588+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:46.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:46.589+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:46.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:46.556+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:46.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:46.556+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:46.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:46.556+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:46.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:46.556+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:47.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:47.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:47.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:47.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:47.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:47.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:47.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:47.456+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:47.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:47.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:47.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:47.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:47.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:47.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:47.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:47.572+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:47.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:47.533+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:47.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:47.533+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:47.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:47.533+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:47.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:47.534+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:48.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:48.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:48.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:48.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:48.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:48.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:48.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:48.429+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:48.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:48.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:48.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:48.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:48.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:48.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:48.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:48.551+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:48.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:48.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:48.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:48.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:48.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:48.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:48.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:48.519+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:49.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:49.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:49.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:49.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:49.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:49.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:49.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:49.402+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:49.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:49.542+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:49.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:49.542+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:49.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:49.542+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:49.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:49.543+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:49.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:49.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:49.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:49.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:49.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:49.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:49.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:49.495+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:50.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:50.529+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:50.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:50.529+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:50.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:50.529+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:50.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:50.530+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:50.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:50.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:50.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:50.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:50.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:50.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:50.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:50.419+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:50.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:50.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:50.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:50.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:50.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:50.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:50.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:50.471+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:51.387 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:22:51.054+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 13959 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:22:51.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:51.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:51.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:51.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:51.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:51.557+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:51.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:51.558+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:51.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:51.519+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:51.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:51.519+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:51.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:51.519+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:51.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:51.519+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:51.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:51.411+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:51.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:51.411+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:51.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:51.411+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:51.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:51.413+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:52.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:52.487+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:52.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:52.487+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:52.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:52.487+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:52.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:52.488+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:52.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:52.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:52.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:52.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:52.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:52.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:52.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:52.425+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:52.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:52.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:52.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:52.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:52.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:52.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:52.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:52.594+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:53.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:53.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:53.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:53.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:53.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:53.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:53.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:53.444+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:53.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:53.525+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:53.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:53.525+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:53.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:53.525+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:53.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:53.526+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:53.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:53.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:53.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:53.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:53.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:53.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:53.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:53.569+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:54.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:54.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:54.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:54.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:54.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:54.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:54.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:54.494+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:54.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:54.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:54.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:54.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:54.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:54.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:54.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:54.526+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:54.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:54.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:54.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:54.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:54.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:54.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:54.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:54.523+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:55.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:55.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:55.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:55.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:55.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:55.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:55.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:55.475+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:55.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:55.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:55.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:55.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:55.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:55.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:55.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:55.563+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:55.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:55.560+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:55.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:55.560+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:55.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:55.560+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:55.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:55.561+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:56.343 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:22:56.056+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14055 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:22:56.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:56.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:56.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:56.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:56.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:56.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:56.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:56.523+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:56.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:56.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:56.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:56.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:56.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:56.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:56.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:56.484+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:56.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:56.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:56.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:56.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:56.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:56.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:56.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:56.599+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:57.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:57.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:57.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:57.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:57.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:57.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:57.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:57.459+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:57.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:57.553+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:57.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:57.553+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:57.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:57.553+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:57.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:57.554+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:57.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:57.581+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:57.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:57.581+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:57.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:57.581+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:57.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:57.582+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:58.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:58.545+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:58.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:58.545+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:58.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:58.545+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:58.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:58.546+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:58.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:58.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:58.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:58.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:58.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:58.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:58.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:58.423+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:58.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:58.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:58.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:58.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:58.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:58.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:58.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:58.581+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:59.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:59.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:59.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:59.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:59.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:59.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:22:59.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:22:59.450+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:59.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:59.609+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:59.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:59.609+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:59.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:59.609+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:22:59.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:22:59.609+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:22:59.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:59.497+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:59.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:59.497+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:59.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:59.497+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:22:59.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:22:59.498+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:00.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:00.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:00.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:00.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:00.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:00.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:00.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:00.491+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:00.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:00.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:00.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:00.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:00.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:00.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:00.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:00.629+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:00.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:00.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:00.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:00.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:00.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:00.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:00.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:00.501+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:01.446 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:23:01.057+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14149 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:23:01.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:01.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:01.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:01.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:01.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:01.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:01.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:01.528+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:01.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:01.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:01.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:01.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:01.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:01.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:01.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:01.659+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:01.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:01.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:01.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:01.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:01.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:01.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:01.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:01.484+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:02.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:02.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:02.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:02.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:02.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:02.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:02.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:02.541+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:02.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:02.475+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:02.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:02.475+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:02.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:02.475+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:02.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:02.475+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:02.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:02.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:02.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:02.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:02.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:02.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:02.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:02.680+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:03.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:03.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:03.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:03.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:03.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:03.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:03.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:03.525+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:03.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:03.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:03.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:03.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:03.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:03.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:03.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:03.652+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:03.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:03.442+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:03.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:03.442+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:03.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:03.442+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:03.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:03.443+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:04.653 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:04.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:04.653 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:04.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:04.653 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:04.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:04.653 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:04.514+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:04.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:04.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:04.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:04.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:04.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:04.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:04.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:04.400+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:04.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:04.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:04.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:04.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:04.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:04.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:04.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:04.677+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:05.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:05.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:05.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:05.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:05.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:05.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:05.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:05.553+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:05.636 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:05.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:05.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:05.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:05.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:05.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:05.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:05.377+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:05.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:05.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:05.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:05.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:05.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:05.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:05.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:05.660+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:06.330 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:23:06.059+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14243 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:23:06.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:06.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:06.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:06.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:06.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:06.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:06.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:06.547+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:06.669 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:06.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:06.669 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:06.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:06.669 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:06.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:06.669 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:06.354+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:06.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:06.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:06.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:06.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:06.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:06.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:06.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:06.693+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:07.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:07.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:07.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:07.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:07.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:07.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:07.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:07.536+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:07.628 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:07.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:07.629 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:07.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:07.629 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:07.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:07.629 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:07.320+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:07.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:07.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:07.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:07.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:07.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:07.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:07.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:07.653+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:08.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:08.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:08.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:08.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:08.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:08.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:08.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:08.366+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:08.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:08.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:08.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:08.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:08.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:08.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:08.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:08.662+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:08.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:08.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:08.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:08.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:08.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:08.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:08.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:08.504+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:09.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:09.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:09.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:09.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:09.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:09.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:09.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:09.416+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:09.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:09.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:09.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:09.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:09.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:09.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:09.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:09.531+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:09.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:09.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:09.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:09.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:09.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:09.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:09.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:09.699+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:10.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:10.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:10.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:10.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:10.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:10.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:10.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:10.683+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:10.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:10.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:10.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:10.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:10.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:10.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:10.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:10.444+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:10.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:10.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:10.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:10.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:10.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:10.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:10.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:10.545+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:11.436 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:23:11.061+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14339 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:23:11.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:11.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:11.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:11.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:11.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:11.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:11.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:11.537+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:11.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:11.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:11.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:11.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:11.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:11.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:11.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:11.707+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:11.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:11.460+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:11.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:11.460+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:11.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:11.460+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:11.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:11.460+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:12.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:12.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:12.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:12.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:12.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:12.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:12.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:12.495+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:12.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:12.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:12.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:12.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:12.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:12.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:12.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:12.414+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:13.133 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:12.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:13.133 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:12.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:13.133 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:12.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:13.133 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:12.756+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:13.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:13.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:13.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:13.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:13.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:13.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:13.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:13.418+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:13.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:13.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:13.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:13.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:13.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:13.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:13.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:13.531+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:13.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:13.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:13.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:13.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:13.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:13.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:13.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:13.718+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:14.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:14.559+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:14.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:14.559+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:14.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:14.559+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:14.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:14.560+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:14.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:14.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:14.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:14.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:14.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:14.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:14.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:14.422+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:14.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:14.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:14.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:14.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:14.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:14.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:14.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:14.681+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:15.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:15.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:15.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:15.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:15.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:15.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:15.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:15.552+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:15.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:15.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:15.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:15.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:15.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:15.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:15.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:15.441+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:16.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:15.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:16.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:15.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:16.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:15.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:16.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:15.727+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:16.343 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:23:16.062+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14435 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:23:16.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:16.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:16.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:16.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:16.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:16.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:16.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:16.566+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:16.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:16.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:16.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:16.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:16.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:16.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:16.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:16.422+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:16.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:16.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:16.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:16.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:16.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:16.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:16.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:16.718+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:17.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:17.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:17.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:17.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:17.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:17.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:17.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:17.602+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:17.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:17.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:17.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:17.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:17.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:17.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:17.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:17.398+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:18.173 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:17.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:18.173 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:17.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:18.174 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:17.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:18.174 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:17.760+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:18.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:18.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:18.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:18.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:18.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:18.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:18.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:18.389+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:18.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:18.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:18.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:18.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:18.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:18.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:18.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:18.568+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:18.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:18.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:18.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:18.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:18.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:18.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:18.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:18.752+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:19.577 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:19.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:19.577 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:19.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:19.577 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:19.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:19.577 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:19.341+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:19.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:19.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:19.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:19.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:19.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:19.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:19.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:19.602+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:19.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:19.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:19.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:19.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:19.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:19.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:19.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:19.753+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:20.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:20.333+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:20.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:20.333+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:20.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:20.333+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:20.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:20.333+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:20.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:20.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:20.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:20.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:20.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:20.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:20.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:20.745+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:20.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:20.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:20.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:20.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:20.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:20.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:20.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:20.596+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:21.446 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:23:21.064+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14531 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:23:21.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:21.306+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:21.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:21.306+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:21.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:21.306+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:21.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:21.306+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:21.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:21.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:21.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:21.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:21.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:21.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:21.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:21.611+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:22.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:21.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:22.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:21.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:22.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:21.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:22.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:21.742+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:22.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:22.280+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:22.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:22.280+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:22.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:22.280+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:22.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:22.280+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:22.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:22.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:22.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:22.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:22.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:22.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:22.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:22.608+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:22.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:22.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:22.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:22.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:22.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:22.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:22.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:22.726+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:23.554 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:23.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:23.554 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:23.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:23.554 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:23.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:23.554 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:23.287+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:23.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:23.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:23.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:23.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:23.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:23.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:23.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:23.610+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:23.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:23.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:23.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:23.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:23.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:23.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:23.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:23.754+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:24.616 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:24.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:24.616 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:24.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:24.616 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:24.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:24.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:24.331+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:24.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:24.709+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:24.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:24.709+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:24.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:24.709+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:24.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:24.710+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:24.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:24.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:24.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:24.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:24.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:24.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:24.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:24.641+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:25.641 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:25.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:25.641 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:25.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:25.641 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:25.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:25.641 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:25.368+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:25.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:25.665+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:25.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:25.665+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:25.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:25.665+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:25.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:25.666+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:25.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:25.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:25.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:25.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:25.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:25.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:25.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:25.665+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:26.343 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:23:26.065+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14620 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:23:26.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:26.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:26.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:26.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:26.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:26.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:26.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:26.403+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:26.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:26.660+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:26.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:26.660+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:26.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:26.660+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:26.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:26.661+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:26.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:26.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:26.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:26.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:26.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:26.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:26.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:26.632+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:27.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:27.692+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:27.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:27.692+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:27.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:27.692+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:27.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:27.693+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:27.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:27.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:27.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:27.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:27.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:27.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:27.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:27.599+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:27.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:27.452+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:27.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:27.452+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:27.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:27.452+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:27.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:27.453+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:28.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:28.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:28.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:28.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:28.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:28.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:28.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:28.446+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:28.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:28.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:28.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:28.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:28.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:28.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:28.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:28.659+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:28.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:28.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:28.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:28.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:28.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:28.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:28.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:28.632+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:29.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:29.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:29.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:29.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:29.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:29.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:29.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:29.627+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:29.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:29.421+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:29.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:29.421+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:29.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:29.421+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:29.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:29.422+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:29.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:29.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:29.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:29.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:29.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:29.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:29.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:29.587+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:30.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:30.453+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:30.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:30.453+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:30.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:30.453+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:30.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:30.453+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:30.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:30.677+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:30.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:30.677+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:30.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:30.677+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:30.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:30.678+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:30.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:30.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:30.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:30.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:30.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:30.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:30.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:30.635+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:31.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:23:31.067+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14709 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:23:31.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:31.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:31.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:31.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:31.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:31.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:31.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:31.414+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:31.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:31.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:31.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:31.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:31.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:31.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:31.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:31.719+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:31.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:31.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:31.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:31.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:31.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:31.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:31.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:31.638+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:32.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:32.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:32.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:32.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:32.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:32.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:32.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:32.366+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:32.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:32.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:32.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:32.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:32.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:32.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:32.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:32.713+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:32.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:32.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:32.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:32.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:32.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:32.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:32.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:32.684+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:33.554 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:33.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:33.555 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:33.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:33.555 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:33.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:33.555 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:33.393+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:33.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:33.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:33.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:33.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:33.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:33.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:33.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:33.713+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:33.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:33.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:33.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:33.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:33.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:33.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:33.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:33.716+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:34.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:34.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:34.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:34.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:34.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:34.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:34.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:34.393+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:34.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:34.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:34.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:34.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:34.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:34.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:34.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:34.684+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:34.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:34.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:34.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:34.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:34.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:34.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:34.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:34.714+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:35.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:35.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:35.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:35.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:35.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:35.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:35.636 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:35.375+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:35.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:35.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:35.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:35.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:35.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:35.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:35.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:35.666+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:35.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:35.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:35.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:35.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:35.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:35.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:35.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:35.659+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:36.343 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:23:36.068+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14799 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:23:36.638 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:36.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:36.639 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:36.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:36.639 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:36.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:36.639 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:36.375+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:36.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:36.679+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:36.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:36.679+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:36.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:36.679+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:36.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:36.680+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:36.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:36.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:36.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:36.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:36.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:36.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:36.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:36.662+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:37.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:37.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:37.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:37.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:37.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:37.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:37.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:37.647+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:37.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:37.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:37.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:37.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:37.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:37.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:37.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:37.423+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:37.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:37.692+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:37.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:37.692+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:37.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:37.692+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:37.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:37.693+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:38.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:38.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:38.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:38.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:38.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:38.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:38.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:38.419+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:38.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:38.714+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:38.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:38.714+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:38.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:38.714+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:38.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:38.715+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:38.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:38.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:38.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:38.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:38.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:38.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:38.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:38.659+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:39.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:39.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:39.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:39.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:39.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:39.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:39.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:39.680+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:39.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:39.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:39.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:39.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:39.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:39.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:39.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:39.455+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:40.132 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:39.749+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:40.132 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:39.749+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:40.132 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:39.749+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:40.132 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:39.750+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:40.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:40.724+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:40.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:40.724+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:40.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:40.724+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:40.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:40.725+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:40.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:40.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:40.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:40.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:40.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:40.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:40.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:40.683+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:40.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:40.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:40.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:40.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:40.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:40.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:40.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:40.495+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:41.442 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:23:41.070+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14889 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:23:41.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:41.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:41.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:41.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:41.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:41.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:41.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:41.465+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:41.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:41.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:41.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:41.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:41.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:41.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:41.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:41.669+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:42.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:41.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:42.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:41.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:42.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:41.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:42.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:41.732+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:42.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:42.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:42.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:42.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:42.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:42.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:42.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:42.635+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:42.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:42.749+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:42.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:42.749+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:42.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:42.749+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:42.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:42.749+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:42.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:42.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:42.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:42.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:42.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:42.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:42.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:42.500+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:43.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:43.458+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:43.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:43.458+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:43.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:43.458+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:43.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:43.458+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:43.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:43.771+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:43.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:43.771+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:43.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:43.771+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:43.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:43.771+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:43.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:43.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:43.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:43.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:43.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:43.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:43.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:43.685+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:44.772 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:44.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:44.773 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:44.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:44.773 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:44.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:44.773 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:44.499+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:44.773 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:44.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:44.773 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:44.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:44.773 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:44.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:44.773 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:44.687+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:45.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:44.796+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:45.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:44.796+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:45.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:44.796+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:45.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:44.797+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:45.802 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:45.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:45.802 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:45.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:45.802 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:45.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:45.802 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:45.518+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:45.802 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:45.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:45.803 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:45.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:45.803 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:45.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:45.803 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:45.643+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:46.060 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:23:46.072+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 14978 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:23:46.061 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:45.826+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:46.061 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:45.826+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:46.061 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:45.826+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:46.061 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:45.827+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:46.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:46.482+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:46.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:46.482+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:46.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:46.482+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:46.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:46.482+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:46.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:46.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:46.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:46.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:46.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:46.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:46.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:46.683+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:46.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:46.796+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:46.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:46.796+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:46.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:46.796+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:46.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:46.796+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:47.803 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:47.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:47.803 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:47.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:47.803 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:47.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:47.803 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:47.508+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:47.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:47.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:47.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:47.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:47.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:47.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:47.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:47.723+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:48.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:47.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:48.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:47.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:48.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:47.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:48.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:47.828+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:48.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:48.472+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:48.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:48.472+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:48.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:48.472+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:48.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:48.472+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:48.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:48.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:48.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:48.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:48.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:48.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:48.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:48.874+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:48.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:48.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:48.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:48.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:48.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:48.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:48.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:48.684+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:49.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:49.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:49.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:49.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:49.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:49.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:49.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:49.437+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:49.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:49.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:49.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:49.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:49.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:49.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:49.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:49.668+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:50.133 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:49.834+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:50.133 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:49.834+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:50.133 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:49.834+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:50.133 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:49.835+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:50.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:50.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:50.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:50.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:50.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:50.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:50.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:50.633+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:50.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:50.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:50.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:50.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:50.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:50.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:50.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:50.426+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:51.060 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:23:51.074+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15068 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:23:51.061 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:50.825+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:51.061 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:50.825+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:51.061 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:50.825+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:51.061 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:50.826+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:51.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:51.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:51.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:51.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:51.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:51.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:51.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:51.587+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:51.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:51.442+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:51.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:51.442+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:51.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:51.442+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:51.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:51.442+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:52.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:51.807+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:52.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:51.807+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:52.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:51.807+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:52.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:51.808+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:52.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:52.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:52.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:52.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:52.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:52.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:52.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:52.635+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:52.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:52.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:52.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:52.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:52.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:52.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:52.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:52.419+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:53.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:52.828+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:53.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:52.828+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:53.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:52.828+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:53.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:52.829+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:53.631 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:53.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:53.631 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:53.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:53.631 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:53.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:53.631 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:53.387+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:53.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:53.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:53.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:53.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:53.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:53.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:53.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:53.656+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:53.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:53.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:53.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:53.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:53.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:53.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:53.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:53.837+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:54.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:54.360+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:54.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:54.360+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:54.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:54.360+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:54.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:54.361+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:54.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:54.842+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:54.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:54.842+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:54.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:54.842+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:54.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:54.843+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:54.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:54.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:54.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:54.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:54.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:54.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:54.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:54.658+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:55.595 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:55.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:55.596 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:55.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:55.596 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:55.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:55.596 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:55.359+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:55.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:55.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:55.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:55.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:55.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:55.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:55.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:55.620+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:56.179 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:23:56.075+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15158 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:23:56.179 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:55.872+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:56.179 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:55.872+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:56.179 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:55.872+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:56.179 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:55.873+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:56.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:56.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:56.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:56.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:56.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:56.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:56.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:56.385+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:56.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:56.641+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:56.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:56.641+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:56.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:56.641+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:56.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:56.642+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:56.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:56.852+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:56.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:56.852+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:56.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:56.852+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:56.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:56.853+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:57.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:57.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:57.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:57.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:57.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:57.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:57.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:57.433+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:57.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:57.656+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:57.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:57.656+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:57.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:57.656+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:57.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:57.657+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:58.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:57.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:58.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:57.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:58.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:57.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:58.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:57.873+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:58.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:58.448+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:58.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:58.448+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:58.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:58.448+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:58.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:58.449+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:58.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:58.687+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:58.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:58.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:58.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:58.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:58.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:58.688+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:59.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:58.831+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:59.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:58.831+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:59.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:58.831+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:23:59.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:58.832+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:59.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:59.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:59.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:59.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:59.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:59.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:23:59.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:23:59.674+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:23:59.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:59.454+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:59.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:59.454+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:59.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:59.454+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:23:59.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:23:59.454+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:00.134 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:59.880+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:00.134 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:59.880+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:00.134 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:59.880+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:00.134 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:23:59.881+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:00.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:00.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:00.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:00.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:00.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:00.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:00.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:00.426+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:00.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:00.898+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:00.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:00.898+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:00.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:00.898+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:00.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:00.898+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:00.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:00.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:00.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:00.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:00.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:00.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:00.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:00.692+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:01.427 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:24:01.077+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15248 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:24:01.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:01.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:01.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:01.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:01.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:01.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:01.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:01.452+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:01.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:01.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:01.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:01.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:01.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:01.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:01.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:01.672+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:02.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:01.892+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:02.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:01.892+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:02.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:01.892+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:02.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:01.893+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:02.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:02.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:02.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:02.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:02.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:02.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:02.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:02.629+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:02.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:02.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:02.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:02.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:02.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:02.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:02.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:02.424+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:03.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:02.904+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:03.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:02.904+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:03.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:02.904+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:03.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:02.905+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:03.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:03.635+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:03.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:03.635+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:03.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:03.635+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:03.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:03.636+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:03.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:03.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:03.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:03.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:03.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:03.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:03.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:03.452+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:04.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:03.924+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:04.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:03.924+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:04.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:03.924+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:04.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:03.925+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:04.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:04.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:04.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:04.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:04.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:04.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:04.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:04.620+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:04.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:04.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:04.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:04.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:04.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:04.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:04.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:04.417+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:05.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:04.905+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:05.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:04.905+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:05.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:04.905+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:05.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:04.905+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:05.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:05.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:05.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:05.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:05.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:05.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:05.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:05.575+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:05.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:05.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:05.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:05.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:05.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:05.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:05.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:05.396+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:06.179 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:24:06.079+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15337 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:24:06.179 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:05.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:06.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:05.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:06.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:05.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:06.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:05.948+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:06.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:06.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:06.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:06.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:06.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:06.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:06.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:06.392+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:06.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:06.539+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:06.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:06.539+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:06.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:06.539+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:06.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:06.540+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:07.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:06.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:07.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:06.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:07.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:06.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:07.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:06.923+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:07.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:07.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:07.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:07.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:07.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:07.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:07.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:07.568+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:07.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:07.403+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:07.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:07.403+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:07.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:07.403+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:07.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:07.403+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:08.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:07.910+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:08.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:07.910+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:08.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:07.910+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:08.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:07.911+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:08.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:08.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:08.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:08.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:08.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:08.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:08.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:08.540+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:08.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:08.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:08.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:08.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:08.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:08.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:08.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:08.354+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:09.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:08.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:09.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:08.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:09.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:08.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:09.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:08.942+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:09.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:09.510+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:09.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:09.510+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:09.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:09.510+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:09.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:09.511+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:09.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:09.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:09.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:09.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:09.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:09.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:09.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:09.390+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:10.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:09.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:10.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:09.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:10.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:09.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:10.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:09.974+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:10.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:10.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:10.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:10.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:10.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:10.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:10.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:10.545+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:10.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:10.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:10.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:10.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:10.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:10.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:10.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:10.426+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:11.183 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:24:11.080+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15427 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:24:11.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:10.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:11.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:10.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:11.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:10.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:11.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:10.946+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:11.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:11.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:11.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:11.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:11.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:11.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:11.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:11.416+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:11.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:11.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:11.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:11.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:11.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:11.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:11.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:11.534+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:12.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:11.963+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:12.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:11.963+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:12.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:11.963+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:12.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:11.964+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:12.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:12.458+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:12.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:12.458+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:12.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:12.458+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:12.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:12.458+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:12.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:12.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:12.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:12.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:12.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:12.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:12.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:12.489+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:13.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:12.987+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:13.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:12.987+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:13.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:12.987+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:13.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:12.988+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:13.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:13.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:13.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:13.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:13.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:13.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:13.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:13.484+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:13.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:13.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:13.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:13.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:13.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:13.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:13.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:13.410+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:14.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:14.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:14.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:14.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:14.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:14.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:14.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:14.013+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:14.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:14.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:14.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:14.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:14.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:14.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:14.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:14.471+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:14.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:14.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:14.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:14.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:14.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:14.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:14.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:14.370+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:15.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:15.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:15.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:15.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:15.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:15.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:15.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:15.053+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:15.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:15.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:15.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:15.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:15.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:15.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:15.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:15.364+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:15.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:15.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:15.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:15.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:15.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:15.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:15.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:15.462+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:16.343 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:24:16.082+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15517 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:24:16.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:16.038+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:16.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:16.038+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:16.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:16.038+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:16.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:16.038+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:16.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:16.398+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:16.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:16.398+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:16.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:16.398+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:16.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:16.398+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:16.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:16.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:16.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:16.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:16.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:16.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:16.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:16.449+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:17.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:17.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:17.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:17.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:17.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:17.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:17.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:17.082+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:17.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:17.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:17.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:17.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:17.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:17.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:17.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:17.457+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:17.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:17.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:17.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:17.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:17.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:17.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:17.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:17.436+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:18.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:18.032+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:18.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:18.032+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:18.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:18.032+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:18.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:18.034+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:18.650 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:18.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:18.650 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:18.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:18.651 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:18.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:18.651 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:18.492+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:18.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:18.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:18.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:18.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:18.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:18.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:18.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:18.402+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:19.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:19.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:19.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:19.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:19.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:19.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:19.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:19.001+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:19.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:19.443+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:19.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:19.443+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:19.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:19.443+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:19.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:19.444+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:19.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:19.401+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:19.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:19.401+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:19.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:19.401+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:19.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:19.401+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:20.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:19.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:20.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:19.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:20.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:19.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:20.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:19.960+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:20.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:20.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:20.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:20.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:20.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:20.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:20.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:20.408+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:20.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:20.431+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:20.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:20.431+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:20.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:20.431+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:20.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:20.431+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:21.367 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:24:21.084+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15607 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:24:21.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:21.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:21.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:21.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:21.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:21.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:21.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:21.008+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:21.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:21.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:21.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:21.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:21.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:21.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:21.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:21.401+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:21.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:21.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:21.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:21.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:21.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:21.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:21.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:21.392+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:22.358 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:21.994+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:22.358 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:21.994+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:22.358 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:21.994+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:22.358 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:21.995+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:22.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:22.381+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:22.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:22.381+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:22.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:22.381+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:22.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:22.382+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:22.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:22.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:22.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:22.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:22.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:22.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:22.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:22.425+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:23.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:22.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:23.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:22.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:23.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:22.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:23.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:22.967+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:23.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:23.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:23.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:23.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:23.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:23.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:23.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:23.408+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:23.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:23.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:23.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:23.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:23.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:23.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:23.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:23.406+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:24.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:24.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:24.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:24.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:24.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:24.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:24.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:24.011+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:24.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:24.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:24.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:24.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:24.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:24.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:24.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:24.370+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:24.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:24.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:24.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:24.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:24.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:24.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:24.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:24.372+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:25.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:24.980+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:25.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:24.980+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:25.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:24.980+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:25.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:24.981+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:25.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:25.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:25.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:25.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:25.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:25.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:25.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:25.385+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:25.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:25.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:25.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:25.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:25.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:25.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:25.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:25.325+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:26.213 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:24:26.086+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15692 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:24:26.213 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:25.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:26.213 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:25.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:26.213 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:25.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:26.213 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:25.987+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:26.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:26.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:26.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:26.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:26.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:26.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:26.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:26.434+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:26.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:26.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:26.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:26.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:26.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:26.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:26.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:26.352+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:27.282 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:27.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:27.282 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:27.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:27.282 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:27.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:27.282 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:27.004+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:27.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:27.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:27.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:27.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:27.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:27.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:27.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:27.453+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:27.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:27.306+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:27.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:27.306+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:27.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:27.306+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:27.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:27.306+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:28.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:28.028+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:28.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:28.028+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:28.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:28.028+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:28.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:28.029+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:28.467 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:28.323+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:28.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:28.323+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:28.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:28.323+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:28.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:28.324+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:28.734 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:28.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:28.734 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:28.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:28.734 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:28.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:28.734 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:28.492+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:29.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:29.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:29.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:29.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:29.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:29.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:29.339 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:29.009+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:29.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:29.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:29.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:29.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:29.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:29.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:29.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:29.362+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:29.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:29.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:29.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:29.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:29.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:29.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:29.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:29.444+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:30.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:30.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:30.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:30.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:30.382 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:30.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:30.382 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:30.047+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:30.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:30.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:30.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:30.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:30.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:30.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:30.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:30.406+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:30.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:30.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:30.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:30.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:30.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:30.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:30.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:30.438+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:31.341 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:24:31.087+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15780 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:24:31.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:31.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:31.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:31.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:31.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:31.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:31.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:31.025+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:31.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:31.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:31.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:31.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:31.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:31.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:31.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:31.365+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:31.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:31.428+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:31.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:31.428+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:31.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:31.428+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:31.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:31.429+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:32.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:32.064+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:32.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:32.064+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:32.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:32.064+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:32.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:32.065+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:32.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:32.426+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:32.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:32.426+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:32.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:32.426+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:32.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:32.427+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:32.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:32.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:32.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:32.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:32.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:32.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:32.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:32.356+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:33.297 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:33.015+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:33.297 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:33.015+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:33.297 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:33.015+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:33.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:33.017+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:33.559 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:33.386+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:33.559 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:33.386+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:33.559 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:33.386+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:33.559 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:33.386+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:33.559 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:33.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:33.559 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:33.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:33.559 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:33.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:33.559 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:33.321+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:34.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:34.064+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:34.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:34.064+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:34.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:34.064+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:34.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:34.065+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:34.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:34.380+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:34.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:34.380+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:34.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:34.380+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:34.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:34.381+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:34.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:34.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:34.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:34.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:34.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:34.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:34.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:34.326+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:35.397 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:35.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:35.397 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:35.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:35.397 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:35.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:35.397 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:35.112+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:35.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:35.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:35.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:35.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:35.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:35.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:35.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:35.348+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:35.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:35.419+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:35.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:35.419+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:35.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:35.419+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:35.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:35.420+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:36.319 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:24:36.088+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15868 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:24:36.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:36.158+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:36.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:36.158+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:36.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:36.158+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:36.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:36.159+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:36.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:36.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:36.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:36.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:36.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:36.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:36.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:36.464+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:36.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:36.342+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:36.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:36.342+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:36.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:36.342+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:36.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:36.342+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:37.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:37.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:37.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:37.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:37.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:37.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:37.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:37.379+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:37.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:37.182+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:37.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:37.182+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:37.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:37.182+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:37.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:37.183+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:37.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:37.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:37.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:37.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:37.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:37.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:37.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:37.483+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:38.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:38.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:38.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:38.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:38.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:38.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:38.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:38.376+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:38.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:38.187+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:38.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:38.187+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:38.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:38.187+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:38.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:38.188+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:38.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:38.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:38.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:38.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:38.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:38.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:38.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:38.521+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:39.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:39.181+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:39.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:39.181+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:39.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:39.181+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:39.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:39.182+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:39.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:39.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:39.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:39.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:39.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:39.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:39.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:39.338+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:39.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:39.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:39.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:39.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:39.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:39.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:39.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:39.533+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:40.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:40.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:40.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:40.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:40.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:40.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:40.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:40.165+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:40.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:40.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:40.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:40.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:40.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:40.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:40.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:40.335+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:40.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:40.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:40.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:40.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:40.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:40.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:40.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:40.491+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:41.347 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:24:41.090+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 15957 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:24:41.347 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:41.144+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:41.347 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:41.144+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:41.347 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:41.144+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:41.347 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:41.145+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:41.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:41.505+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:41.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:41.505+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:41.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:41.505+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:41.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:41.505+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:41.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:41.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:41.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:41.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:41.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:41.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:41.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:41.371+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:42.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:42.134+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:42.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:42.134+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:42.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:42.134+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:42.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:42.134+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:42.662 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:42.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:42.662 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:42.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:42.662 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:42.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:42.662 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:42.391+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:42.662 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:42.469+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:42.662 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:42.469+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:42.662 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:42.469+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:42.662 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:42.469+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:43.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:43.162+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:43.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:43.162+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:43.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:43.162+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:43.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:43.163+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:43.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:43.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:43.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:43.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:43.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:43.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:43.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:43.379+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:43.862 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:43.477+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:43.863 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:43.477+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:43.863 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:43.477+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:43.863 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:43.478+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:44.430 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:44.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:44.431 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:44.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:44.431 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:44.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:44.431 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:44.354+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:44.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:44.198+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:44.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:44.198+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:44.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:44.198+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:44.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:44.199+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:44.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:44.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:44.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:44.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:44.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:44.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:44.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:44.455+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:45.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:45.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:45.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:45.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:45.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:45.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:45.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:45.459+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:45.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:45.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:45.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:45.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:45.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:45.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:45.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:45.338+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:45.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:45.240+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:45.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:45.240+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:45.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:45.240+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:45.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:45.241+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:46.342 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:24:46.092+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16045 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:24:46.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:46.298+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:46.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:46.298+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:46.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:46.298+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:46.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:46.298+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:46.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:46.290+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:46.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:46.290+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:46.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:46.290+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:46.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:46.291+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:46.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:46.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:46.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:46.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:46.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:46.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:46.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:46.483+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:47.498 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:47.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:47.498 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:47.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:47.498 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:47.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:47.498 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:47.337+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:47.499 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:47.251+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:47.499 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:47.251+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:47.499 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:47.251+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:47.499 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:47.251+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:47.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:47.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:47.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:47.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:47.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:47.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:47.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:47.522+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:48.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:48.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:48.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:48.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:48.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:48.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:48.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:48.287+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:48.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:48.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:48.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:48.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:48.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:48.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:48.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:48.514+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:48.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:48.318+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:48.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:48.318+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:48.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:48.318+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:48.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:48.319+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:49.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:49.327+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:49.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:49.327+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:49.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:49.327+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:49.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:49.327+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:49.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:49.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:49.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:49.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:49.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:49.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:49.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:49.495+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:49.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:49.271+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:49.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:49.271+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:49.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:49.271+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:49.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:49.272+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:50.422 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:50.327+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:50.422 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:50.327+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:50.422 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:50.327+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:50.422 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:50.327+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:50.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:50.278+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:50.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:50.278+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:50.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:50.278+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:50.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:50.279+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:50.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:50.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:50.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:50.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:50.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:50.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:50.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:50.447+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:51.399 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:24:51.094+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16134 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:24:51.399 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:51.329+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:51.399 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:51.329+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:51.399 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:51.329+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:51.399 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:51.329+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:51.400 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:51.270+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:51.400 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:51.270+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:51.400 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:51.270+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:51.400 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:51.271+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:51.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:51.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:51.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:51.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:51.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:51.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:51.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:51.424+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:52.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:52.313+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:52.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:52.313+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:52.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:52.313+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:52.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:52.314+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:52.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:52.405+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:52.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:52.406+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:52.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:52.406+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:52.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:52.407+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:52.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:52.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:52.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:52.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:52.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:52.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:52.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:52.368+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:53.560 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:53.296+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:53.560 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:53.296+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:53.560 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:53.296+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:53.560 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:53.297+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:53.561 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:53.417+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:53.561 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:53.417+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:53.561 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:53.417+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:53.561 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:53.418+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:53.561 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:53.390+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:53.561 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:53.390+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:53.561 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:53.390+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:53.562 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:53.391+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:54.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:54.249+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:54.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:54.249+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:54.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:54.249+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:54.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:54.249+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:54.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:54.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:54.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:54.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:54.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:54.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:54.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:54.412+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:54.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:54.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:54.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:54.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:54.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:54.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:54.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:54.456+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:55.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:55.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:55.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:55.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:55.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:55.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:55.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:55.205+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:55.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:55.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:55.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:55.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:55.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:55.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:55.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:55.407+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:55.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:55.460+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:55.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:55.460+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:55.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:55.460+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:55.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:55.461+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:56.342 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:24:56.095+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16223 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:24:56.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:56.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:56.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:56.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:56.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:56.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:56.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:56.187+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:56.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:56.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:56.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:56.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:56.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:56.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:56.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:56.384+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:56.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:56.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:56.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:56.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:56.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:56.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:56.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:56.466+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:57.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:57.199+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:57.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:57.199+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:57.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:57.199+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:57.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:57.200+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:57.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:57.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:57.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:57.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:57.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:57.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:57.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:57.501+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:57.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:57.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:57.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:57.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:57.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:57.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:57.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:57.380+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:58.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:58.177+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:58.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:58.177+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:58.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:58.177+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:58.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:58.177+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:58.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:58.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:58.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:58.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:58.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:58.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:58.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:58.550+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:58.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:58.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:58.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:58.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:58.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:58.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:58.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:58.365+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:59.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:59.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:59.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:59.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:59.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:59.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:24:59.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:24:59.129+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:59.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:59.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:59.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:59.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:59.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:59.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:24:59.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:24:59.593+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:24:59.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:59.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:59.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:59.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:59.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:59.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:24:59.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:24:59.392+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:00.363 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:00.092+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:00.363 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:00.092+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:00.364 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:00.092+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:00.364 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:00.092+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:00.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:00.565+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:00.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:00.565+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:00.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:00.565+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:00.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:00.566+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:00.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:00.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:00.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:00.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:00.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:00.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:00.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:00.388+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:01.345 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:25:01.097+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16312 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:25:01.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:01.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:01.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:01.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:01.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:01.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:01.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:01.086+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:01.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:01.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:01.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:01.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:01.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:01.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:01.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:01.538+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:01.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:01.370+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:01.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:01.370+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:01.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:01.370+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:01.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:01.370+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:02.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:02.063+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:02.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:02.063+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:02.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:02.063+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:02.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:02.064+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:02.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:02.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:02.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:02.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:02.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:02.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:02.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:02.573+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:02.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:02.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:02.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:02.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:02.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:02.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:02.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:02.327+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:03.327 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:03.083+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:03.328 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:03.083+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:03.328 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:03.083+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:03.328 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:03.084+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:03.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:03.588+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:03.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:03.588+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:03.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:03.588+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:03.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:03.588+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:03.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:03.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:03.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:03.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:03.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:03.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:03.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:03.352+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:04.329 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:04.077+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:04.329 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:04.077+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:04.329 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:04.077+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:04.329 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:04.077+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:04.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:04.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:04.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:04.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:04.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:04.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:04.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:04.552+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:04.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:04.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:04.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:04.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:04.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:04.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:04.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:04.353+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:05.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:05.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:05.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:05.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:05.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:05.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:05.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:05.117+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:05.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:05.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:05.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:05.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:05.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:05.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:05.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:05.369+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:05.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:05.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:05.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:05.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:05.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:05.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:05.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:05.597+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:06.342 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:25:06.099+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16398 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:25:06.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:06.130+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:06.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:06.130+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:06.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:06.130+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:06.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:06.131+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:06.605 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:06.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:06.605 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:06.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:06.605 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:06.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:06.605 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:06.391+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:06.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:06.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:06.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:06.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:06.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:06.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:06.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:06.629+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:07.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:07.097+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:07.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:07.097+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:07.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:07.097+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:07.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:07.098+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:07.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:07.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:07.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:07.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:07.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:07.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:07.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:07.595+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:07.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:07.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:07.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:07.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:07.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:07.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:07.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:07.376+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:08.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:08.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:08.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:08.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:08.354 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:08.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:08.354 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:08.054+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:08.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:08.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:08.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:08.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:08.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:08.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:08.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:08.570+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:08.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:08.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:08.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:08.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:08.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:08.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:08.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:08.378+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:09.348 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:09.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:09.348 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:09.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:09.349 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:09.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:09.349 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:09.089+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:09.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:09.542+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:09.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:09.542+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:09.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:09.542+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:09.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:09.543+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:09.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:09.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:09.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:09.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:09.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:09.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:09.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:09.373+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:10.354 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:10.112+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:10.354 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:10.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:10.354 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:10.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:10.354 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:10.113+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:10.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:10.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:10.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:10.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:10.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:10.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:10.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:10.547+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:10.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:10.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:10.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:10.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:10.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:10.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:10.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:10.378+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:11.395 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:25:11.101+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16485 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:25:11.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:11.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:11.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:11.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:11.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:11.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:11.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:11.155+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:11.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:11.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:11.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:11.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:11.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:11.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:11.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:11.559+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:11.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:11.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:11.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:11.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:11.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:11.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:11.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:11.420+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:12.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:12.191+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:12.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:12.191+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:12.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:12.191+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:12.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:12.192+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:12.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:12.437+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:12.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:12.437+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:12.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:12.437+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:12.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:12.438+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:12.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:12.605+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:12.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:12.605+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:12.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:12.605+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:12.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:12.605+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:13.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:13.205+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:13.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:13.205+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:13.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:13.205+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:13.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:13.206+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:13.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:13.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:13.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:13.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:13.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:13.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:13.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:13.420+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:13.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:13.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:13.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:13.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:13.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:13.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:13.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:13.560+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:14.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:14.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:14.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:14.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:14.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:14.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:14.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:14.194+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:14.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:14.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:14.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:14.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:14.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:14.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:14.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:14.436+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:14.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:14.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:14.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:14.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:14.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:14.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:14.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:14.575+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:15.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:15.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:15.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:15.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:15.405 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:15.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:15.405 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:15.160+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:15.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:15.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:15.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:15.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:15.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:15.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:15.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:15.429+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:15.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:15.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:15.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:15.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:15.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:15.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:15.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:15.584+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:16.343 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:25:16.102+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16572 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:25:16.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:16.142+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:16.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:16.142+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:16.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:16.142+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:16.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:16.143+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:16.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:16.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:16.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:16.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:16.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:16.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:16.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:16.594+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:16.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:16.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:16.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:16.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:16.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:16.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:16.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:16.445+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:17.386 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:17.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:17.386 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:17.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:17.386 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:17.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:17.386 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:17.151+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:17.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:17.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:17.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:17.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:17.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:17.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:17.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:17.559+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:17.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:17.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:17.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:17.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:17.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:17.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:17.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:17.410+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:18.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:18.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:18.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:18.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:18.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:18.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:18.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:18.111+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:18.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:18.440+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:18.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:18.440+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:18.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:18.440+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:18.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:18.440+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:18.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:18.538+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:18.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:18.538+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:18.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:18.538+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:18.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:18.539+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:19.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:19.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:19.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:19.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:19.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:19.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:19.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:19.112+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:19.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:19.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:19.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:19.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:19.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:19.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:19.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:19.483+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:19.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:19.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:19.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:19.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:19.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:19.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:19.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:19.535+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:20.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:20.107+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:20.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:20.107+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:20.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:20.107+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:20.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:20.108+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:20.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:20.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:20.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:20.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:20.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:20.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:20.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:20.513+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:20.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:20.457+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:20.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:20.457+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:20.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:20.457+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:20.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:20.458+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:21.400 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:25:21.104+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16660 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:25:21.401 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:21.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:21.401 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:21.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:21.401 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:21.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:21.401 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:21.104+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:21.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:21.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:21.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:21.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:21.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:21.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:21.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:21.536+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:21.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:21.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:21.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:21.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:21.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:21.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:21.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:21.426+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:22.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:22.099+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:22.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:22.099+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:22.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:22.099+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:22.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:22.100+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:22.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:22.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:22.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:22.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:22.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:22.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:22.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:22.433+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:22.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:22.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:22.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:22.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:22.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:22.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:22.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:22.561+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:23.384 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:23.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:23.384 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:23.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:23.384 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:23.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:23.384 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:23.055+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:23.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:23.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:23.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:23.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:23.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:23.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:23.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:23.409+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:23.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:23.570+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:23.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:23.570+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:23.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:23.570+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:23.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:23.572+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:24.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:24.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:24.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:24.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:24.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:24.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:24.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:24.054+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:24.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:24.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:24.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:24.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:24.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:24.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:24.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:24.367+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:24.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:24.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:24.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:24.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:24.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:24.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:24.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:24.606+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:25.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:25.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:25.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:25.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:25.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:25.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:25.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:25.027+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:25.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:25.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:25.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:25.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:25.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:25.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:25.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:25.383+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:25.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:25.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:25.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:25.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:25.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:25.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:25.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:25.600+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:26.342 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:25:26.106+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16747 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:25:26.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:26.006+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:26.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:26.006+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:26.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:26.006+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:26.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:26.006+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:26.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:26.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:26.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:26.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:26.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:26.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:26.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:26.608+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:26.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:26.421+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:26.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:26.421+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:26.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:26.421+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:26.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:26.421+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:27.365 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:27.030+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:27.365 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:27.030+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:27.365 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:27.030+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:27.366 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:27.030+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:27.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:27.603+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:27.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:27.603+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:27.654 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:27.603+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:27.654 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:27.604+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:27.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:27.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:27.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:27.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:27.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:27.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:27.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:27.390+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:28.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:28.038+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:28.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:28.038+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:28.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:28.038+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:28.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:28.038+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:28.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:28.377+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:28.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:28.377+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:28.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:28.377+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:28.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:28.377+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:28.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:28.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:28.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:28.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:28.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:28.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:28.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:28.615+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:29.308 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:29.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:29.308 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:29.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:29.308 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:29.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:29.308 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:29.062+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:29.605 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:29.332+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:29.605 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:29.332+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:29.605 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:29.332+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:29.605 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:29.333+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:29.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:29.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:29.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:29.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:29.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:29.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:29.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:29.629+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:30.355 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:30.023+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:30.355 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:30.023+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:30.355 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:30.023+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:30.355 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:30.023+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:30.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:30.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:30.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:30.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:30.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:30.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:30.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:30.379+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:30.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:30.600+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:30.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:30.600+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:30.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:30.600+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:30.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:30.601+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:31.388 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:25:31.107+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16835 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:25:31.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:31.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:31.389 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:31.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:31.389 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:31.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:31.389 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:31.051+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:31.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:31.411+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:31.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:31.411+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:31.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:31.411+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:31.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:31.412+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:31.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:31.585+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:31.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:31.585+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:31.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:31.585+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:31.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:31.586+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:32.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:32.042+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:32.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:32.042+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:32.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:32.042+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:32.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:32.043+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:32.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:32.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:32.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:32.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:32.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:32.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:32.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:32.603+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:32.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:32.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:32.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:32.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:32.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:32.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:32.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:32.433+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:33.399 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:33.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:33.399 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:33.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:33.399 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:33.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:33.399 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:33.003+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:33.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:33.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:33.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:33.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:33.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:33.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:33.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:33.573+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:33.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:33.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:33.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:33.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:33.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:33.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:33.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:33.423+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:34.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:33.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:34.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:33.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:34.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:33.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:34.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:33.974+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:34.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:34.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:34.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:34.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:34.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:34.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:34.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:34.545+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:34.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:34.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:34.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:34.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:34.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:34.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:34.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:34.386+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:35.356 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:34.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:35.356 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:34.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:35.356 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:34.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:35.356 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:34.972+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:35.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:35.563+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:35.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:35.563+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:35.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:35.563+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:35.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:35.564+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:35.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:35.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:35.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:35.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:35.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:35.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:35.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:35.379+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:36.342 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:25:36.108+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 16923 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:25:36.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:35.993+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:36.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:35.993+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:36.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:35.993+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:36.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:35.994+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:36.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:36.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:36.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:36.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:36.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:36.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:36.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:36.590+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:36.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:36.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:36.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:36.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:36.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:36.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:36.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:36.423+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:37.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:37.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:37.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:37.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:37.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:37.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:37.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:37.001+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:37.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:37.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:37.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:37.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:37.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:37.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:37.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:37.427+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:37.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:37.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:37.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:37.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:37.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:37.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:37.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:37.600+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:38.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:37.982+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:38.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:37.982+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:38.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:37.982+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:38.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:37.982+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:38.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:38.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:38.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:38.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:38.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:38.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:38.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:38.397+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:38.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:38.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:38.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:38.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:38.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:38.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:38.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:38.592+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:39.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:38.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:39.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:38.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:39.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:38.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:39.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:38.946+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:39.586 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:39.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:39.587 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:39.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:39.587 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:39.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:39.587 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:39.447+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:39.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:39.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:39.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:39.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:39.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:39.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:39.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:39.610+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:40.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:39.901+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:40.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:39.901+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:40.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:39.901+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:40.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:39.901+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:40.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:40.635+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:40.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:40.635+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:40.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:40.635+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:40.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:40.635+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:40.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:40.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:40.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:40.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:40.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:40.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:40.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:40.414+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:41.195 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:25:41.110+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17011 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:25:41.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:40.934+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:41.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:40.934+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:41.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:40.934+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:41.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:40.935+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:41.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:41.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:41.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:41.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:41.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:41.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:41.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:41.449+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:41.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:41.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:41.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:41.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:41.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:41.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:41.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:41.903+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:41.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:41.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:41.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:41.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:41.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:41.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:41.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:41.667+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:42.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:42.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:42.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:42.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:42.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:42.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:42.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:42.446+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:42.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:42.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:42.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:42.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:42.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:42.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:42.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:42.948+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:42.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:42.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:42.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:42.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:42.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:42.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:42.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:42.717+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:43.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:43.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:43.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:43.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:43.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:43.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:43.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:43.695+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:43.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:43.476+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:43.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:43.476+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:43.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:43.476+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:43.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:43.476+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:44.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:43.923+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:44.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:43.923+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:44.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:43.923+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:44.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:43.925+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:44.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:44.525+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:44.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:44.525+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:44.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:44.525+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:44.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:44.526+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:44.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:44.943+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:44.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:44.943+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:44.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:44.943+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:44.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:44.944+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:44.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:44.700+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:44.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:44.700+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:44.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:44.700+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:44.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:44.701+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:45.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:45.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:45.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:45.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:45.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:45.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:45.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:45.738+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:45.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:45.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:45.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:45.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:45.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:45.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:45.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:45.494+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:46.196 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:25:46.111+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17098 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:25:46.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:45.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:46.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:45.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:46.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:45.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:46.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:45.938+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:46.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:46.487+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:46.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:46.487+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:46.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:46.487+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:46.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:46.488+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:46.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:46.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:46.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:46.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:46.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:46.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:46.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:46.900+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:46.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:46.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:46.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:46.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:46.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:46.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:46.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:46.712+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:47.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:47.684+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:47.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:47.684+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:47.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:47.684+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:47.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:47.685+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:47.861 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:47.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:47.861 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:47.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:47.861 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:47.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:47.861 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:47.470+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:48.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:47.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:48.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:47.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:48.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:47.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:48.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:47.886+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:48.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:48.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:48.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:48.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:48.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:48.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:48.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:48.467+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:48.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:48.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:48.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:48.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:48.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:48.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:48.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:48.645+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:49.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:48.840+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:49.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:48.840+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:49.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:48.840+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:49.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:48.841+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:49.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:49.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:49.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:49.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:49.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:49.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:49.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:49.599+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:49.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:49.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:49.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:49.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:49.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:49.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:49.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:49.449+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:50.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:49.869+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:50.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:49.869+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:50.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:49.869+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:50.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:49.870+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:50.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:50.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:50.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:50.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:50.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:50.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:50.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:50.445+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:50.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:50.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:50.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:50.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:50.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:50.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:50.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:50.583+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:51.114 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:25:51.115+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17186 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:25:51.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:50.883+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:51.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:50.883+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:51.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:50.883+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:51.115 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:50.883+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:51.863 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:51.491+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:51.864 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:51.491+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:51.864 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:51.491+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:51.864 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:51.491+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:51.864 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:51.536+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:51.864 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:51.536+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:51.864 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:51.536+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:51.864 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:51.537+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:52.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:51.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:52.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:51.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:52.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:51.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:52.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:51.888+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:52.840 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:52.509+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:52.840 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:52.509+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:52.840 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:52.509+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:52.840 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:52.510+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:52.841 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:52.548+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:52.841 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:52.548+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:52.841 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:52.548+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:52.841 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:52.549+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:53.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:52.864+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:53.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:52.864+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:53.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:52.864+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:53.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:52.865+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:53.828 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:53.484+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:53.828 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:53.484+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:53.828 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:53.484+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:53.828 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:53.485+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:53.828 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:53.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:53.828 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:53.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:53.828 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:53.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:53.828 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:53.541+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:54.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:53.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:54.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:53.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:54.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:53.852+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:54.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:53.853+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:54.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:54.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:54.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:54.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:54.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:54.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:54.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:54.577+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:54.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:54.454+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:54.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:54.454+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:54.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:54.454+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:54.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:54.455+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:55.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:54.900+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:55.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:54.900+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:55.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:54.900+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:55.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:54.901+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:55.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:55.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:55.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:55.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:55.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:55.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:55.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:55.536+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:55.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:55.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:55.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:55.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:55.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:55.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:55.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:55.415+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:56.196 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:25:56.116+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17274 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:25:56.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:55.943+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:56.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:55.943+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:56.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:55.943+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:56.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:55.944+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:56.484 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:56.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:56.484 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:56.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:56.484 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:56.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:56.484 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:56.413+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:56.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:56.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:56.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:56.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:56.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:56.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:56.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:56.509+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:57.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:56.961+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:57.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:56.961+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:57.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:56.961+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:57.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:56.962+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:57.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:57.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:57.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:57.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:57.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:57.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:57.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:57.531+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:57.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:57.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:57.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:57.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:57.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:57.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:57.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:57.372+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:58.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:57.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:58.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:57.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:58.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:57.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:58.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:57.965+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:58.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:58.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:58.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:58.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:58.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:58.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:58.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:58.406+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:58.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:58.542+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:58.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:58.542+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:58.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:58.542+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:58.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:58.543+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:59.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:58.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:59.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:58.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:59.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:58.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:25:59.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:58.921+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:59.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:59.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:59.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:59.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:59.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:59.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:25:59.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:25:59.396+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:25:59.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:59.536+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:59.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:59.536+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:59.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:59.536+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:25:59.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:25:59.537+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:00.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:59.928+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:00.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:59.928+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:00.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:59.928+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:00.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:25:59.929+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:00.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:00.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:00.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:00.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:00.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:00.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:00.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:00.503+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:00.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:00.429+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:00.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:00.429+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:00.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:00.429+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:00.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:00.429+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:01.196 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:26:01.118+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17362 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:26:01.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:00.942+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:01.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:00.942+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:01.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:00.942+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:01.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:00.942+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:01.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:01.431+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:01.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:01.431+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:01.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:01.431+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:01.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:01.432+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:01.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:01.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:01.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:01.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:01.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:01.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:01.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:01.486+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:02.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:01.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:02.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:01.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:02.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:01.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:02.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:01.931+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:02.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:02.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:02.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:02.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:02.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:02.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:02.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:02.451+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:02.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:02.447+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:02.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:02.447+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:02.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:02.447+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:02.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:02.447+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:03.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:02.963+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:03.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:02.963+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:03.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:02.963+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:03.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:02.963+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:03.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:03.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:03.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:03.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:03.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:03.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:03.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:03.489+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:03.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:03.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:03.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:03.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:03.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:03.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:03.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:03.420+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:04.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:03.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:04.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:03.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:04.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:03.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:04.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:03.964+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:04.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:04.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:04.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:04.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:04.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:04.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:04.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:04.438+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:04.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:04.528+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:04.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:04.528+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:04.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:04.528+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:04.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:04.529+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:05.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:04.953+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:05.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:04.953+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:05.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:04.953+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:05.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:04.955+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:05.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:05.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:05.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:05.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:05.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:05.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:05.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:05.446+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:05.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:05.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:05.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:05.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:05.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:05.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:05.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:05.550+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:06.196 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:26:06.120+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17449 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:26:06.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:05.962+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:06.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:05.962+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:06.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:05.962+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:06.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:05.962+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:06.545 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:06.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:06.545 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:06.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:06.545 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:06.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:06.545 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:06.434+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:06.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:06.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:06.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:06.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:06.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:06.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:06.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:06.570+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:07.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:07.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:07.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:07.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:07.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:07.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:07.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:07.007+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:07.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:07.616+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:07.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:07.616+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:07.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:07.616+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:07.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:07.617+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:07.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:07.476+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:07.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:07.476+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:07.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:07.476+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:07.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:07.477+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:08.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:08.023+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:08.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:08.023+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:08.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:08.023+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:08.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:08.024+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:08.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:08.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:08.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:08.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:08.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:08.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:08.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:08.589+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:08.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:08.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:08.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:08.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:08.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:08.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:08.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:08.508+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:09.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:09.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:09.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:09.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:09.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:09.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:09.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:09.022+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:09.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:09.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:09.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:09.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:09.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:09.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:09.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:09.551+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:09.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:09.509+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:09.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:09.509+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:09.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:09.509+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:09.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:09.510+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:10.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:10.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:10.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:10.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:10.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:10.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:10.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:10.042+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:10.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:10.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:10.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:10.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:10.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:10.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:10.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:10.580+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:10.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:10.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:10.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:10.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:10.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:10.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:10.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:10.530+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:11.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:26:11.121+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17536 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:26:11.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:11.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:11.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:11.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:11.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:11.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:11.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:11.005+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:11.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:11.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:11.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:11.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:11.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:11.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:11.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:11.549+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:11.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:11.587+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:11.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:11.587+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:11.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:11.587+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:11.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:11.587+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:12.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:11.969+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:12.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:11.969+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:12.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:11.969+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:12.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:11.970+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:12.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:12.588+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:12.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:12.588+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:12.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:12.588+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:12.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:12.589+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:12.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:12.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:12.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:12.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:12.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:12.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:12.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:12.546+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:13.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:12.943+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:13.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:12.943+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:13.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:12.943+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:13.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:12.944+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:13.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:13.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:13.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:13.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:13.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:13.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:13.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:13.557+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:13.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:13.636+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:13.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:13.636+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:13.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:13.636+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:13.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:13.637+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:14.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:13.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:14.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:13.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:14.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:13.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:14.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:13.922+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:14.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:14.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:14.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:14.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:14.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:14.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:14.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:14.532+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:14.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:14.675+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:14.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:14.675+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:14.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:14.675+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:14.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:14.675+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:15.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:14.911+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:15.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:14.912+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:15.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:14.912+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:15.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:14.913+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:15.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:15.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:15.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:15.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:15.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:15.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:15.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:15.487+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:15.856 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:15.645+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:15.856 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:15.645+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:15.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:15.645+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:15.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:15.646+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:16.109 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:26:16.123+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17624 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:26:16.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:15.880+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:16.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:15.880+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:16.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:15.880+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:16.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:15.881+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:16.869 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:16.489+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:16.869 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:16.489+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:16.869 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:16.489+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:16.870 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:16.490+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:16.870 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:16.597+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:16.870 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:16.597+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:16.870 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:16.597+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:16.870 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:16.597+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:17.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:16.910+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:17.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:16.910+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:17.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:16.910+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:17.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:16.911+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:17.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:17.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:17.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:17.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:17.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:17.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:17.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:17.512+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:17.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:17.607+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:17.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:17.607+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:17.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:17.607+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:17.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:17.607+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:18.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:17.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:18.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:17.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:18.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:17.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:18.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:17.939+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:18.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:18.648+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:18.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:18.648+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:18.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:18.648+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:18.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:18.649+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:18.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:18.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:18.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:18.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:18.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:18.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:18.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:18.557+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:19.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:18.985+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:19.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:18.985+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:19.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:18.985+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:19.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:18.986+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:19.878 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:19.688+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:19.879 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:19.688+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:19.879 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:19.688+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:19.879 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:19.688+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:19.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:19.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:19.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:19.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:19.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:19.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:19.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:19.576+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:20.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:19.982+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:20.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:19.982+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:20.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:19.982+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:20.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:19.982+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:20.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:20.678+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:20.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:20.678+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:20.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:20.678+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:20.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:20.678+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:20.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:20.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:20.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:20.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:20.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:20.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:20.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:20.608+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:21.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:26:21.125+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17713 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:26:21.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:21.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:21.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:21.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:21.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:21.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:21.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:21.023+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:21.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:21.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:21.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:21.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:21.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:21.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:21.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:21.654+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:21.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:21.660+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:21.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:21.660+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:21.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:21.660+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:21.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:21.660+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:22.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:22.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:22.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:22.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:22.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:22.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:22.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:22.048+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:22.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:22.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:22.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:22.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:22.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:22.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:22.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:22.621+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:22.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:22.695+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:22.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:22.695+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:22.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:22.695+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:22.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:22.696+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:23.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:23.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:23.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:23.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:23.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:23.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:23.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:23.054+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:23.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:23.581+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:23.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:23.581+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:23.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:23.581+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:23.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:23.582+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:23.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:23.648+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:23.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:23.648+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:23.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:23.648+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:23.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:23.648+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:24.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:24.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:24.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:24.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:24.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:24.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:24.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:24.014+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:24.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:24.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:24.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:24.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:24.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:24.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:24.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:24.593+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:24.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:24.614+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:24.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:24.614+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:24.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:24.614+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:24.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:24.614+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:25.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:25.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:25.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:25.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:25.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:25.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:25.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:25.064+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:25.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:25.557+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:25.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:25.557+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:25.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:25.557+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:25.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:25.558+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:25.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:25.653+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:25.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:25.653+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:25.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:25.653+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:25.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:25.654+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:26.342 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:26:26.127+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17801 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:26:26.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:26.106+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:26.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:26.106+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:26.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:26.106+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:26.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:26.107+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:26.621 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:26.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:26.621 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:26.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:26.621 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:26.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:26.621 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:26.524+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:26.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:26.645+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:26.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:26.645+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:26.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:26.645+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:26.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:26.645+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:27.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:27.130+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:27.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:27.130+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:27.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:27.130+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:27.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:27.131+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:27.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:27.672+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:27.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:27.672+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:27.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:27.672+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:27.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:27.672+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:27.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:27.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:27.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:27.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:27.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:27.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:27.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:27.556+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:28.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:28.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:28.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:28.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:28.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:28.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:28.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:28.136+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:28.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:28.657+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:28.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:28.657+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:28.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:28.657+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:28.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:28.657+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:28.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:28.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:28.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:28.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:28.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:28.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:28.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:28.546+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:29.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:29.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:29.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:29.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:29.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:29.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:29.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:29.116+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:29.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:29.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:29.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:29.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:29.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:29.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:29.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:29.522+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:29.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:29.673+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:29.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:29.673+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:29.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:29.673+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:29.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:29.673+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:30.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:30.133+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:30.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:30.133+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:30.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:30.133+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:30.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:30.134+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:30.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:30.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:30.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:30.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:30.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:30.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:30.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:30.536+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:30.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:30.692+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:30.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:30.692+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:30.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:30.692+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:30.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:30.693+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:31.446 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:26:31.128+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17890 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:26:31.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:31.137+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:31.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:31.137+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:31.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:31.137+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:31.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:31.139+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:31.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:31.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:31.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:31.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:31.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:31.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:31.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:31.534+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:31.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:31.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:31.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:31.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:31.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:31.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:31.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:31.730+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:32.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:32.133+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:32.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:32.133+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:32.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:32.133+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:32.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:32.134+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:32.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:32.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:32.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:32.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:32.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:32.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:32.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:32.523+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:32.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:32.736+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:32.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:32.736+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:32.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:32.736+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:32.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:32.736+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:33.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:33.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:33.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:33.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:33.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:33.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:33.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:33.151+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:33.748 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:33.484+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:33.748 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:33.484+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:33.748 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:33.484+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:33.748 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:33.485+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:34.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:33.771+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:34.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:33.771+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:34.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:33.771+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:34.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:33.772+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:34.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:34.144+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:34.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:34.144+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:34.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:34.144+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:34.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:34.145+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:34.767 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:34.469+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:34.767 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:34.469+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:34.767 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:34.469+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:34.767 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:34.470+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:35.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:34.816+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:35.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:34.816+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:35.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:34.816+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:35.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:34.816+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:35.433 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:35.183+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:35.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:35.183+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:35.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:35.183+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:35.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:35.183+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:35.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:35.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:35.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:35.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:35.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:35.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:35.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:35.457+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:36.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:35.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:36.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:35.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:36.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:35.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:36.109 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:35.822+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:36.390 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:26:36.130+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 17979 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:26:36.390 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:36.157+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:36.390 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:36.157+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:36.391 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:36.157+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:36.391 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:36.157+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:36.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:36.413+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:36.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:36.413+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:36.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:36.413+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:36.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:36.414+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:37.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:36.840+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:37.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:36.840+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:37.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:36.840+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:37.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:36.840+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:37.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:37.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:37.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:37.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:37.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:37.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:37.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:37.122+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:37.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:37.462+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:37.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:37.462+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:37.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:37.462+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:37.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:37.463+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:38.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:37.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:38.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:37.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:38.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:37.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:38.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:37.822+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:38.422 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:38.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:38.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:38.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:38.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:38.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:38.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:38.083+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:38.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:38.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:38.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:38.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:38.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:38.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:38.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:38.446+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:39.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:38.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:39.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:38.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:39.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:38.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:39.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:38.864+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:39.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:39.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:39.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:39.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:39.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:39.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:39.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:39.037+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:39.806 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:39.481+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:39.806 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:39.481+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:39.806 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:39.481+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:39.806 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:39.481+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:40.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:40.035+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:40.144 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:40.035+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:40.144 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:40.035+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:40.144 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:40.035+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:40.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:39.890+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:40.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:39.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:40.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:39.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:40.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:39.891+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:40.818 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:40.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:40.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:40.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:40.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:40.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:40.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:40.483+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:41.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:41.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:41.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:41.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:41.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:41.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:41.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:41.082+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:41.110 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:40.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:41.110 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:40.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:41.110 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:40.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:41.110 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:40.842+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:41.380 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:26:41.132+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18068 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:26:41.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:41.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:41.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:41.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:41.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:41.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:41.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:41.470+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:42.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:42.086+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:42.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:42.086+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:42.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:42.086+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:42.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:42.087+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:42.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:41.859+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:42.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:41.859+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:42.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:41.859+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:42.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:41.859+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:42.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:42.428+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:42.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:42.428+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:42.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:42.428+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:42.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:42.429+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:43.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:43.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:43.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:43.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:43.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:43.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:43.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:43.094+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:43.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:42.877+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:43.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:42.877+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:43.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:42.877+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:43.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:42.877+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:43.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:43.417+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:43.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:43.417+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:43.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:43.417+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:43.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:43.418+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:44.115 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:43.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:44.115 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:43.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:44.115 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:43.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:44.115 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:43.843+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:44.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:44.139+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:44.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:44.139+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:44.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:44.139+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:44.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:44.140+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:44.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:44.405+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:44.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:44.405+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:44.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:44.405+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:44.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:44.406+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:45.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:44.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:45.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:44.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:45.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:44.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:45.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:44.850+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:45.387 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:45.146+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:45.387 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:45.146+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:45.387 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:45.146+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:45.387 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:45.147+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:45.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:45.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:45.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:45.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:45.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:45.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:45.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:45.412+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:46.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:46.099+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:46.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:46.099+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:46.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:46.099+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:46.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:46.100+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:46.111 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:45.879+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:46.111 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:45.879+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:46.111 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:45.879+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:46.111 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:45.879+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:46.366 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:26:46.135+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18157 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:26:46.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:26:46.626+0000 7f770c428700 0 monclient(hunting): authenticate timed out after 300 2024-04-27T02:26:46.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:46.391+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:46.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:46.391+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:46.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:46.391+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:46.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:46.392+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:47.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:47.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:47.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:47.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:47.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:47.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:47.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:47.136+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:47.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:46.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:47.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:46.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:47.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:46.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:47.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:46.909+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:47.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:47.396+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:47.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:47.396+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:47.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:47.396+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:47.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:47.397+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:48.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:47.886+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:48.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:47.886+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:48.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:47.886+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:48.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:47.887+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:48.422 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:48.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:48.422 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:48.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:48.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:48.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:48.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:48.177+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:48.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:48.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:48.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:48.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:48.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:48.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:48.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:48.447+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:49.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:48.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:49.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:48.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:49.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:48.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:49.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:48.918+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:49.468 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:49.224+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:49.469 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:49.224+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:49.469 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:49.224+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:49.469 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:49.224+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:49.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:49.493+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:49.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:49.493+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:49.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:49.493+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:49.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:49.493+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:50.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:49.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:50.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:49.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:50.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:49.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:50.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:49.902+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:50.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:50.254+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:50.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:50.254+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:50.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:50.254+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:50.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:50.254+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:50.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:50.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:50.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:50.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:50.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:50.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:50.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:50.509+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:51.195 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:26:51.136+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18247 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:26:51.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:50.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:51.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:50.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:51.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:50.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:51.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:50.942+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:51.505 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:51.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:51.506 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:51.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:51.506 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:51.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:51.506 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:51.290+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:51.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:51.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:51.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:51.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:51.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:51.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:51.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:51.531+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:52.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:51.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:52.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:51.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:52.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:51.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:52.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:51.940+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:52.510 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:52.267+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:52.510 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:52.267+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:52.510 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:52.267+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:52.510 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:52.268+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:52.923 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:52.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:52.923 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:52.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:52.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:52.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:52.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:52.535+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:53.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:52.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:53.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:52.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:53.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:52.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:53.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:52.948+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:53.484 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:53.225+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:53.484 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:53.225+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:53.484 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:53.225+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:53.484 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:53.226+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:53.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:53.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:53.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:53.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:53.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:53.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:53.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:53.509+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:54.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:53.938+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:54.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:53.938+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:54.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:53.938+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:54.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:53.938+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:54.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:54.218+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:54.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:54.218+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:54.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:54.218+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:54.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:54.219+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:54.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:54.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:54.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:54.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:54.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:54.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:54.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:54.554+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:55.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:54.902+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:55.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:54.902+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:55.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:54.902+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:55.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:54.902+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:55.533 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:55.225+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:55.533 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:55.225+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:55.533 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:55.225+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:55.533 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:55.226+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:55.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:55.557+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:55.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:55.557+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:55.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:55.557+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:55.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:55.558+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:56.140 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:26:56.139+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18337 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:26:56.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:55.907+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:56.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:55.907+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:56.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:55.907+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:56.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:55.908+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:56.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:56.255+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:56.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:56.255+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:56.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:56.255+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:56.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:56.256+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:56.923 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:56.585+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:56.923 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:56.585+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:56.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:56.585+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:56.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:56.586+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:57.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:56.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:57.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:56.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:57.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:56.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:57.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:56.948+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:57.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:57.234+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:57.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:57.234+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:57.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:57.234+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:57.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:57.235+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:57.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:57.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:57.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:57.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:57.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:57.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:57.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:57.635+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:58.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:57.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:58.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:57.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:58.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:57.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:58.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:57.968+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:58.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:58.243+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:58.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:58.243+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:58.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:58.243+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:58.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:58.244+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:58.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:58.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:58.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:58.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:58.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:58.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:58.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:58.667+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:59.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:58.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:59.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:58.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:59.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:58.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:26:59.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:58.964+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:59.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:59.218+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:59.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:59.218+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:59.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:59.218+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:26:59.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:26:59.219+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:26:59.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:59.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:59.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:59.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:59.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:59.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:26:59.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:26:59.620+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:00.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:59.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:00.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:59.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:00.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:59.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:00.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:26:59.925+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:00.547 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:00.230+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:00.547 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:00.230+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:00.547 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:00.230+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:00.547 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:00.230+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:00.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:00.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:00.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:00.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:00.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:00.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:00.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:00.572+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:01.195 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:27:01.141+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18426 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:27:01.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:00.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:01.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:00.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:01.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:00.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:01.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:00.931+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:01.545 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:01.261+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:01.545 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:01.261+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:01.545 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:01.261+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:01.545 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:01.262+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:01.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:01.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:01.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:01.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:01.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:01.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:01.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:01.570+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:02.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:01.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:02.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:01.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:02.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:01.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:02.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:01.961+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:02.510 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:02.247+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:02.510 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:02.247+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:02.510 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:02.247+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:02.510 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:02.248+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:02.885 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:02.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:02.885 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:02.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:02.885 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:02.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:02.885 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:02.535+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:03.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:02.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:03.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:02.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:03.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:02.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:03.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:02.938+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:03.525 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:03.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:03.525 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:03.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:03.526 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:03.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:03.526 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:03.243+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:03.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:03.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:03.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:03.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:03.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:03.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:03.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:03.551+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:04.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:04.197+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:04.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:04.197+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:04.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:04.197+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:04.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:04.198+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:04.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:03.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:04.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:03.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:04.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:03.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:04.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:03.965+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:04.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:04.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:04.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:04.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:04.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:04.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:04.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:04.523+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:05.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:05.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:05.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:05.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:05.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:05.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:05.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:05.165+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:05.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:05.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:05.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:05.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:05.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:05.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:05.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:05.004+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:05.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:05.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:05.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:05.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:05.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:05.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:05.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:05.511+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:06.195 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:27:06.142+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18513 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:27:06.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:06.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:06.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:06.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:06.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:06.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:06.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:06.173+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:06.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:05.962+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:06.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:05.962+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:06.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:05.962+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:06.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:05.963+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:06.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:06.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:06.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:06.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:06.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:06.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:06.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:06.473+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:07.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:07.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:07.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:07.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:07.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:07.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:07.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:07.129+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:07.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:06.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:07.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:06.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:07.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:06.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:07.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:06.980+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:07.918 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:07.479+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:07.918 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:07.479+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:07.918 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:07.479+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:07.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:07.480+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:08.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:08.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:08.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:08.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:08.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:08.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:08.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:08.123+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:08.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:07.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:08.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:07.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:08.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:07.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:08.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:07.943+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:08.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:08.441+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:08.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:08.441+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:08.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:08.441+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:08.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:08.442+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:09.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:09.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:09.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:09.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:09.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:09.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:09.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:09.146+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:09.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:08.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:09.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:08.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:09.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:08.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:09.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:08.931+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:09.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:09.447+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:09.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:09.447+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:09.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:09.447+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:09.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:09.448+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:10.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:10.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:10.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:10.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:10.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:10.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:10.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:10.121+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:10.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:09.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:10.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:09.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:10.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:09.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:10.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:09.971+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:10.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:10.428+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:10.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:10.428+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:10.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:10.428+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:10.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:10.429+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:11.245 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:27:11.144+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18602 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:27:11.245 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:10.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:11.245 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:10.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:11.245 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:10.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:11.245 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:10.991+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:11.246 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:11.087+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:11.246 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:11.087+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:11.246 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:11.087+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:11.246 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:11.088+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:11.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:11.428+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:11.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:11.428+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:11.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:11.428+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:11.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:11.429+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:12.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:11.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:12.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:11.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:12.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:11.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:12.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:11.970+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:12.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:12.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:12.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:12.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:12.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:12.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:12.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:12.063+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:12.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:12.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:12.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:12.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:12.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:12.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:12.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:12.422+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:13.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:13.094+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:13.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:13.094+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:13.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:13.094+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:13.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:13.094+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:13.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:12.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:13.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:12.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:13.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:12.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:13.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:12.941+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:13.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:13.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:13.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:13.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:13.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:13.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:13.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:13.398+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:14.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:14.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:14.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:14.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:14.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:14.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:14.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:14.091+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:14.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:13.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:14.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:13.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:14.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:13.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:14.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:13.918+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:14.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:14.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:14.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:14.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:14.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:14.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:14.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:14.396+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:15.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:15.069+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:15.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:15.069+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:15.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:15.069+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:15.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:15.069+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:15.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:14.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:15.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:14.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:15.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:14.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:15.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:14.915+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:15.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:15.355+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:15.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:15.355+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:15.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:15.355+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:45.143619+0000 (oldest deadline 2024-04-27T02:17:05.143619+0000) 2024-04-27T02:27:15.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:15.356+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:16.195 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:27:16.146+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18691 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:27:16.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:15.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:16.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:15.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:16.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:15.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:16.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:15.959+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:16.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:16.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:16.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:16.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:16.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:16.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:16.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:16.082+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:16.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:16.324+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:17.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:16.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:17.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:16.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:17.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:16.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:17.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:16.974+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:17.306 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:17.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:17.306 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:17.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:17.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:17.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:17.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:17.119+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:17.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:17.330+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:18.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:18.002+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:18.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:18.002+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:18.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:18.002+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:18.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:18.002+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:18.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:18.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:18.272 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:18.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:18.272 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:18.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:18.272 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:18.151+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:18.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:18.295+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:19.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:19.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:19.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:19.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:19.127 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:19.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:19.127 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:19.001+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:19.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:19.271+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:19.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:19.151+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:19.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:19.151+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:19.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:19.151+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:19.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:19.151+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:20.253 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:20.181+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:20.253 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:20.181+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:20.253 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:20.181+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:20.253 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:20.181+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:20.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:20.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:20.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:20.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:20.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:20.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:20.254 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:20.020+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:20.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:20.277+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:21.246 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:27:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:27:21.149+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18780 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:27:21.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:20.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:21.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:20.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:21.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:20.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:21.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:20.998+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:21.247 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:21.225+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:21.247 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:21.225+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:21.248 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:21.225+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:21.248 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:21.226+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:21.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:21.308+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:22.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:22.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:22.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:22.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:22.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:22.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:22.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:22.013+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:22.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:22.239+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:22.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:22.239+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:22.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:22.239+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:22.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:22.240+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:22.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:22.327+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:23.339 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:23.235+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:23.339 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:23.235+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:23.339 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:23.235+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:23.339 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:23.236+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:23.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:23.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:23.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:23.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:23.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:23.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:23.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:23.004+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:23.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:23.363+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:24.305 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:24.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:24.305 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:24.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:24.305 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:24.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:24.305 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:24.038+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:24.306 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:24.218+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:24.306 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:24.218+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:24.306 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:24.218+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:24.306 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:24.219+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:24.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:24.329+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:25.282 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:25.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:25.282 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:25.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:25.282 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:25.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:25.282 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:25.211+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:25.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:25.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:25.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:25.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:25.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:25.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:25.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:25.052+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:25.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:25.306+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:26.342 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:27:26.150+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18867 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:27:26.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:26.293+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:26.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:26.192+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:26.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:26.192+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:26.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:26.192+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:26.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:26.193+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:26.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:26.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:26.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:26.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:26.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:26.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:55.792104+0000 (oldest deadline 2024-04-27T02:17:15.792104+0000) 2024-04-27T02:27:26.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:26.056+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:27.304 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:27.044+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:27.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:27.152+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:27.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:27.152+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:27.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:27.152+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:27.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:27.153+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:27.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:27.328+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:28.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:28.016+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:28.306 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:28.115+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:28.306 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:28.115+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:28.306 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:28.115+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:28.306 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:28.116+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:28.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:28.330+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:29.312 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:28.976+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:29.313 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:29.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:29.313 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:29.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:29.313 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:29.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:16:57.033090+0000 (oldest deadline 2024-04-27T02:17:17.033090+0000) 2024-04-27T02:27:29.313 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:29.091+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:29.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:29.336+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:30.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:30.136+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:30.295 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:29.989+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:30.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:30.317+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:31.252 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:27:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:27:31.151+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 18955 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:27:31.252 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:31.112+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:31.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:31.021+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:31.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:31.323+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:32.273 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:32.080+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:32.273 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:32.001+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:32.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:32.296+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:33.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:33.257+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:33.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:33.098+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:33.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:33.014+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:34.236 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:34.095+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:34.236 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:34.037+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:34.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:34.260+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:35.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:35.080+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:35.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:35.265+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:35.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:35.040+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:36.255 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:27:36.153+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19043 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:27:36.256 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:36.077+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:36.256 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:35.999+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:36.256 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:36.228+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:36.256 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:36.228+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:36.256 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:36.228+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:36.257 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:36.228+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:37.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:36.961+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:37.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:37.081+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:37.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:37.248+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:37.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:37.248+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:37.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:37.248+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:37.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:37.249+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:38.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:38.048+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:38.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:37.928+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:38.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:38.281+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:38.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:38.281+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:38.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:38.281+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:38.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:38.282+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:39.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:39.020+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:39.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:38.893+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:39.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:39.317+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:39.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:39.317+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:39.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:39.317+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:39.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:39.318+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:40.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:40.037+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:40.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:39.871+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:40.332+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:40.332+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:40.332+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:40.333+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:41.132 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:41.025+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:41.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:40.844+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:41.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:27:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:27:41.157+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19131 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:27:41.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:41.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:41.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:41.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:41.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:41.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:41.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:41.294+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:42.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:41.999+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:42.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:41.862+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:42.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:42.322+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:42.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:42.322+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:42.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:42.322+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:42.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:42.322+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:43.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:43.027+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:43.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:42.892+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:43.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:43.294+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:43.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:43.294+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:43.572 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:43.294+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:43.572 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:43.295+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:44.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:43.995+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:44.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:43.918+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:44.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:44.255+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:44.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:44.255+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:44.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:44.255+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:44.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:44.256+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:45.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:44.956+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:45.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:44.909+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:45.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:45.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:45.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:45.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:45.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:45.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:45.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:45.223+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:46.146 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:27:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:27:46.158+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19218 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:27:46.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:45.991+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:46.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:45.955+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:46.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:46.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:46.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:46.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:46.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:46.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:46.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:46.251+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:47.263 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:46.973+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:47.263 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:46.982+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:47.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:47.288+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:47.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:47.288+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:47.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:47.288+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:47.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:47.288+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:48.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:47.931+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:48.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:47.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:48.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:47.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:48.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:47.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:48.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:47.993+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:48.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:48.308+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:48.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:48.308+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:48.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:48.308+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:48.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:48.309+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:49.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:48.889+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:49.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:48.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:49.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:48.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:49.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:48.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:49.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:48.952+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:49.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:49.278+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:49.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:49.278+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:49.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:49.278+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:49.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:49.279+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:50.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:49.938+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:50.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:49.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:50.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:49.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:50.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:49.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:50.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:49.932+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:50.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:50.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:50.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:50.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:50.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:50.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:50.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:50.262+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:51.148 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:27:51.161+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19306 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:27:51.148 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:50.949+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:51.148 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:50.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:51.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:50.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:51.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:50.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:51.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:50.918+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:51.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:51.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:51.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:51.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:51.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:51.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:51.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:51.226+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:52.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:51.912+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:52.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:51.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:52.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:51.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:52.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:51.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:52.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:51.895+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:52.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:52.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:52.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:52.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:52.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:52.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:52.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:52.247+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:53.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:52.911+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:53.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:52.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:53.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:52.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:53.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:52.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:53.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:52.924+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:53.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:53.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:53.572 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:53.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:53.572 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:53.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:53.572 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:53.251+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:54.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:53.921+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:54.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:53.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:54.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:53.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:54.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:53.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:54.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:53.948+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:54.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:54.229+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:54.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:54.229+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:54.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:54.229+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:54.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:54.229+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:55.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:54.952+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:55.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:55.186+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:55.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:55.186+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:55.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:55.186+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:55.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:55.187+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:55.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:54.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:55.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:54.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:55.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:54.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:55.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:54.947+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:56.195 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:27:56.162+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19394 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:27:56.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:55.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:27:56.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:55.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:27:56.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:55.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:27:56.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:55.990+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:56.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:56.154+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:56.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:56.154+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:56.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:56.154+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:56.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:56.154+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:56.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:55.950+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:56.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:55.950+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:56.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:55.950+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:56.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:55.951+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:57.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:57.138+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:57.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:57.138+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:57.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:57.138+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:57.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:57.138+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:57.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:56.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:57.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:56.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:57.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:56.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:57.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:56.937+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:57.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:57.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:27:57.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:57.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:27:57.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:57.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:27:57.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:57.010+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:58.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:58.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:27:58.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:58.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:27:58.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:58.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:27:58.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:58.042+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:58.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:58.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:58.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:58.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:58.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:58.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:58.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:58.140+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:58.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:57.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:58.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:57.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:58.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:57.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:58.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:57.936+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:59.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:59.027+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:27:59.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:59.027+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:27:59.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:59.027+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:27:59.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:27:59.028+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:59.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:59.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:59.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:59.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:59.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:59.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:27:59.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:27:59.129+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:27:59.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:58.950+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:59.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:58.950+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:59.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:58.950+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:27:59.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:58.951+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:00.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:00.086+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:00.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:00.086+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:00.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:00.086+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:00.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:00.087+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:00.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:00.056+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:00.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:00.056+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:00.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:00.056+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:00.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:00.057+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:00.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:59.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:00.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:59.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:00.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:59.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:00.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:27:59.998+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:01.252 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:28:01.165+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19482 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:28:01.252 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:01.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:01.253 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:01.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:01.253 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:01.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:01.253 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:01.089+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:01.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:01.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:01.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:01.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:01.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:01.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:01.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:01.010+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:01.253 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:01.051+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:01.254 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:01.051+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:01.254 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:01.051+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:01.254 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:01.052+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:02.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:02.079+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:02.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:02.079+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:02.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:02.079+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:02.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:02.080+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:02.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:02.134+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:02.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:02.134+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:02.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:02.134+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:02.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:02.134+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:02.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:02.019+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:02.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:02.019+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:02.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:02.019+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:02.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:02.019+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:03.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:03.039+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:03.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:03.039+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:03.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:03.039+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:03.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:03.039+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:03.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:03.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:03.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:03.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:03.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:03.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:03.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:03.124+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:03.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:03.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:03.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:03.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:03.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:03.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:03.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:03.022+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:04.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:04.057+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:04.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:04.057+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:04.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:04.057+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:04.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:04.058+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:04.325 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:04.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:04.325 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:04.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:04.325 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:04.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:04.325 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:04.113+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:04.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:04.037+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:04.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:04.037+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:04.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:04.037+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:04.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:04.037+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:05.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:05.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:05.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:05.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:05.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:05.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:05.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:05.023+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:05.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:05.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:05.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:05.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:05.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:05.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:05.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:05.111+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:05.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:05.071+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:05.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:05.071+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:05.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:05.071+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:05.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:05.071+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:06.264 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:28:06.167+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19570 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:28:06.264 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:06.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:06.264 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:06.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:06.264 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:06.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:06.264 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:06.063+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:06.265 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:06.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:06.265 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:06.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:06.265 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:06.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:06.265 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:06.063+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:06.265 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:06.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:06.265 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:06.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:06.265 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:06.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:06.265 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:06.030+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:07.397 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:07.025+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:07.397 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:07.025+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:07.397 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:07.025+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:07.397 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:07.026+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:07.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:07.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:07.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:07.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:07.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:07.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:07.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:07.092+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:07.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:06.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:07.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:06.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:07.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:06.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:07.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:06.988+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:08.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:08.069+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:08.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:08.069+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:08.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:08.069+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:08.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:08.070+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:08.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:08.085+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:08.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:08.085+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:08.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:08.085+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:08.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:08.086+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:08.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:07.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:08.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:07.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:08.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:07.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:08.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:07.940+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:09.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:09.070+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:09.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:09.070+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:09.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:09.070+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:09.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:09.072+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:09.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:09.076+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:09.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:09.076+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:09.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:09.076+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:09.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:09.077+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:09.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:08.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:09.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:08.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:09.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:08.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:09.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:08.914+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:10.100 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:10.031+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:10.100 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:10.031+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:10.100 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:10.031+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:10.100 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:10.032+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:10.101 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:09.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:10.101 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:09.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:10.101 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:09.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:10.101 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:09.874+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:10.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:10.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:10.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:10.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:10.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:10.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:10.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:10.125+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:11.135 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:11.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:11.135 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:11.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:11.135 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:11.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:11.135 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:11.047+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:11.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:10.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:11.136 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:10.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:11.136 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:10.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:11.136 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:10.901+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:11.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:28:11.169+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19658 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:28:11.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:11.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:11.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:11.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:11.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:11.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:11.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:11.160+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:12.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:12.095+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:12.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:12.095+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:12.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:12.095+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:12.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:12.095+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:12.184 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:11.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:12.184 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:11.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:12.184 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:11.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:12.184 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:11.892+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:12.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:12.207+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:12.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:12.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:12.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:12.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:12.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:12.209+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:13.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:13.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:13.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:13.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:13.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:13.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:13.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:13.086+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:13.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:12.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:13.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:12.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:13.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:12.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:13.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:12.846+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:13.541 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:13.255+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:13.541 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:13.255+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:13.541 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:13.255+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:13.541 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:13.256+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:14.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:13.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:14.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:13.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:14.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:13.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:14.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:13.894+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:14.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:14.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:14.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:14.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:14.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:14.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:14.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:14.073+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:14.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:14.254+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:14.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:14.254+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:14.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:14.254+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:14.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:14.255+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:15.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:14.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:15.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:14.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:15.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:14.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:15.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:14.905+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:15.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:15.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:15.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:15.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:15.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:15.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:15.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:15.051+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:15.583 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:15.301+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:15.583 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:15.301+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:15.584 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:15.301+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:15.584 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:15.301+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:16.156 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:28:16.171+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19749 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:28:16.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:16.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:16.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:16.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:16.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:16.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:16.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:16.023+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:16.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:15.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:16.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:15.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:16.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:15.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:16.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:15.921+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:16.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:16.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:16.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:16.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:16.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:16.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:16.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:16.330+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:17.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:16.949+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:17.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:16.949+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:17.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:16.949+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:17.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:16.950+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:17.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:17.003+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:17.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:17.003+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:17.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:17.003+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:17.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:17.004+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:17.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:17.308+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:17.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:17.308+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:17.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:17.308+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:17.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:17.309+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:18.251 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:17.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:18.252 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:17.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:18.252 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:17.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:18.252 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:17.981+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:18.252 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:18.028+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:18.252 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:18.028+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:18.252 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:18.028+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:18.252 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:18.029+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:18.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:18.276+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:18.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:18.276+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:18.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:18.276+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:18.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:18.277+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:19.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:19.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:19.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:19.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:19.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:19.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:19.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:19.250+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:19.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:19.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:19.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:19.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:19.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:19.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:19.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:19.023+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:19.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:19.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:19.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:19.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:19.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:19.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:19.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:19.004+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:20.232 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:19.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:20.232 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:19.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:20.232 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:19.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:20.232 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:19.982+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:20.232 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:20.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:20.233 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:20.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:20.233 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:20.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:20.233 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:20.049+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:20.623 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:20.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:20.624 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:20.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:20.624 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:20.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:20.624 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:20.257+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:21.273 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:28:21.173+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19837 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:28:21.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:21.066+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:21.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:21.066+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:21.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:21.066+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:21.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:21.066+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:21.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:21.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:21.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:21.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:21.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:21.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:21.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:21.023+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:21.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:21.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:21.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:21.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:21.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:21.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:21.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:21.262+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:22.244 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:22.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:22.244 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:22.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:22.244 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:22.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:22.244 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:22.072+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:22.244 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:21.991+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:22.244 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:21.991+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:22.244 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:21.991+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:22.244 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:21.992+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:22.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:22.268+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:22.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:22.268+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:22.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:22.268+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:22.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:22.269+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:23.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:22.946+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:23.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:22.946+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:23.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:22.946+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:23.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:22.947+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:23.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:23.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:23.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:23.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:23.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:23.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:23.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:23.027+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:23.573 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:23.285+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:23.574 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:23.285+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:23.574 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:23.285+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:23.574 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:23.286+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:24.300 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:23.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:24.300 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:23.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:24.300 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:23.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:24.300 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:23.994+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:24.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:23.982+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:24.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:23.982+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:24.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:23.982+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:24.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:23.983+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:24.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:24.324+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:24.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:24.324+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:24.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:24.324+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:24.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:24.326+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:25.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:24.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:25.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:24.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:25.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:24.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:25.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:24.955+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:25.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:24.958+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:25.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:24.958+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:25.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:24.958+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:25.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:24.960+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:25.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:25.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:25.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:25.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:25.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:25.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:25.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:25.317+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:26.195 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:28:26.175+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 19927 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:28:26.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:25.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:26.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:25.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:26.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:25.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:26.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:25.960+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:26.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:25.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:26.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:25.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:26.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:25.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:26.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:25.995+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:26.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:26.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:26.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:26.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:26.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:26.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:26.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:26.316+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:27.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:27.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:27.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:27.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:27.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:27.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:27.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:27.008+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:27.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:26.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:27.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:26.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:27.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:26.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:27.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:26.969+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:27.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:27.340+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:27.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:27.340+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:27.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:27.340+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:27.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:27.341+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:28.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:28.019+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:28.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:28.019+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:28.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:28.019+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:28.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:28.021+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:28.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:27.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:28.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:27.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:28.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:27.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:28.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:27.969+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:28.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:28.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:28.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:28.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:28.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:28.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:28.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:28.331+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:29.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:28.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:29.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:28.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:29.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:28.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:29.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:28.993+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:29.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:28.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:29.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:28.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:29.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:28.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:29.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:28.948+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:29.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:29.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:29.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:29.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:29.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:29.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:29.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:29.345+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:30.272 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:29.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:30.272 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:29.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:30.272 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:29.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:30.272 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:29.972+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:30.272 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:30.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:30.273 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:30.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:30.273 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:30.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:30.273 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:30.002+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:30.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:30.295+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:30.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:30.295+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:30.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:30.295+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:30.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:30.297+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:30.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:30.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:30.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:30.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:30.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:30.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:30.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:30.960+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:31.261 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:28:31.176+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20015 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:28:31.262 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:30.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:31.262 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:30.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:31.262 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:30.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:31.262 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:30.994+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:31.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:31.285+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:31.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:31.285+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:31.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:31.285+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:31.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:31.286+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:32.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:31.961+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:32.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:31.961+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:32.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:31.961+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:32.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:31.961+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:32.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:32.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:32.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:32.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:32.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:32.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:32.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:32.007+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:32.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:32.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:32.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:32.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:32.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:32.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:32.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:32.317+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:33.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:33.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:33.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:33.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:33.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:33.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:33.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:33.001+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:33.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:32.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:33.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:32.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:33.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:32.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:33.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:32.973+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:33.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:33.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:33.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:33.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:33.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:33.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:33.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:33.366+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:34.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:33.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:34.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:33.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:34.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:33.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:34.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:33.974+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:34.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:34.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:34.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:34.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:34.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:34.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:34.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:34.009+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:34.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:34.405+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:34.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:34.405+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:34.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:34.405+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:34.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:34.406+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:35.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:34.970+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:35.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:34.970+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:35.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:34.970+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:35.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:34.970+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:35.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:34.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:35.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:34.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:35.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:34.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:35.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:34.928+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:35.621 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:35.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:35.621 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:35.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:35.621 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:35.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:35.621 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:35.393+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:35.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:35.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:35.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:35.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:35.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:35.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:35.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:35.885+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:36.177 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:28:36.178+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20106 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:28:36.178 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:35.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:36.178 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:35.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:36.178 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:35.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:36.178 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:35.948+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:36.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:36.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:36.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:36.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:36.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:36.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:36.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:36.393+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:37.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:36.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:37.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:36.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:37.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:36.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:37.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:36.921+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:37.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:36.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:37.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:36.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:37.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:36.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:37.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:36.914+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:37.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:37.346+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:37.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:37.346+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:37.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:37.346+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:37.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:37.347+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:38.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:37.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:38.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:37.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:38.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:37.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:38.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:37.915+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:38.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:37.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:38.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:37.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:38.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:37.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:38.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:37.938+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:38.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:38.333+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:38.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:38.333+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:38.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:38.333+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:38.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:38.333+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:39.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:38.913+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:39.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:38.913+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:39.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:38.913+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:39.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:38.913+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:39.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:38.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:39.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:38.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:39.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:38.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:39.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:38.964+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:39.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:39.361+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:39.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:39.361+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:39.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:39.361+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:39.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:39.362+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:40.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:39.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:40.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:39.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:40.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:39.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:40.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:39.985+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:40.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:39.912+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:40.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:39.912+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:40.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:39.912+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:40.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:39.912+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:40.341+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:40.341+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:40.341+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:40.341+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:41.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:40.901+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:41.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:40.901+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:41.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:40.901+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:41.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:40.902+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:41.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:40.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:41.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:40.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:41.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:40.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:41.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:40.953+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:41.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:28:41.181+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20195 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:28:41.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:41.300+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:41.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:41.300+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:41.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:41.300+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:41.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:41.302+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:42.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:41.883+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:42.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:41.883+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:42.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:41.883+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:42.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:41.884+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:42.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:41.926+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:42.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:41.926+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:42.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:41.926+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:42.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:41.926+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:42.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:42.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:42.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:42.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:42.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:42.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:42.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:42.264+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:43.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:42.875+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:43.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:42.875+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:43.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:42.875+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:43.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:42.876+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:43.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:42.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:43.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:42.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:43.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:42.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:43.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:42.948+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:43.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:43.221+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:43.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:43.221+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:43.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:43.221+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:43.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:43.221+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:44.170 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:43.905+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:44.170 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:43.905+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:44.170 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:43.905+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:44.170 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:43.906+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:44.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:43.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:44.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:43.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:44.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:43.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:44.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:43.956+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:44.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:44.194+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:44.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:44.194+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:44.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:44.194+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:44.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:44.195+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:45.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:44.867+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:45.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:44.867+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:45.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:44.867+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:45.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:44.868+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:45.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:44.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:45.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:44.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:45.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:44.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:45.181 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:44.948+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:45.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:45.204+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:45.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:45.204+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:45.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:45.204+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:45.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:45.205+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:46.133 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:45.877+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:46.134 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:45.877+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:46.134 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:45.877+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:46.134 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:45.878+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:46.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:45.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:46.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:45.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:46.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:45.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:46.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:45.995+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:46.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:28:46.183+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20285 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:28:46.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:46.158+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:46.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:46.158+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:46.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:46.158+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:46.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:46.159+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:47.175 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:46.923+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:47.176 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:46.923+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:47.176 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:46.923+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:47.176 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:46.923+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:47.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:47.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:47.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:47.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:47.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:47.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:47.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:47.007+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:47.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:47.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:47.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:47.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:47.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:47.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:47.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:47.201+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:48.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:47.942+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:48.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:47.942+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:48.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:47.942+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:48.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:47.942+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:48.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:48.186+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:48.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:48.186+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:48.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:48.186+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:48.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:48.187+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:48.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:47.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:48.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:47.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:48.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:47.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:48.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:47.966+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:49.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:48.945+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:49.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:48.945+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:49.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:48.945+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:49.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:48.945+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:49.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:48.949+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:49.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:48.949+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:49.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:48.949+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:49.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:48.949+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:49.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:49.149+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:49.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:49.149+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:49.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:49.149+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:49.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:49.150+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:50.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:50.130+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:50.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:50.130+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:50.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:50.130+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:50.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:50.131+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:50.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:49.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:50.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:49.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:50.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:49.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:50.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:49.910+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:50.152 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:49.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:50.152 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:49.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:50.152 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:49.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:50.152 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:49.969+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:51.131 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:50.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:51.131 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:50.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:51.131 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:50.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:51.132 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:50.968+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:51.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:50.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:51.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:50.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:51.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:50.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:51.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:50.901+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:51.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:28:51.186+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20374 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:28:51.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:51.155+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:51.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:51.155+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:51.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:51.155+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:51.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:51.156+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:52.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:51.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:52.162 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:51.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:52.162 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:51.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:52.162 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:51.971+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:52.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:51.876+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:52.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:51.876+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:52.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:51.876+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:52.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:51.877+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:52.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:52.186+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:52.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:52.186+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:52.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:52.186+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:52.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:52.187+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:53.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:52.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:53.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:52.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:53.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:52.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:53.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:52.871+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:53.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:52.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:53.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:52.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:53.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:52.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:53.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:52.934+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:53.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:53.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:53.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:53.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:53.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:53.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:53.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:53.208+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:53.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:53.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:53.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:53.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:53.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:53.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:53.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:53.849+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:54.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:53.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:54.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:53.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:54.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:53.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:54.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:53.940+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:54.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:54.240+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:54.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:54.240+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:54.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:54.240+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:54.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:54.241+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:55.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:54.950+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:55.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:54.950+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:55.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:54.950+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:55.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:54.951+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:55.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:54.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:55.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:54.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:55.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:54.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:55.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:54.895+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:55.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:55.288+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:55.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:55.288+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:55.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:55.288+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:55.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:55.290+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:56.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:55.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:56.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:55.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:56.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:55.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:56.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:55.885+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:56.164 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:55.962+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:56.164 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:55.962+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:56.164 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:55.962+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:56.164 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:55.963+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:56.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:28:56.188+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20464 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:28:56.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:56.290+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:56.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:56.290+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:56.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:56.290+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:56.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:56.291+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:57.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:56.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:57.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:56.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:57.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:56.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:57.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:56.973+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:57.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:56.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:57.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:56.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:57.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:56.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:57.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:56.879+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:57.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:57.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:57.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:57.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:57.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:57.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:57.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:57.264+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:58.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:57.953+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:58.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:57.953+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:58.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:57.953+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:58.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:57.954+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:58.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:57.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:58.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:57.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:58.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:57.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:58.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:57.864+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:58.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:58.233+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:58.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:58.233+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:58.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:58.233+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:58.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:58.234+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:59.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:58.965+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:59.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:58.965+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:59.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:58.965+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:28:59.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:58.966+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:59.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:58.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:59.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:58.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:59.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:58.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:28:59.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:58.902+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:28:59.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:59.207+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:59.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:59.207+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:59.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:59.207+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:28:59.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:28:59.208+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:00.152 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:59.917+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:00.152 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:59.917+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:00.152 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:59.917+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:00.152 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:28:59.918+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:00.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:59.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:00.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:59.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:00.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:59.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:00.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:28:59.944+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:00.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:00.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:00.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:00.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:00.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:00.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:00.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:00.259+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:01.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:00.926+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:01.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:00.926+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:01.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:00.926+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:01.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:00.927+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:01.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:00.926+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:01.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:00.926+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:01.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:00.926+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:01.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:00.927+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:01.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:29:01.190+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20553 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:29:01.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:01.306+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:01.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:01.306+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:01.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:01.306+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:01.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:01.306+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:02.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:01.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:02.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:01.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:02.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:01.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:02.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:01.965+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:02.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:01.896+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:02.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:01.896+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:02.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:01.896+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:02.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:01.897+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:02.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:02.276+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:02.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:02.276+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:02.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:02.276+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:02.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:02.276+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:02.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:02.944+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:02.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:02.944+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:02.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:02.944+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:02.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:02.946+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:02.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:02.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:02.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:02.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:02.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:02.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:02.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:02.898+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:03.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:03.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:03.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:03.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:03.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:03.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:03.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:03.298+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:04.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:03.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:04.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:03.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:04.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:03.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:04.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:03.853+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:04.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:03.914+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:04.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:03.914+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:04.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:03.914+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:04.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:03.915+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:04.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:04.315+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:04.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:04.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:04.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:04.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:04.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:04.316+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:05.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:04.854+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:05.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:04.854+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:05.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:04.854+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:05.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:04.855+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:05.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:04.871+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:05.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:04.871+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:05.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:04.871+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:05.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:04.872+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:05.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:05.295+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:05.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:05.295+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:05.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:05.295+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:05.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:05.296+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:06.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:05.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:06.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:05.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:06.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:05.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:06.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:05.852+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:06.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:05.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:06.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:05.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:06.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:05.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:06.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:05.864+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:06.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:29:06.192+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20642 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:29:06.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:06.267+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:06.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:06.268+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:06.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:06.268+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:06.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:06.269+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:07.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:06.840+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:07.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:06.840+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:07.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:06.840+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:07.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:06.841+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:07.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:06.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:07.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:06.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:07.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:06.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:07.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:06.895+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:07.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:07.294+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:07.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:07.294+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:07.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:07.294+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:07.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:07.295+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:08.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:07.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:08.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:07.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:08.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:07.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:08.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:07.895+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:08.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:07.811+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:08.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:07.811+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:08.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:07.811+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:08.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:07.811+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:08.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:08.260+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:08.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:08.260+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:08.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:08.260+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:08.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:08.261+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:09.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:08.855+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:09.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:08.855+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:09.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:08.855+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:09.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:08.855+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:09.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:08.876+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:09.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:08.876+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:09.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:08.876+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:09.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:08.876+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:09.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:09.276+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:09.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:09.276+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:09.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:09.276+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:09.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:09.277+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:10.153 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:09.819+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:10.153 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:09.819+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:10.153 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:09.819+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:10.153 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:09.820+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:10.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:09.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:10.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:09.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:10.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:09.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:10.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:09.909+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:10.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:10.249+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:10.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:10.249+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:10.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:10.249+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:10.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:10.250+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:11.170 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:10.862+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:11.170 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:10.862+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:11.170 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:10.862+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:11.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:10.863+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:11.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:10.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:11.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:10.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:11.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:10.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:11.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:10.935+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:11.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:29:11.194+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20731 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:29:11.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:11.209+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:11.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:11.209+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:11.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:11.209+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:11.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:11.210+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:12.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:11.862+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:12.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:11.862+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:12.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:11.862+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:12.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:11.863+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:12.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:11.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:12.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:11.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:12.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:11.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:12.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:11.946+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:12.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:12.238+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:12.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:12.238+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:12.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:12.238+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:12.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:12.239+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:13.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:12.842+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:13.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:12.842+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:13.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:12.842+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:13.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:12.843+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:13.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:12.970+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:13.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:12.970+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:13.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:12.970+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:13.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:12.970+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:13.578 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:13.278+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:13.578 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:13.278+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:13.578 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:13.278+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:13.578 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:13.279+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:13.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:13.832+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:13.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:13.832+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:13.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:13.832+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:13.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:13.833+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:14.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:13.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:14.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:13.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:14.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:13.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:14.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:13.995+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:14.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:14.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:14.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:14.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:14.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:14.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:14.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:14.309+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:15.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:14.791+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:15.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:14.791+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:15.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:14.791+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:15.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:14.792+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:15.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:14.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:15.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:14.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:15.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:14.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:15.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:14.982+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:15.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:15.285+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:15.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:15.285+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:15.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:15.285+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:15.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:15.285+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:16.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:15.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:16.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:15.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:16.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:15.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:16.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:15.999+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:16.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:15.840+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:16.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:15.840+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:16.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:15.840+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:16.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:15.840+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:16.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:29:16.197+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20819 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:29:16.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:16.240+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:16.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:16.240+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:16.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:16.240+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:16.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:16.241+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:17.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:17.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:17.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:17.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:17.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:17.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:17.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:17.028+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:17.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:16.888+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:17.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:16.888+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:17.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:16.888+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:17.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:16.889+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:17.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:17.248+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:17.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:17.248+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:17.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:17.248+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:17.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:17.249+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:18.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:18.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:18.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:18.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:18.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:18.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:18.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:18.069+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:18.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:17.862+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:18.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:17.862+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:18.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:17.862+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:18.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:17.863+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:18.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:18.286+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:18.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:18.286+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:18.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:18.286+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:18.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:18.287+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:19.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:18.883+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:19.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:18.883+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:19.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:18.883+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:19.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:18.884+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:19.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:19.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:19.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:19.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:19.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:19.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:19.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:19.096+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:19.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:19.244+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:19.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:19.244+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:19.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:19.244+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:19.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:19.245+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:20.153 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:19.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:20.153 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:19.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:20.153 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:19.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:20.154 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:19.886+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:20.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:20.134+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:20.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:20.134+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:20.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:20.134+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:20.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:20.135+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:20.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:20.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:20.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:20.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:20.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:20.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:20.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:20.265+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:21.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:20.858+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:21.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:20.858+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:21.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:20.858+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:21.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:20.859+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:21.103 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:21.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:21.103 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:21.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:21.103 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:21.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:21.104 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:21.105+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:21.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:29:21.199+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20905 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:29:21.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:21.227+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:21.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:21.228+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:21.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:21.228+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:21.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:21.228+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:22.058 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:21.906+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:22.058 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:21.906+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:22.058 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:21.906+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:22.058 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:21.907+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:22.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:22.210+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:22.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:22.210+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:22.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:22.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:22.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:22.211+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:22.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:22.082+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:22.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:22.082+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:22.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:22.082+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:22.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:22.083+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:23.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:22.917+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:23.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:22.917+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:23.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:22.917+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:23.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:22.918+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:23.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:23.112+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:23.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:23.112+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:23.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:23.112+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:23.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:23.113+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:23.578 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:23.225+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:23.578 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:23.225+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:23.578 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:23.225+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:23.579 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:23.226+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:24.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:24.081+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:24.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:24.081+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:24.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:24.081+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:24.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:24.081+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:24.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:23.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:24.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:23.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:24.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:23.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:24.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:23.969+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:24.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:24.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:24.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:24.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:24.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:24.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:24.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:24.258+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:25.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:25.229+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:25.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:25.229+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:25.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:25.229+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:25.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:25.229+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:25.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:25.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:25.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:25.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:25.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:25.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:25.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:25.111+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:25.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:25.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:25.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:25.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:25.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:25.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:25.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:25.016+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:26.197 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:29:26.201+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 20992 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:29:26.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:26.221+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:26.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:26.221+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:26.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:26.221+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:26.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:25.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:26.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:25.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:26.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:25.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:26.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:25.972+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:26.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:26.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:26.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:26.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:26.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:26.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:26.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:26.147+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:26.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:26.222+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:27.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:27.166+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:27.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:27.166+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:27.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:27.166+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:27.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:27.166+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:27.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:26.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:27.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:26.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:27.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:26.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:27.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:26.945+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:27.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:27.198+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:27.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:27.198+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:27.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:27.198+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:27.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:27.199+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:28.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:28.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:28.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:28.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:28.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:28.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:28.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:28.119+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:28.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:28.195+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:28.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:28.195+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:28.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:28.195+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:28.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:28.196+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:28.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:27.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:28.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:27.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:28.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:27.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:28.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:27.967+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:29.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:29.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:29.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:29.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:29.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:29.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:29.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:29.106+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:29.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:28.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:29.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:28.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:29.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:28.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:29.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:28.956+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:29.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:29.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:29.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:29.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:29.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:29.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:29.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:29.222+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:30.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:29.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:30.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:29.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:30.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:29.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:30.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:29.933+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:30.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:30.089+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:30.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:30.089+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:30.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:30.089+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:30.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:30.089+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:30.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:30.230+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:30.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:30.230+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:30.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:30.230+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:30.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:30.231+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:31.189 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:29:31.202+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21072 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:29:31.189 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:30.953+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:31.189 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:30.953+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:31.189 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:30.953+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:31.189 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:30.953+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:31.190 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:31.114+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:31.190 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:31.114+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:31.190 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:31.114+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:31.190 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:31.114+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:31.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:31.246+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:31.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:31.246+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:31.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:31.246+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:31.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:31.247+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:32.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:32.125+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:32.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:32.125+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:32.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:32.125+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:32.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:32.126+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:32.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:31.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:32.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:31.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:32.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:31.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:32.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:31.991+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:32.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:32.243+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:32.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:32.243+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:32.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:32.243+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:32.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:32.243+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:33.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:33.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:33.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:33.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:33.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:33.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:33.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:33.011+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:33.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:33.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:33.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:33.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:33.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:33.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:33.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:33.149+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:33.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:33.213+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:33.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:33.213+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:33.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:33.213+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:33.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:33.213+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:34.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:34.207+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:34.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:34.207+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:34.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:34.207+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:34.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:34.207+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:34.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:33.970+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:34.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:33.970+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:34.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:33.970+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:34.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:33.970+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:34.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:34.196+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:34.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:34.196+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:34.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:34.196+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:34.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:34.196+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:35.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:34.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:35.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:34.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:35.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:34.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:35.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:34.937+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:35.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:35.253+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:35.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:35.253+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:35.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:35.253+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:35.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:35.253+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:35.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:35.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:35.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:35.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:35.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:35.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:35.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:35.220+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:36.204 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:29:36.205+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21154 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:29:36.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:35.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:36.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:35.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:36.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:35.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:36.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:35.973+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:36.484 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:36.238+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:36.484 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:36.238+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:36.484 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:36.238+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:36.484 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:36.238+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:36.484 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:36.268+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:36.485 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:36.268+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:36.485 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:36.268+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:36.485 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:36.269+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:37.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:37.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:37.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:37.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:37.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:37.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:37.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:37.203+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:37.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:36.958+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:37.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:36.958+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:37.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:36.958+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:37.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:36.958+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:37.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:37.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:37.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:37.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:37.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:37.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:37.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:37.226+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:38.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:38.183+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:38.217 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:38.183+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:38.217 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:38.183+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:38.217 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:38.183+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:38.217 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:37.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:38.217 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:37.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:38.217 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:37.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:38.217 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:37.984+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:38.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:38.240+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:38.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:38.240+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:38.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:38.240+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:38.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:38.241+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:39.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:38.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:39.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:38.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:39.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:38.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:39.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:38.990+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:39.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:39.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:39.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:39.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:39.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:39.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:39.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:39.223+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:39.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:39.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:39.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:39.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:39.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:39.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:39.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:39.219+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:40.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:39.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:40.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:39.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:40.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:39.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:40.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:39.965+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:40.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:40.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:40.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:40.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:40.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:40.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:40.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:40.200+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:40.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:40.223+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:40.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:40.223+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:40.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:40.223+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:40.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:40.224+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:41.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:40.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:41.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:40.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:41.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:40.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:41.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:40.940+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:41.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:29:41.208+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21237 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:29:41.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:41.214+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:41.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:41.214+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:41.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:41.214+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:41.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:41.215+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:41.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:41.211+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:41.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:41.211+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:41.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:41.211+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:41.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:41.212+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:42.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:41.943+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:42.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:41.943+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:42.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:41.943+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:42.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:41.943+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:42.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:42.254+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:42.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:42.254+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:42.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:42.254+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:42.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:42.255+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:42.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:42.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:42.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:42.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:42.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:42.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:42.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:42.207+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:43.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:43.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:43.254 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:43.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:43.254 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:43.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:43.254 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:43.169+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:43.254 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:42.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:43.254 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:42.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:43.254 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:42.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:43.254 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:42.975+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:43.579 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:43.278+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:43.579 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:43.278+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:43.580 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:43.278+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:43.580 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:43.279+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:44.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:44.200+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:44.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:44.200+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:44.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:44.200+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:44.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:44.200+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:44.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:43.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:44.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:43.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:44.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:43.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:44.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:43.967+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:44.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:44.275+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:44.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:44.275+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:44.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:44.275+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:44.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:44.275+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:45.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:44.949+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:45.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:44.949+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:45.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:44.949+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:45.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:44.949+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:45.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:45.285+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:45.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:45.285+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:45.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:45.285+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:45.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:45.286+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:45.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:45.244+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:45.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:45.244+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:45.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:45.244+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:45.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:45.244+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:46.186 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:45.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:46.186 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:45.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:46.186 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:45.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:46.186 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:45.934+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:46.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:29:46.210+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21321 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:29:46.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:46.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:46.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:46.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:46.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:46.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:46.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:46.270+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:46.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:46.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:46.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:46.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:46.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:46.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:46.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:46.285+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:47.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:46.963+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:47.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:46.963+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:47.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:46.963+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:47.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:46.964+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:47.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:47.273+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:47.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:47.273+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:47.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:47.273+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:47.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:47.273+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:47.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:47.286+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:47.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:47.286+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:47.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:47.286+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:47.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:47.286+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:48.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:47.944+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:48.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:47.944+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:48.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:47.944+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:48.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:47.945+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:48.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:48.318+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:48.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:48.318+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:48.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:48.318+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:48.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:48.318+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:48.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:48.258+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:48.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:48.258+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:48.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:48.258+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:48.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:48.259+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:49.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:48.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:49.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:48.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:49.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:48.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:49.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:48.948+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:49.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:49.271+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:49.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:49.271+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:49.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:49.271+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:49.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:49.272+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:49.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:49.250+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:49.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:49.250+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:49.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:49.250+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:49.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:49.250+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:50.155 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:49.908+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:50.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:49.908+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:50.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:49.908+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:50.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:49.909+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:50.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:50.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:50.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:50.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:50.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:50.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:50.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:50.285+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:50.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:50.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:50.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:50.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:50.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:50.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:50.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:50.248+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:51.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:50.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:51.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:50.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:51.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:50.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:51.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:50.888+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:51.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:29:51.213+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21407 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:29:51.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:51.273+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:51.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:51.273+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:51.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:51.273+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:51.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:51.273+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:51.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:51.248+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:51.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:51.248+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:51.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:51.248+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:51.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:51.248+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:52.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:51.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:52.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:51.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:52.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:51.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:52.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:51.891+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:52.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:52.248+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:52.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:52.248+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:52.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:52.248+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:52.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:52.249+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:52.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:52.251+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:52.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:52.251+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:52.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:52.251+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:52.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:52.251+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:53.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:52.896+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:53.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:52.896+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:53.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:52.896+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:53.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:52.898+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:53.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:53.277+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:53.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:53.277+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:53.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:53.277+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:53.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:53.277+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:53.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:53.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:53.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:53.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:53.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:53.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:53.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:53.209+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:54.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:53.909+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:54.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:53.909+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:54.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:53.909+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:54.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:53.910+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:54.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:54.272+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:54.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:54.272+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:54.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:54.272+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:54.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:54.273+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:54.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:54.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:54.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:54.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:54.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:54.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:54.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:54.254+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:55.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:54.863+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:55.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:54.863+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:55.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:54.863+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:55.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:54.864+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:55.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:55.227+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:55.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:55.227+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:55.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:55.227+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:55.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:55.228+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:55.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:55.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:55.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:55.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:55.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:55.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:55.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:55.303+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:56.190 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:55.869+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:56.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:55.869+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:56.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:55.869+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:56.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:55.870+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:56.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:29:56.215+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21498 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:29:56.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:56.275+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:56.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:56.275+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:56.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:56.275+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:56.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:56.276+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:56.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:56.304+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:56.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:56.304+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:56.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:56.304+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:56.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:56.304+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:57.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:56.857+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:57.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:56.857+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:57.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:56.857+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:57.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:56.857+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:57.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:57.322+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:57.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:57.322+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:57.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:57.322+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:57.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:57.323+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:57.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:57.277+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:57.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:57.277+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:57.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:57.277+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:57.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:57.277+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:58.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:57.868+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:58.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:57.868+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:58.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:57.868+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:58.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:57.869+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:58.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:58.242+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:58.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:58.242+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:58.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:58.242+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:58.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:58.242+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:58.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:58.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:58.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:58.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:58.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:58.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:58.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:58.311+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:59.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:58.840+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:59.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:58.840+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:59.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:58.840+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:29:59.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:58.841+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:59.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:59.272+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:59.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:59.272+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:59.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:59.272+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:29:59.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:29:59.272+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:29:59.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:59.354+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:59.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:59.354+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:59.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:59.354+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:29:59.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:29:59.355+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:00.155 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:59.801+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:00.155 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:59.801+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:00.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:59.801+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:00.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:29:59.801+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:00.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:00.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:00.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:00.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:00.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:00.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:00.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:00.256+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:00.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:00.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:00.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:00.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:00.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:00.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:00.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:00.359+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:01.060 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:00.810+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:01.060 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:00.810+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:01.060 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:00.810+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:01.060 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:00.810+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:01.317 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:30:01.217+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21587 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:30:01.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:01.312+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:01.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:01.312+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:01.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:01.312+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:01.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:01.313+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:01.318 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:01.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:01.318 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:01.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:01.318 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:01.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:01.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:01.284+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:02.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:01.802+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:02.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:01.802+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:02.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:01.802+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:02.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:01.803+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:02.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:02.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:02.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:02.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:02.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:02.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:02.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:02.330+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:02.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:02.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:02.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:02.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:02.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:02.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:02.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:02.331+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:03.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:02.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:03.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:02.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:03.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:02.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:03.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:02.828+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:03.580 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:03.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:03.581 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:03.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:03.581 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:03.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:03.581 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:03.292+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:03.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:03.332+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:03.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:03.332+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:03.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:03.332+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:03.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:03.332+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:03.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:03.788+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:03.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:03.788+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:03.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:03.788+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:03.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:03.789+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:04.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:04.246+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:04.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:04.246+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:04.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:04.246+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:04.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:04.247+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:04.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:04.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:04.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:04.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:04.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:04.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:04.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:04.381+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:05.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:04.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:05.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:04.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:05.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:04.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:05.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:04.825+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:05.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:05.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:05.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:05.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:05.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:05.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:05.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:05.396+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:05.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:05.268+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:05.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:05.268+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:05.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:05.268+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:05.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:05.269+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:06.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:05.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:06.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:05.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:06.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:05.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:06.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:05.828+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:06.479 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:30:06.220+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21677 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:30:06.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:06.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:06.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:06.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:06.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:06.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:06.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:06.353+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:06.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:06.285+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:06.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:06.285+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:06.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:06.285+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:06.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:06.286+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:07.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:06.823+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:07.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:06.823+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:07.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:06.823+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:07.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:06.824+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:07.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:07.400+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:07.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:07.400+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:07.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:07.400+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:07.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:07.401+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:07.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:07.325+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:07.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:07.325+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:07.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:07.325+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:07.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:07.326+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:08.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:07.849+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:08.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:07.849+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:08.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:07.849+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:08.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:07.850+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:08.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:08.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:08.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:08.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:08.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:08.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:08.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:08.394+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:08.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:08.302+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:08.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:08.302+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:08.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:08.302+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:08.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:08.303+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:09.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:08.876+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:09.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:08.876+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:09.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:08.876+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:09.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:08.877+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:09.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:09.357+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:09.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:09.357+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:09.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:09.357+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:09.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:09.358+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:09.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:09.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:09.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:09.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:09.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:09.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:09.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:09.331+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:10.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:09.879+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:10.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:09.879+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:10.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:09.879+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:10.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:09.879+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:10.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:10.300+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:10.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:10.300+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:10.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:10.300+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:10.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:10.300+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:10.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:10.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:10.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:10.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:10.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:10.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:10.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:10.365+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:11.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:10.862+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:11.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:10.862+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:11.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:10.862+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:11.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:10.862+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:11.526 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:30:11.222+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21766 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:30:11.526 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:11.303+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:11.526 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:11.303+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:11.526 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:11.303+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:11.526 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:11.304+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:11.527 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:11.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:11.527 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:11.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:11.527 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:11.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:11.527 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:11.402+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:12.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:11.879+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:12.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:11.879+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:12.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:11.879+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:12.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:11.880+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:12.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:12.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:12.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:12.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:12.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:12.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:12.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:12.419+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:12.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:12.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:12.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:12.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:12.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:12.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:12.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:12.274+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:13.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:12.879+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:13.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:12.879+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:13.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:12.879+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:13.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:12.879+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:13.581 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:13.237+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:13.581 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:13.237+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:13.581 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:13.237+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:13.582 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:13.237+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:13.582 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:13.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:13.582 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:13.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:13.582 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:13.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:13.582 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:13.380+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:14.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:13.881+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:14.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:13.881+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:14.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:13.881+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:14.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:13.881+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:14.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:14.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:14.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:14.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:14.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:14.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:14.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:14.281+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:14.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:14.377+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:14.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:14.377+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:14.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:14.377+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:14.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:14.378+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:15.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:14.905+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:15.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:14.905+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:15.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:14.905+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:15.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:14.906+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:15.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:15.318+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:15.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:15.318+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:15.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:15.318+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:15.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:15.319+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:15.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:15.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:15.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:15.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:15.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:15.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:15.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:15.397+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:16.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:15.878+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:16.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:15.878+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:16.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:15.878+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:16.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:15.879+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:16.561 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:30:16.224+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21857 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:30:16.562 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:16.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:16.562 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:16.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:16.562 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:16.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:16.562 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:16.298+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:16.562 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:16.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:16.562 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:16.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:16.562 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:16.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:16.563 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:16.423+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:17.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:16.876+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:17.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:16.876+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:17.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:16.876+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:17.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:16.877+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:17.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:17.271+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:17.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:17.271+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:17.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:17.271+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:17.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:17.273+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:17.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:17.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:17.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:17.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:17.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:17.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:17.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:17.430+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:18.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:17.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:18.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:17.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:18.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:17.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:18.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:17.894+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:18.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:18.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:18.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:18.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:18.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:18.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:18.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:18.305+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:18.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:18.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:18.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:18.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:18.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:18.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:18.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:18.423+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:19.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:18.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:19.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:18.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:19.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:18.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:19.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:18.875+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:19.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:19.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:19.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:19.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:19.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:19.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:19.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:19.294+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:19.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:19.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:19.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:19.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:19.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:19.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:19.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:19.466+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:20.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:19.891+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:20.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:19.891+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:20.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:19.891+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:20.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:19.891+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:20.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:20.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:20.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:20.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:20.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:20.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:20.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:20.251+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:20.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:20.507+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:20.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:20.507+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:20.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:20.507+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:20.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:20.508+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:21.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:20.923+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:21.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:20.923+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:21.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:20.923+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:21.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:20.924+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:21.600 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:30:21.227+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 21946 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:30:21.601 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:21.282+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:21.601 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:21.282+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:21.601 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:21.282+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:21.601 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:21.283+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:21.601 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:21.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:21.601 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:21.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:21.601 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:21.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:21.601 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:21.459+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:22.250 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:21.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:22.250 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:21.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:22.250 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:21.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:22.250 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:21.973+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:22.564 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:22.275+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:22.564 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:22.275+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:22.564 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:22.275+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:22.564 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:22.276+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:22.564 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:22.457+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:22.564 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:22.457+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:22.564 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:22.457+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:22.564 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:22.457+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:23.245 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:22.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:23.246 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:22.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:23.246 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:22.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:23.246 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:22.997+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:23.582 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:23.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:23.582 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:23.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:23.582 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:23.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:23.582 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:23.272+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:23.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:23.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:23.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:23.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:23.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:23.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:23.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:23.459+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:24.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:24.230+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:24.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:24.230+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:24.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:24.230+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:24.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:24.231+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:24.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:24.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:24.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:24.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:24.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:24.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:24.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:24.015+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:24.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:24.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:24.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:24.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:24.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:24.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:24.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:24.428+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:25.370 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:24.977+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:25.371 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:24.977+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:25.371 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:24.977+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:25.371 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:24.978+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:25.371 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:25.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:25.371 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:25.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:25.371 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:25.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:25.371 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:25.218+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:25.636 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:25.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:25.636 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:25.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:25.636 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:25.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:25.636 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:25.396+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:26.190 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:25.952+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:26.190 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:25.952+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:26.190 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:25.952+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:26.190 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:25.953+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:26.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:30:26.229+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22036 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:30:26.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:26.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:26.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:26.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:26.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:26.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:26.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:26.216+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:26.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:26.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:26.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:26.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:26.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:26.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:26.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:26.444+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:27.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:26.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:27.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:26.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:27.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:26.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:27.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:26.968+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:27.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:27.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:27.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:27.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:27.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:27.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:27.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:27.171+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:27.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:27.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:27.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:27.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:27.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:27.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:27.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:27.484+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:28.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:28.003+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:28.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:28.003+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:28.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:28.003+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:28.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:28.004+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:28.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:28.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:28.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:28.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:28.440 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:28.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:28.440 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:28.165+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:28.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:28.463+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:28.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:28.463+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:28.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:28.463+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:28.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:28.463+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:29.407 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:29.142+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:29.407 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:29.142+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:29.407 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:29.142+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:29.407 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:29.143+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:29.407 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:29.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:29.407 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:29.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:29.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:29.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:29.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:29.040+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:29.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:29.431+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:29.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:29.431+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:29.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:29.431+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:29.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:29.431+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:30.380 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:30.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:30.380 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:30.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:30.380 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:30.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:30.380 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:30.130+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:30.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:30.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:30.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:30.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:30.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:30.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:30.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:30.065+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:30.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:30.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:30.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:30.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:30.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:30.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:30.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:30.404+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:31.330 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:30:31.230+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22125 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:30:31.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:31.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:31.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:31.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:31.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:31.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:31.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:31.084+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:31.331 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:31.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:31.331 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:31.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:31.331 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:31.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:31.332 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:31.024+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:31.682 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:31.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:31.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:31.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:31.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:31.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:31.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:31.406+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:32.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:32.017+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:32.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:32.017+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:32.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:32.017+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:32.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:32.018+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:32.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:32.119+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:32.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:32.119+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:32.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:32.119+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:32.389 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:32.120+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:32.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:32.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:32.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:32.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:32.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:32.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:32.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:32.412+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:33.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:32.983+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:33.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:32.983+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:33.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:32.983+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:33.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:32.984+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:33.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:33.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:33.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:33.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:33.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:33.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:33.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:33.099+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:33.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:33.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:33.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:33.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:33.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:33.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:33.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:33.386+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:34.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:34.079+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:34.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:34.079+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:34.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:34.079+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:34.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:34.080+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:34.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:33.960+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:34.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:33.960+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:34.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:33.960+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:34.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:33.961+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:34.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:34.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:34.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:34.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:34.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:34.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:34.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:34.384+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:35.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:34.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:35.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:34.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:35.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:34.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:35.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:34.985+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:35.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:35.094+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:35.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:35.094+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:35.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:35.094+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:35.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:35.095+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:35.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:35.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:35.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:35.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:35.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:35.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:35.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:35.383+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:36.329 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:30:36.233+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22215 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:30:36.330 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:36.017+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:36.330 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:36.017+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:36.330 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:36.017+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:36.330 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:36.018+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:36.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:36.060+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:36.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:36.060+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:36.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:36.060+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:36.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:36.060+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:36.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:36.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:36.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:36.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:36.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:36.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:36.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:36.395+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:37.067 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:36.977+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:37.067 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:36.977+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:37.067 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:36.977+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:37.067 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:36.978+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:37.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:37.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:37.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:37.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:37.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:37.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:37.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:37.092+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:37.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:37.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:37.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:37.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:37.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:37.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:37.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:37.352+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:38.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:37.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:38.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:37.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:38.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:37.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:38.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:37.971+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:38.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:38.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:38.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:38.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:38.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:38.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:38.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:38.139+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:38.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:38.329+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:38.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:38.329+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:38.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:38.329+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:38.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:38.329+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:39.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:39.157+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:39.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:39.157+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:39.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:39.157+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:39.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:39.158+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:39.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:38.948+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:39.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:38.948+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:39.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:38.948+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:39.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:38.948+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:39.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:39.299+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:39.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:39.299+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:39.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:39.299+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:39.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:39.299+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:40.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:39.909+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:40.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:39.909+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:40.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:39.909+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:40.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:39.910+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:40.191+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:40.191+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:40.191+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:40.192+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:40.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:40.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:40.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:40.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:40.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:40.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:40.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:40.347+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:41.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:40.932+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:41.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:40.932+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:41.193 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:40.932+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:41.193 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:40.932+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:41.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:30:41.236+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22304 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:30:41.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:41.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:41.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:41.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:41.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:41.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:41.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:41.218+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:41.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:41.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:41.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:41.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:41.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:41.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:41.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:41.313+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:42.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:41.952+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:42.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:41.952+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:42.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:41.952+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:42.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:41.953+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:42.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:42.217+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:42.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:42.217+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:42.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:42.217+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:42.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:42.218+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:42.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:42.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:42.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:42.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:42.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:42.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:42.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:42.283+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:43.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:43.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:43.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:43.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:43.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:43.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:43.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:43.003+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:43.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:43.196+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:43.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:43.196+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:43.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:43.196+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:43.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:43.197+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:43.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:43.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:43.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:43.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:43.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:43.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:43.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:43.327+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:44.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:44.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:44.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:44.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:44.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:44.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:44.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:44.027+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:44.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:44.182+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:44.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:44.182+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:44.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:44.182+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:44.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:44.183+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:44.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:44.303+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:44.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:44.303+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:44.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:44.303+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:44.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:44.303+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:45.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:45.184+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:45.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:45.184+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:45.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:45.184+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:45.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:45.185+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:45.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:45.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:45.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:45.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:45.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:45.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:45.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:45.014+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:45.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:45.318+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:45.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:45.318+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:45.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:45.318+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:45.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:45.319+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:46.224 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:30:46.239+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22393 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:30:46.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:45.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:46.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:45.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:46.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:45.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:46.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:45.993+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:46.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:46.168+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:46.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:46.168+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:46.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:46.168+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:46.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:46.169+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:46.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:46.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:46.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:46.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:46.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:46.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:46.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:46.313+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:47.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:47.029+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:47.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:47.029+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:47.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:47.029+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:47.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:47.030+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:47.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:47.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:47.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:47.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:47.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:47.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:47.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:47.135+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:47.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:47.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:47.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:47.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:47.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:47.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:47.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:47.284+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:48.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:48.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:48.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:48.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:48.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:48.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:48.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:48.012+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:48.300 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:48.163+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:48.300 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:48.163+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:48.300 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:48.163+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:48.300 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:48.164+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:48.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:48.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:48.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:48.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:48.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:48.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:48.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:48.325+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:49.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:49.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:49.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:49.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:49.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:49.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:49.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:49.162+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:49.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:49.043+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:49.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:49.044+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:49.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:49.044+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:49.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:49.044+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:49.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:49.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:49.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:49.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:49.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:49.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:49.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:49.280+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:50.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:50.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:50.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:50.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:50.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:50.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:50.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:50.161+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:50.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:50.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:50.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:50.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:50.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:50.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:50.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:50.276+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:50.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:50.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:50.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:50.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:50.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:50.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:50.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:50.061+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:51.338 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:30:51.241+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22482 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:30:51.339 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:51.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:51.339 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:51.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:51.339 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:51.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:51.339 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:51.051+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:51.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:51.259+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:51.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:51.259+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:51.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:51.259+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:51.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:51.259+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:51.340 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:51.205+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:51.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:51.205+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:51.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:51.205+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:51.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:51.205+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:52.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:52.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:52.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:52.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:52.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:52.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:52.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:52.012+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:52.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:52.305+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:52.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:52.305+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:52.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:52.305+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:52.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:52.306+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:52.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:52.206+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:52.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:52.206+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:52.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:52.206+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:52.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:52.207+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:53.241 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:53.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:53.241 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:53.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:53.242 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:53.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:53.242 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:53.001+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:53.242 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:53.248+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:53.242 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:53.248+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:53.242 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:53.248+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:53.242 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:53.249+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:53.584 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:53.266+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:53.584 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:53.266+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:53.584 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:53.266+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:53.584 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:53.267+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:54.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:54.272+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:54.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:54.272+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:54.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:54.272+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:54.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:54.273+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:54.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:54.256+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:54.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:54.256+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:54.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:54.256+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:54.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:54.257+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:54.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:54.045+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:54.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:54.045+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:54.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:54.045+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:54.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:54.046+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:55.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:55.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:55.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:55.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:55.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:55.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:55.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:55.015+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:55.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:55.274+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:55.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:55.274+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:55.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:55.274+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:55.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:55.274+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:55.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:55.226+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:55.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:55.226+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:55.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:55.226+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:55.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:55.226+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:56.241 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:30:56.243+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22572 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:30:56.242 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:56.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:56.242 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:56.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:56.242 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:56.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:56.242 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:56.178+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:56.242 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:56.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:56.242 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:56.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:56.242 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:56.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:56.242 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:56.015+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:56.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:56.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:56.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:56.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:56.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:56.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:56.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:56.307+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:57.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:57.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:57.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:57.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:57.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:57.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:57.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:57.040+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:57.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:57.278+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:57.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:57.278+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:57.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:57.278+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:57.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:57.279+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:57.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:57.213+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:57.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:57.213+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:57.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:57.213+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:57.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:57.213+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:58.248 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:58.019+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:58.248 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:58.019+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:58.248 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:58.019+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:58.248 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:58.020+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:58.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:58.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:58.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:58.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:58.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:58.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:58.249 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:58.215+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:58.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:58.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:58.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:58.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:58.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:58.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:58.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:58.274+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:59.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:58.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:59.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:58.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:59.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:58.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:30:59.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:58.982+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:59.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:59.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:59.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:59.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:59.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:59.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:30:59.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:30:59.180+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:30:59.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:59.254+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:59.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:59.254+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:59.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:59.254+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:30:59.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:30:59.255+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:00.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:59.953+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:00.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:59.953+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:00.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:59.953+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:00.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:30:59.954+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:00.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:00.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:00.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:00.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:00.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:00.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:00.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:00.132+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:00.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:00.267+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:00.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:00.267+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:00.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:00.267+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:00.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:00.268+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:01.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:00.983+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:01.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:00.983+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:01.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:00.983+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:01.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:00.984+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:01.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:01.138+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:01.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:01.138+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:01.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:01.138+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:01.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:01.139+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:01.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:31:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:31:01.246+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22661 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:31:01.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:01.276+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:01.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:01.276+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:01.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:01.276+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:01.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:01.277+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:02.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:02.165+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:02.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:02.165+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:02.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:02.165+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:02.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:02.166+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:02.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:01.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:02.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:01.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:02.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:01.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:02.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:01.960+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:02.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:02.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:02.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:02.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:02.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:02.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:02.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:02.292+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:03.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:02.985+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:03.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:02.985+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:03.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:02.985+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:03.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:02.986+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:03.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:03.164+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:03.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:03.164+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:03.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:03.164+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:03.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:03.164+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:03.585 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:03.254+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:03.585 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:03.254+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:03.585 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:03.254+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:03.585 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:03.254+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:04.260 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:04.029+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:04.260 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:04.029+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:04.261 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:04.029+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:04.261 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:04.029+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:04.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:04.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:04.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:04.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:04.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:04.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:04.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:04.134+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:04.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:04.286+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:04.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:04.286+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:04.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:04.286+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:04.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:04.287+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:05.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:05.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:05.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:05.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:05.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:05.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:05.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:05.148+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:05.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:05.071+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:05.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:05.071+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:05.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:05.071+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:05.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:05.072+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:05.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:05.283+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:05.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:05.283+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:05.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:05.283+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:05.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:05.284+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:06.286 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:31:06.249+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22750 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:31:06.287 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:06.035+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:06.287 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:06.035+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:06.287 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:06.035+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:06.287 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:06.036+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:06.287 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:06.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:06.287 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:06.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:06.288 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:06.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:06.288 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:06.109+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:06.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:06.311+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:06.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:06.311+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:06.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:06.311+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:06.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:06.312+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:07.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:06.993+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:07.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:06.993+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:07.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:06.993+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:07.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:06.993+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:07.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:07.091+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:07.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:07.091+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:07.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:07.091+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:07.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:07.092+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:07.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:07.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:07.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:07.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:07.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:07.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:07.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:07.345+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:08.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:08.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:08.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:08.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:08.025+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:08.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:08.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:08.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:08.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:08.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:08.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:08.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:08.112+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:08.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:08.345+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:08.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:08.345+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:08.646 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:08.345+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:08.646 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:08.346+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:09.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:09.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:09.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:09.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:09.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:09.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:09.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:09.001+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:09.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:09.078+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:09.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:09.078+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:09.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:09.078+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:09.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:09.078+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:09.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:09.299+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:09.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:09.299+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:09.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:09.299+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:09.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:09.300+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:10.256 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:10.098+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:10.257 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:10.098+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:10.257 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:10.098+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:10.257 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:10.099+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:10.257 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:10.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:10.257 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:10.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:10.257 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:10.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:10.257 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:10.015+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:10.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:10.281+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:10.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:10.281+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:10.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:10.281+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:10.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:10.282+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:11.120 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:11.006+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:11.121 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:11.006+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:11.121 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:11.006+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:11.121 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:11.007+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:11.444 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:31:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:31:11.251+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22839 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:31:11.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:11.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:11.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:11.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:11.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:11.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:11.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:11.146+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:11.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:11.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:11.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:11.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:11.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:11.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:11.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:11.315+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:12.255 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:12.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:12.255 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:12.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:12.255 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:12.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:12.255 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:12.140+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:12.255 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:12.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:12.255 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:12.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:12.255 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:12.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:12.255 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:12.010+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:12.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:12.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:12.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:12.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:12.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:12.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:12.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:12.281+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:13.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:13.116+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:13.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:13.116+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:13.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:13.116+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:13.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:13.116+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:13.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:13.246+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:13.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:13.246+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:13.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:13.246+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:13.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:13.247+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:13.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:13.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:13.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:13.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:13.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:13.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:13.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:13.013+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:14.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:14.087+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:14.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:14.087+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:14.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:14.087+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:14.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:14.087+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:14.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:14.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:14.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:14.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:14.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:14.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:14.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:14.034+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:14.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:14.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:14.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:14.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:14.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:14.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:14.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:14.203+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:15.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:15.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:15.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:15.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:15.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:15.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:15.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:15.045+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:15.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:15.063+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:15.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:15.063+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:15.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:15.063+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:15.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:15.065+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:15.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:15.248+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:15.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:15.248+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:15.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:15.248+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:15.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:15.249+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:16.250 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:31:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:31:16.253+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 22929 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:31:16.250 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:16.114+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:16.250 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:16.114+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:16.250 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:16.114+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:16.251 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:16.115+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:16.251 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:16.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:16.251 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:16.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:16.251 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:16.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:16.251 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:16.023+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:16.251 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:16.210+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:16.251 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:16.210+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:16.252 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:16.210+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:16.252 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:16.210+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:17.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:17.157+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:17.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:17.157+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:17.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:17.157+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:17.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:17.158+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:17.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:17.238+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:17.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:17.238+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:17.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:17.238+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:17.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:17.239+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:17.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:17.060+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:17.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:17.060+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:17.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:17.060+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:17.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:17.060+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:18.243 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:18.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:18.244 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:18.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:18.244 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:18.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:18.244 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:18.157+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:18.244 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:18.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:18.244 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:18.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:18.244 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:18.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:18.244 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:18.017+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:18.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:18.268+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:18.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:18.268+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:18.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:18.268+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:18.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:18.269+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:19.288 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:19.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:19.288 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:19.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:19.288 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:19.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:19.288 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:19.204+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:19.288 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:19.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:19.288 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:19.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:19.288 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:19.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:19.288 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:19.006+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:19.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:19.312+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:19.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:19.312+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:19.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:19.312+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:19.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:19.314+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:20.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:20.178+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:20.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:20.178+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:20.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:20.178+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:20.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:20.178+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:20.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:20.016+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:20.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:20.016+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:20.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:20.016+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:20.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:20.017+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:20.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:20.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:20.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:20.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:20.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:20.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:20.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:20.346+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:21.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:21.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:21.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:21.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:21.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:21.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:21.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:21.154+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:21.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:21.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:21.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:21.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:21.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:21.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:21.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:21.065+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:21.554 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:31:21.256+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23018 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:31:21.554 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:21.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:21.554 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:21.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:21.554 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:21.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:21.554 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:21.310+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:22.328 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:22.169+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:22.328 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:22.169+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:22.328 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:22.169+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:22.328 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:22.170+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:22.329 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:22.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:22.329 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:22.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:22.329 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:22.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:22.329 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:22.021+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:22.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:22.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:22.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:22.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:22.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:22.353+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:22.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:22.353+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:23.326 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:23.189+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:23.326 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:23.189+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:23.326 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:23.189+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:23.326 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:23.190+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:23.326 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:22.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:23.327 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:22.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:23.327 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:22.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:23.327 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:22.976+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:23.586 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:23.351+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:23.586 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:23.351+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:23.586 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:23.351+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:23.586 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:23.352+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:24.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:23.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:24.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:23.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:24.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:23.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:24.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:23.942+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:24.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:24.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:24.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:24.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:24.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:24.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:24.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:24.194+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:24.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:24.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:24.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:24.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:24.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:24.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:24.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:24.380+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:25.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:24.907+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:25.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:24.907+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:25.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:24.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:25.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:24.908+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:25.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:25.362+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:25.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:25.362+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:25.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:25.362+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:25.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:25.363+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:25.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:25.200+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:25.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:25.201+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:25.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:25.201+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:25.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:25.202+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:26.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:25.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:26.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:25.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:26.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:25.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:26.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:25.895+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:26.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:31:26.258+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23107 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:31:26.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:26.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:26.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:26.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:26.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:26.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:26.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:26.395+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:26.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:26.202+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:26.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:26.202+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:26.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:26.202+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:26.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:26.203+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:27.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:26.906+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:27.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:26.906+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:27.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:26.906+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:27.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:26.906+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:27.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:27.198+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:27.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:27.198+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:27.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:27.198+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:27.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:27.198+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:27.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:27.371+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:27.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:27.371+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:27.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:27.371+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:27.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:27.372+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:28.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:27.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:28.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:27.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:28.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:27.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:28.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:27.929+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:28.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:28.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:28.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:28.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:28.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:28.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:28.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:28.243+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:28.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:28.364+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:28.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:28.364+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:28.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:28.364+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:28.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:28.365+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:29.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:28.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:29.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:28.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:29.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:28.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:29.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:28.937+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:29.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:29.200+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:29.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:29.200+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:29.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:29.200+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:29.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:29.201+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:29.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:29.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:29.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:29.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:29.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:29.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:29.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:29.384+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:30.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:29.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:30.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:29.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:30.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:29.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:30.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:29.930+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:30.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:30.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:30.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:30.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:30.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:30.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:30.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:30.194+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:30.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:30.381+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:30.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:30.381+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:30.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:30.381+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:30.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:30.382+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:31.235 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:30.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:31.235 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:30.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:31.235 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:30.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:31.235 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:30.969+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:31.235 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:31.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:31.235 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:31.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:31.235 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:31.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:31.235 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:31.173+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:31.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:31:31.261+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23196 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:31:31.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:31.338+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:31.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:31.338+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:31.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:31.338+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:31.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:31.339+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:32.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:32.149+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:32.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:32.149+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:32.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:32.149+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:32.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:32.150+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:32.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:31.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:32.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:31.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:32.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:31.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:32.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:31.946+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:32.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:32.341+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:32.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:32.341+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:32.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:32.341+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:32.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:32.341+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:33.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:33.133+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:33.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:33.133+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:33.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:33.133+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:33.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:33.135+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:33.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:32.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:33.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:32.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:33.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:32.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:33.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:32.918+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:33.587 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:33.346+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:33.587 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:33.346+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:33.588 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:33.346+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:33.588 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:33.347+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:34.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:33.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:34.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:33.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:34.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:33.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:34.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:33.912+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:34.138 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:34.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:34.138 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:34.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:34.139 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:34.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:34.139 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:34.162+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:34.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:34.322+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:34.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:34.322+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:34.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:34.322+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:34.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:34.323+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:35.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:35.112+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:35.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:35.112+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:35.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:35.112+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:35.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:35.113+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:35.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:34.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:35.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:34.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:35.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:34.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:35.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:34.953+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:35.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:35.359+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:35.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:35.359+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:35.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:35.359+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:35.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:35.360+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:36.240 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:36.133+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:36.240 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:36.133+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:36.240 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:36.133+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:36.240 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:36.133+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:36.240 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:35.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:36.240 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:35.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:36.240 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:35.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:36.240 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:35.978+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:36.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:31:36.263+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23286 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:31:36.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:36.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:36.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:36.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:36.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:36.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:36.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:36.367+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:37.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:37.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:37.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:37.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:37.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:37.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:37.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:37.160+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:37.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:36.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:37.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:36.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:37.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:36.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:37.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:36.940+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:37.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:37.320+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:37.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:37.320+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:37.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:37.320+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:37.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:37.320+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:38.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:37.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:38.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:37.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:38.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:37.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:38.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:37.894+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:38.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:38.200+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:38.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:38.200+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:38.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:38.200+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:38.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:38.200+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:38.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:38.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:38.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:38.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:38.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:38.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:38.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:38.348+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:39.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:38.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:39.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:38.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:39.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:38.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:39.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:38.918+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:39.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:39.220+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:39.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:39.220+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:39.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:39.220+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:39.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:39.221+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:39.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:39.315+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:39.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:39.315+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:39.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:39.315+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:39.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:39.316+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:40.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:39.876+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:40.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:39.876+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:40.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:39.876+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:40.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:39.876+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:40.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:40.258+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:40.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:40.258+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:40.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:40.258+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:40.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:40.259+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:40.349+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:40.349+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:40.349+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:40.350+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:41.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:40.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:41.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:40.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:41.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:40.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:41.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:40.899+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:41.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:31:41.267+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23375 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:31:41.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:41.301+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:41.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:41.301+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:41.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:41.301+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:41.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:41.302+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:41.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:41.332+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:41.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:41.332+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:41.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:41.332+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:41.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:41.333+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:42.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:41.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:42.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:41.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:42.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:41.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:42.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:41.940+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:42.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:42.326+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:42.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:42.326+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:42.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:42.326+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:42.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:42.327+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:42.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:42.375+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:42.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:42.375+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:42.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:42.375+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:42.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:42.376+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:43.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:42.896+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:43.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:42.896+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:43.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:42.896+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:43.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:42.896+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:43.589 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:43.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:43.589 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:43.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:43.589 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:43.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:43.589 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:43.305+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:43.589 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:43.405+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:43.589 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:43.405+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:43.589 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:43.405+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:43.589 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:43.405+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:44.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:43.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:44.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:43.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:44.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:43.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:44.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:43.867+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:44.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:44.268+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:44.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:44.268+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:44.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:44.268+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:44.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:44.269+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:44.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:44.430+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:44.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:44.430+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:44.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:44.430+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:44.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:44.431+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:45.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:44.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:45.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:44.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:45.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:44.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:45.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:44.824+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:45.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:45.294+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:45.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:45.294+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:45.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:45.294+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:45.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:45.295+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:45.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:45.429+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:45.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:45.429+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:45.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:45.429+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:45.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:45.430+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:46.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:45.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:46.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:45.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:46.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:45.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:46.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:45.863+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:31:46.627+0000 7f770c428700 0 monclient(hunting): authenticate timed out after 300 2024-04-27T02:31:46.602 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:31:46.269+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23464 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:31:46.602 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:46.328+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:46.602 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:46.328+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:46.602 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:46.328+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:46.602 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:46.329+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:46.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:46.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:46.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:46.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:46.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:46.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:46.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:46.412+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:46.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:46.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:46.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:46.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:46.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:46.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:46.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:46.875+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:47.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:47.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:47.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:47.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:47.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:47.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:47.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:47.455+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:47.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:47.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:47.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:47.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:47.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:47.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:47.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:47.322+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:48.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:47.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:48.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:47.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:48.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:47.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:48.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:47.853+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:48.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:48.428+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:48.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:48.428+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:48.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:48.428+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:48.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:48.428+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:48.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:48.287+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:48.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:48.287+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:48.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:48.287+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:48.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:48.287+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:49.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:48.876+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:49.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:48.876+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:49.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:48.876+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:49.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:48.877+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:49.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:49.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:49.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:49.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:49.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:49.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:49.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:49.410+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:49.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:49.288+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:49.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:49.288+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:49.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:49.288+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:49.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:49.289+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:50.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:49.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:50.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:49.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:50.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:49.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:50.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:49.917+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:50.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:50.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:50.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:50.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:50.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:50.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:50.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:50.396+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:50.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:50.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:50.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:50.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:50.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:50.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:50.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:50.253+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:51.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:50.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:51.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:50.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:51.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:50.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:51.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:50.963+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:51.651 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:31:51.272+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23553 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:31:51.652 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:51.222+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:51.652 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:51.222+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:51.652 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:51.222+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:51.652 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:51.223+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:51.652 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:51.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:51.652 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:51.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:51.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:51.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:51.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:51.360+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:52.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:52.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:52.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:52.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:52.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:52.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:52.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:52.177+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:52.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:51.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:52.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:51.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:52.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:51.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:52.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:51.952+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:52.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:52.370+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:52.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:52.370+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:52.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:52.370+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:52.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:52.370+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:53.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:52.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:53.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:52.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:53.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:52.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:53.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:52.930+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:53.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:53.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:53.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:53.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:53.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:53.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:53.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:53.212+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:53.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:53.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:53.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:53.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:53.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:53.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:53.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:53.402+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:54.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:53.942+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:54.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:53.942+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:54.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:53.942+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:54.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:53.942+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:54.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:54.221+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:54.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:54.221+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:54.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:54.221+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:54.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:54.222+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:54.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:54.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:54.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:54.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:54.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:54.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:54.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:54.380+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:55.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:54.945+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:55.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:54.945+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:55.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:54.945+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:55.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:54.945+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:55.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:55.377+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:55.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:55.377+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:55.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:55.377+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:55.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:55.378+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:55.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:55.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:55.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:55.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:55.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:55.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:55.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:55.194+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:56.250 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:56.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:56.250 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:56.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:56.250 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:56.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:56.250 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:56.175+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:56.250 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:55.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:56.250 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:55.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:56.250 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:55.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:56.251 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:55.973+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:56.695 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:31:56.275+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23679 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:31:56.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:56.360+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:56.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:56.360+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:56.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:56.360+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:56.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:56.361+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:57.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:57.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:57.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:57.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:57.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:57.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:57.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:57.184+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:57.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:56.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:57.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:56.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:57.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:56.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:57.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:56.966+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:57.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:57.339+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:57.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:57.339+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:57.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:57.339+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:57.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:57.340+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:58.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:57.928+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:58.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:57.928+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:58.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:57.928+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:58.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:57.929+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:58.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:58.146+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:58.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:58.146+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:58.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:58.146+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:58.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:58.147+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:58.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:58.341+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:58.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:58.341+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:58.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:58.341+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:58.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:58.342+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:59.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:59.115+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:59.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:59.115+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:59.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:59.115+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:31:59.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:31:59.116+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:59.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:58.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:59.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:58.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:59.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:58.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:31:59.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:58.925+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:31:59.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:59.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:59.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:59.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:59.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:59.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:31:59.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:31:59.294+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:00.164 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:00.074+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:00.164 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:00.074+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:00.164 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:00.074+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:00.164 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:00.076+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:00.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:59.902+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:00.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:59.902+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:00.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:59.902+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:00.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:31:59.903+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:00.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:00.255+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:00.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:00.255+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:00.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:00.255+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:00.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:00.256+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:01.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:00.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:01.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:00.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:01.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:00.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:01.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:00.916+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:01.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:01.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:01.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:01.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:01.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:01.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:01.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:01.053+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:01.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:32:01.277+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23767 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:32:01.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:01.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:01.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:01.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:01.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:01.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:01.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:01.257+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:02.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:02.097+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:02.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:02.097+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:02.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:02.097+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:02.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:02.098+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:02.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:01.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:02.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:01.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:02.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:01.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:02.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:01.889+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:02.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:02.259+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:02.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:02.259+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:02.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:02.259+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:02.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:02.260+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:03.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:03.063+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:03.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:03.063+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:03.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:03.063+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:03.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:03.064+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:03.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:02.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:03.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:02.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:03.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:02.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:03.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:02.870+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:03.589 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:03.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:03.590 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:03.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:03.590 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:03.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:03.590 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:03.308+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:03.940 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:03.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:03.940 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:03.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:03.940 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:03.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:03.940 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:03.863+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:04.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:04.112+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:04.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:04.112+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:04.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:04.112+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:04.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:04.113+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:04.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:04.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:04.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:04.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:04.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:04.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:04.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:04.331+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:05.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:05.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:05.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:05.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:05.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:05.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:05.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:05.119+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:05.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:04.909+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:05.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:04.909+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:05.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:04.909+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:05.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:04.909+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:05.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:05.281+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:05.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:05.281+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:05.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:05.281+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:05.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:05.282+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:06.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:06.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:06.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:06.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:06.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:06.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:06.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:06.155+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:06.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:05.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:06.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:05.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:06.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:05.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:06.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:05.941+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:06.695 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:32:06.280+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23857 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:32:06.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:06.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:06.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:06.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:06.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:06.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:06.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:06.270+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:07.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:07.157+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:07.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:07.157+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:07.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:07.157+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:07.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:07.158+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:07.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:06.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:07.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:06.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:07.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:06.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:07.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:06.928+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:07.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:07.244+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:07.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:07.244+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:07.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:07.244+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:07.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:07.246+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:08.175 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:08.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:08.176 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:08.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:08.176 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:08.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:08.176 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:08.124+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:08.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:07.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:08.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:07.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:08.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:07.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:08.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:07.892+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:08.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:08.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:08.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:08.201+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:08.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:08.201+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:08.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:08.202+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:09.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:09.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:09.186 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:09.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:09.186 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:09.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:09.186 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:09.130+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:09.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:08.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:09.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:08.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:09.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:08.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:09.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:08.923+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:09.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:09.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:09.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:09.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:09.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:09.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:09.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:09.211+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:10.178 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:10.142+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:10.178 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:10.142+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:10.178 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:10.142+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:10.178 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:10.143+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:10.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:09.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:10.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:09.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:10.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:09.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:10.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:09.947+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:10.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:10.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:10.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:10.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:10.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:10.203+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:10.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:10.203+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:11.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:11.174+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:11.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:11.174+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:11.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:11.174+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:11.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:11.175+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:11.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:10.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:11.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:10.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:11.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:10.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:11.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:10.959+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:11.695 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:32:11.283+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 23945 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:32:11.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:11.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:11.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:11.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:11.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:11.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:11.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:11.227+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:12.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:12.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:12.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:12.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:12.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:12.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:12.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:12.171+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:12.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:11.959+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:12.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:11.959+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:12.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:11.959+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:12.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:11.960+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:12.505 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:12.275+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:12.505 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:12.275+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:12.505 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:12.275+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:12.505 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:12.275+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:13.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:12.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:13.181 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:12.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:13.181 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:12.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:13.181 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:12.955+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:13.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:13.206+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:13.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:13.206+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:13.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:13.206+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:13.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:13.207+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:13.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:13.245+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:13.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:13.245+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:13.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:13.245+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:13.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:13.246+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:14.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:14.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:14.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:14.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:14.160 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:14.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:14.160 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:14.185+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:14.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:13.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:14.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:13.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:14.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:13.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:14.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:13.935+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:14.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:14.238+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:14.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:14.238+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:14.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:14.238+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:14.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:14.238+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:15.155 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:14.920+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:15.155 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:14.920+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:15.155 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:14.920+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:15.155 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:14.920+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:15.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:15.252+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:15.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:15.252+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:15.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:15.252+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:15.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:15.253+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:15.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:15.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:15.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:15.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:15.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:15.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:15.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:15.181+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:16.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:15.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:16.187 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:15.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:16.187 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:15.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:16.187 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:15.947+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:16.187 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:16.143+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:16.187 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:16.143+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:16.187 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:16.143+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:16.187 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:16.144+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:16.444 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:32:16.285+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24036 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:32:16.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:16.212+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:16.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:16.212+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:16.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:16.212+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:16.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:16.213+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:17.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:17.139+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:17.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:17.139+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:17.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:17.139+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:17.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:17.140+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:17.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:17.178+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:17.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:17.178+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:17.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:17.178+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:17.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:17.178+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:17.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:16.933+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:17.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:16.933+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:17.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:16.933+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:17.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:16.934+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:18.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:18.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:18.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:18.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:18.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:18.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:18.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:18.136+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:18.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:18.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:18.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:18.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:18.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:18.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:18.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:18.131+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:18.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:17.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:18.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:17.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:18.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:17.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:18.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:17.964+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:19.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:19.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:19.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:19.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:19.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:19.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:19.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:19.132+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:19.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:19.115+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:19.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:19.115+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:19.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:19.115+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:19.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:19.115+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:19.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:18.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:19.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:18.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:19.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:18.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:19.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:18.915+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:20.165 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:20.139+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:20.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:20.139+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:20.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:20.139+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:20.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:20.140+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:20.166 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:20.137+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:20.166 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:20.137+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:20.166 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:20.137+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:20.166 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:20.138+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:20.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:19.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:20.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:19.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:20.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:19.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:20.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:19.929+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:21.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:21.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:21.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:21.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:21.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:21.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:21.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:21.118+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:21.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:20.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:21.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:20.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:21.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:20.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:21.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:20.921+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:21.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:21.117+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:21.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:21.117+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:21.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:21.117+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:21.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:21.118+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:21.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:32:21.288+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24124 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:32:22.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:22.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:22.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:22.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:22.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:22.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:22.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:22.163+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:22.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:22.103+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:22.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:22.103+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:22.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:22.103+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:22.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:22.104+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:22.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:21.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:22.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:21.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:22.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:21.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:22.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:21.952+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:23.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:22.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:23.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:22.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:23.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:22.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:23.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:22.931+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:23.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:23.163+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:23.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:23.163+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:23.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:23.163+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:23.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:23.164+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:23.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:23.064+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:23.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:23.064+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:23.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:23.064+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:23.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:23.065+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:24.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:24.140+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:24.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:24.140+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:24.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:24.140+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:24.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:24.141+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:24.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:24.037+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:24.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:24.037+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:24.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:24.037+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:24.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:24.038+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:24.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:23.964+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:24.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:23.964+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:24.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:23.964+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:24.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:23.964+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:25.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:25.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:25.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:25.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:25.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:25.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:25.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:25.001+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:25.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:24.962+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:25.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:24.962+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:25.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:24.962+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:25.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:24.963+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:25.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:25.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:25.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:25.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:25.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:25.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:25.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:25.157+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:26.265 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:26.131+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:26.265 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:26.131+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:26.266 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:26.131+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:26.266 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:26.131+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:26.266 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:26.007+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:26.266 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:26.007+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:26.266 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:26.007+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:26.266 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:26.008+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:26.267 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:26.003+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:26.267 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:26.003+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:26.267 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:26.003+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:26.267 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:26.003+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:26.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:32:26.289+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24214 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:32:27.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:27.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:27.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:27.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:27.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:27.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:27.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:27.136+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:27.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:27.049+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:27.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:27.049+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:27.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:27.049+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:27.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:27.050+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:27.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:26.982+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:27.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:26.982+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:27.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:26.982+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:27.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:26.982+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:28.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:28.115+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:28.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:28.115+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:28.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:28.115+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:28.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:28.116+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:28.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:28.041+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:28.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:28.041+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:28.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:28.041+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:28.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:28.042+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:28.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:28.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:28.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:28.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:28.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:28.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:28.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:28.020+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:29.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:29.036+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:29.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:29.036+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:29.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:29.036+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:29.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:29.037+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:29.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:29.107+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:29.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:29.107+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:29.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:29.107+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:29.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:29.109+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:29.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:28.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:29.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:28.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:29.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:28.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:29.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:29.000+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:30.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:30.074+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:30.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:30.074+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:30.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:30.074+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:30.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:30.075+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:30.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:30.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:30.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:30.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:30.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:30.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:30.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:30.156+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:30.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:30.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:30.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:30.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:30.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:30.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:30.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:30.030+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:31.284 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:32:31.292+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24302 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:32:31.284 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:31.127+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:31.284 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:31.127+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:31.284 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:31.127+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:31.284 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:31.127+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:31.285 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:31.043+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:31.285 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:31.043+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:31.285 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:31.043+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:31.285 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:31.043+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:31.285 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:31.075+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:31.285 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:31.075+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:31.285 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:31.075+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:31.285 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:31.075+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:32.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:32.032+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:32.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:32.032+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:32.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:32.032+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:32.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:32.033+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:32.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:32.125+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:32.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:32.125+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:32.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:32.125+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:32.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:32.126+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:32.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:32.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:32.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:32.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:32.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:32.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:32.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:32.048+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:33.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:32.994+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:33.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:32.994+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:33.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:32.994+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:33.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:32.995+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:33.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:33.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:33.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:33.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:33.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:33.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:33.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:33.106+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:33.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:33.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:33.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:33.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:33.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:33.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:33.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:33.026+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:34.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:33.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:34.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:33.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:34.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:33.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:34.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:33.951+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:34.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:34.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:34.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:34.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:34.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:34.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:34.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:34.153+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:34.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:34.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:34.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:34.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:34.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:34.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:34.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:34.013+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:35.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:35.130+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:35.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:35.130+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:35.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:35.130+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:35.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:35.130+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:35.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:35.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:35.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:35.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:35.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:35.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:35.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:35.029+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:35.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:34.964+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:35.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:34.964+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:35.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:34.964+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:35.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:34.964+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:36.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:36.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:36.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:36.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:36.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:36.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:36.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:36.180+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:36.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:35.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:36.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:35.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:36.272 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:35.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:36.272 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:35.988+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:36.272 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:35.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:36.272 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:35.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:36.272 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:35.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:36.272 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:35.997+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:36.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:32:36.295+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24393 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:32:37.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:36.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:37.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:36.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:37.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:36.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:37.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:36.975+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:37.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:37.198+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:37.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:37.198+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:37.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:37.198+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:37.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:37.199+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:37.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:36.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:37.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:36.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:37.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:36.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:37.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:36.999+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:38.001 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:37.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:38.002 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:37.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:38.002 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:37.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:38.002 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:37.926+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:38.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:38.181+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:38.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:38.181+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:38.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:38.181+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:38.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:38.182+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:38.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:38.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:38.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:38.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:38.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:38.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:38.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:38.027+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:39.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:38.888+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:39.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:38.888+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:39.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:38.888+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:39.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:38.889+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:39.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:38.979+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:39.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:38.979+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:39.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:38.979+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:39.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:38.980+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:39.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:39.223+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:39.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:39.223+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:39.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:39.223+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:39.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:39.224+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:40.166 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:39.939+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:40.167 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:39.939+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:40.167 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:39.939+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:40.167 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:39.940+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:40.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:40.018+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:40.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:40.018+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:40.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:40.018+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:40.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:40.018+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:40.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:40.202+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:40.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:40.202+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:40.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:40.202+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:40.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:40.203+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:41.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:41.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:41.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:41.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:41.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:41.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:41.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:41.002+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:41.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:40.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:41.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:40.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:41.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:40.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:41.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:40.926+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:41.398 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:32:41.299+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24481 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:32:41.398 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:41.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:41.398 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:41.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:41.398 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:41.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:41.398 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:41.161+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:42.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:41.901+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:42.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:41.901+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:42.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:41.901+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:42.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:41.902+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:42.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:41.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:42.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:41.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:42.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:41.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:42.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:41.986+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:42.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:42.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:42.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:42.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:42.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:42.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:42.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:42.117+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:43.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:42.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:43.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:42.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:43.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:42.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:43.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:42.899+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:43.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:42.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:43.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:42.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:43.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:42.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:43.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:42.957+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:43.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:43.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:43.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:43.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:43.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:43.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:43.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:43.076+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:44.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:43.941+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:44.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:43.941+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:44.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:43.941+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:44.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:43.941+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:44.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:43.909+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:44.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:43.909+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:44.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:43.909+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:44.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:43.909+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:44.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:44.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:44.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:44.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:44.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:44.034+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:44.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:44.035+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:45.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:44.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:45.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:44.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:45.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:44.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:45.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:44.980+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:45.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:45.015+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:45.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:45.015+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:45.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:45.015+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:45.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:45.016+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:45.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:44.942+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:45.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:44.942+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:45.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:44.942+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:45.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:44.942+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:46.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:45.948+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:46.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:45.948+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:46.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:45.948+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:46.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:45.949+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:46.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:46.044+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:46.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:46.044+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:46.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:46.044+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:46.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:46.045+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:46.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:45.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:46.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:45.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:46.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:45.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:46.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:45.895+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:46.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:32:46.301+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24571 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:32:47.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:47.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:47.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:47.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:47.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:47.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:47.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:47.090+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:47.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:46.954+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:47.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:46.954+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:47.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:46.954+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:47.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:46.955+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:47.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:46.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:47.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:46.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:47.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:46.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:47.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:46.848+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:48.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:47.906+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:48.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:47.906+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:48.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:47.906+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:48.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:47.907+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:48.042 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:47.853+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:48.042 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:47.853+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:48.042 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:47.853+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:48.042 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:47.854+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:48.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:48.066+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:48.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:48.066+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:48.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:48.066+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:48.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:48.068+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:49.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:49.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:49.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:49.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:49.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:49.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:49.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:49.078+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:49.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:48.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:49.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:48.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:49.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:48.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:49.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:48.894+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:49.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:48.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:49.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:48.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:49.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:48.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:49.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:48.861+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:50.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:50.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:50.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:50.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:50.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:50.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:50.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:50.103+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:50.167 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:49.897+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:50.167 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:49.897+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:50.167 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:49.897+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:50.167 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:49.898+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:50.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:49.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:50.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:49.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:50.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:49.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:50.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:49.930+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:51.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:50.931+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:51.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:50.931+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:51.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:50.931+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:51.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:50.932+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:51.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:50.972+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:51.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:50.972+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:51.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:50.972+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:51.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:50.972+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:51.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:51.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:51.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:51.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:51.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:51.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:51.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:51.117+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:51.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:32:51.304+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24659 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:32:52.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:51.919+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:52.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:51.919+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:52.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:51.919+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:52.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:51.920+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:52.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:52.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:52.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:52.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:52.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:52.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:52.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:52.077+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:52.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:51.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:52.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:51.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:52.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:51.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:52.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:51.994+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:53.092 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:53.056+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:53.092 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:53.056+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:53.092 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:53.056+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:53.092 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:53.057+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:53.092 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:52.893+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:53.092 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:52.893+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:53.092 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:52.893+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:53.092 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:52.894+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:53.093 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:53.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:53.093 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:53.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:53.093 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:53.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:53.093 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:53.038+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:54.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:54.046+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:54.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:54.046+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:54.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:54.046+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:54.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:54.046+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:54.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:54.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:54.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:54.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:54.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:54.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:54.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:54.013+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:54.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:53.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:54.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:53.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:54.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:53.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:54.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:53.930+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:55.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:55.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:55.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:55.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:55.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:55.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:55.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:55.043+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:55.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:54.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:55.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:54.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:55.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:54.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:55.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:54.971+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:55.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:54.951+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:55.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:54.951+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:55.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:54.951+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:55.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:54.952+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:56.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:56.063+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:56.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:56.063+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:56.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:56.063+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:56.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:56.064+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:56.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:55.950+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:56.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:55.950+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:56.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:55.950+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:56.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:55.950+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:56.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:55.954+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:56.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:55.954+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:56.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:55.954+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:56.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:55.955+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:56.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:32:56.307+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24750 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:32:57.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:56.961+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:57.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:56.961+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:57.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:56.961+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:57.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:56.962+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:57.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:57.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:57.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:57.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:57.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:57.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:57.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:57.085+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:57.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:56.948+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:57.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:56.948+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:57.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:56.948+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:57.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:56.949+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:58.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:57.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:58.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:57.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:58.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:57.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:58.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:57.939+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:58.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:57.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:58.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:57.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:58.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:57.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:58.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:57.900+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:58.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:58.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:58.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:58.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:58.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:58.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:58.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:58.044+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:59.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:58.858+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:59.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:58.858+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:59.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:58.858+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:32:59.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:58.859+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:59.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:59.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:59.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:59.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:59.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:59.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:32:59.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:32:59.014+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:32:59.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:58.905+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:59.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:58.905+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:59.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:58.905+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:32:59.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:58.905+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:00.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:59.904+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:00.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:59.904+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:00.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:59.904+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:00.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:32:59.906+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:00.167 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:59.939+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:00.168 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:59.939+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:00.168 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:59.939+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:00.168 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:32:59.940+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:00.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:00.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:00.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:00.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:00.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:00.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:00.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:00.037+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:01.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:01.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:01.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:01.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:01.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:01.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:01.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:01.057+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:01.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:00.912+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:01.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:00.912+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:01.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:00.912+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:01.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:00.913+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:01.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:00.942+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:01.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:00.942+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:01.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:00.942+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:01.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:00.943+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:01.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:33:01.309+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24838 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:33:02.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:01.980+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:02.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:01.980+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:02.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:01.980+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:02.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:01.981+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:02.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:02.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:02.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:02.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:02.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:02.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:02.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:02.031+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:02.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:01.886+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:02.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:01.886+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:02.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:01.886+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:02.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:01.887+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:03.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:02.991+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:03.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:02.991+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:03.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:02.991+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:03.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:02.992+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:03.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:03.062+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:03.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:03.062+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:03.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:03.062+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:03.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:03.062+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:03.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:02.867+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:03.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:02.867+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:03.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:02.867+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:03.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:02.868+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:03.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:03.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:03.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:03.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:03.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:03.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:03.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:03.861+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:04.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:04.031+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:04.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:04.031+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:04.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:04.031+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:04.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:04.032+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:04.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:04.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:04.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:04.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:04.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:04.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:04.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:04.112+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:05.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:05.043+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:05.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:05.043+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:05.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:05.043+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:05.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:05.043+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:05.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:05.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:05.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:05.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:05.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:05.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:05.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:05.124+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:05.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:04.901+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:05.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:04.901+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:05.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:04.901+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:05.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:04.902+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:06.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:06.046+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:06.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:06.046+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:06.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:06.046+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:06.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:06.047+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:06.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:06.091+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:06.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:06.091+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:06.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:06.091+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:06.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:06.091+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:06.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:05.909+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:06.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:05.909+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:06.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:05.909+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:06.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:05.910+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:06.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:33:06.312+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 24928 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:33:07.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:07.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:07.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:07.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:07.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:07.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:07.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:07.044+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:07.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:06.957+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:07.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:06.957+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:07.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:06.957+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:07.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:06.959+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:07.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:07.003+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:07.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:07.003+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:07.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:07.003+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:07.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:07.003+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:08.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:07.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:08.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:07.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:08.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:07.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:08.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:07.955+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:08.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:08.038+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:08.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:08.038+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:08.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:08.038+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:08.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:08.039+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:08.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:08.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:08.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:08.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:08.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:08.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:08.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:08.048+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:09.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:08.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:09.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:08.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:09.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:08.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:09.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:08.982+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:09.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:09.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:09.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:09.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:09.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:09.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:09.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:09.009+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:09.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:09.064+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:09.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:09.064+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:09.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:09.064+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:09.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:09.065+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:10.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:10.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:10.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:10.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:10.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:10.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:10.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:10.025+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:10.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:10.109+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:10.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:10.109+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:10.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:10.109+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:10.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:10.110+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:10.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:10.007+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:10.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:10.007+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:10.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:10.007+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:10.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:10.008+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:11.290 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:11.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:11.290 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:11.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:11.290 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:11.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:11.290 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:11.056+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:11.291 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:11.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:11.291 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:11.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:11.291 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:11.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:11.291 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:11.151+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:11.291 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:11.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:11.291 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:11.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:11.291 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:11.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:11.291 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:11.025+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:11.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:33:11.315+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25016 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:33:12.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:12.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:12.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:12.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:12.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:12.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:12.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:12.091+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:12.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:12.105+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:12.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:12.105+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:12.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:12.105+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:12.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:12.106+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:12.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:12.018+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:12.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:12.018+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:12.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:12.018+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:12.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:12.019+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:13.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:13.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:13.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:13.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:13.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:13.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:13.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:13.141+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:13.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:13.132+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:13.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:13.132+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:13.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:13.132+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:13.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:13.134+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:13.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:13.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:13.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:13.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:13.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:13.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:13.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:13.032+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:14.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:14.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:14.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:14.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:14.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:14.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:14.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:14.129+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:14.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:14.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:14.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:14.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:14.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:14.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:14.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:14.125+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:14.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:14.011+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:14.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:14.011+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:14.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:14.011+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:14.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:14.011+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:15.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:15.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:15.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:15.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:15.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:15.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:15.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:15.132+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:15.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:15.126+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:15.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:15.126+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:15.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:15.126+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:15.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:15.127+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:15.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:14.986+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:15.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:14.986+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:15.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:14.986+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:15.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:14.987+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:16.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:16.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:16.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:16.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:16.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:16.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:16.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:16.119+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:16.293 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:15.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:16.293 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:15.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:16.293 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:15.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:16.293 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:15.995+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:16.294 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:16.179+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:16.294 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:16.179+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:16.294 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:16.179+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:16.294 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:16.179+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:16.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:33:16.318+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25107 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:33:17.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:17.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:17.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:17.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:17.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:17.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:17.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:17.041+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:17.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:17.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:17.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:17.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:17.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:17.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:17.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:17.085+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:17.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:17.136+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:17.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:17.136+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:17.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:17.136+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:17.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:17.137+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:18.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:18.183+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:18.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:18.183+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:18.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:18.183+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:18.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:18.183+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:18.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:18.051+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:18.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:18.051+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:18.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:18.051+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:18.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:18.052+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:18.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:18.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:18.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:18.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:18.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:18.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:18.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:18.083+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:19.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:19.049+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:19.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:19.049+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:19.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:19.049+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:19.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:19.050+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:19.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:19.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:19.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:19.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:19.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:19.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:19.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:19.014+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:19.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:19.210+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:19.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:19.210+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:19.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:19.210+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:19.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:19.211+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:20.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:20.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:20.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:20.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:20.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:20.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:20.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:20.062+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:20.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:20.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:20.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:20.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:20.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:20.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:20.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:20.215+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:20.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:20.060+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:20.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:20.060+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:20.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:20.060+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:20.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:20.061+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:21.297 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:21.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:21.297 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:21.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:21.297 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:21.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:21.297 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:21.076+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:21.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:21.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:21.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:21.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:21.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:21.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:21.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:21.218+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:21.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:21.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:21.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:21.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:21.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:21.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:21.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:21.062+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:21.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:33:21.321+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25195 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:33:22.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:22.109+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:22.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:22.109+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:22.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:22.109+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:22.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:22.110+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:22.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:22.058+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:22.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:22.058+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:22.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:22.058+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:22.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:22.060+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:22.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:22.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:22.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:22.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:22.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:22.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:22.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:22.212+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:23.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:23.137+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:23.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:23.137+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:23.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:23.137+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:23.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:23.137+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:23.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:23.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:23.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:23.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:23.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:23.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:23.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:23.076+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:23.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:23.184+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:23.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:23.184+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:23.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:23.184+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:23.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:23.185+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:24.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:24.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:24.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:24.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:24.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:24.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:24.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:24.118+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:24.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:24.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:24.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:24.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:24.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:24.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:24.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:24.034+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:24.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:24.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:24.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:24.216+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:24.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:24.216+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:24.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:24.216+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:25.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:25.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:25.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:25.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:25.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:25.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:25.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:25.077+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:25.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:25.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:25.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:25.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:25.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:25.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:25.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:25.062+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:25.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:25.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:25.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:25.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:25.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:25.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:25.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:25.239+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:26.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:26.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:26.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:26.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:26.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:26.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:26.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:26.053+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:26.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:26.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:26.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:26.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:26.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:26.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:26.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:26.044+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:26.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:26.219+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:26.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:26.219+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:26.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:26.219+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:26.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:26.220+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:26.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:33:26.322+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25285 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:33:27.160 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:27.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:27.160 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:27.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:27.160 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:27.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:27.160 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:27.056+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:27.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:26.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:27.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:26.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:27.161 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:26.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:27.161 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:27.000+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:27.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:27.184+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:27.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:27.184+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:27.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:27.184+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:27.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:27.185+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:28.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:28.068+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:28.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:28.068+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:28.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:28.068+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:28.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:28.069+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:28.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:28.212+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:28.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:28.212+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:28.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:28.212+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:28.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:28.213+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:28.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:28.003+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:28.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:28.003+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:28.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:28.003+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:28.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:28.004+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:29.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:29.181+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:29.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:29.181+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:29.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:29.181+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:29.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:29.182+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:29.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:29.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:29.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:29.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:29.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:29.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:29.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:29.013+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:29.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:29.057+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:29.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:29.058+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:29.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:29.058+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:29.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:29.059+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:30.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:30.070+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:30.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:30.070+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:30.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:30.070+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:30.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:30.071+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:30.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:30.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:30.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:30.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:30.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:30.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:30.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:30.223+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:30.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:29.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:30.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:29.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:30.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:29.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:30.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:29.991+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:31.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:31.069+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:31.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:31.069+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:31.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:31.069+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:31.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:31.070+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:31.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:30.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:31.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:30.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:31.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:30.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:31.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:30.947+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:31.425 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:33:31.325+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25367 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:33:31.426 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:31.197+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:31.426 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:31.197+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:31.426 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:31.197+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:31.426 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:31.198+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:32.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:31.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:32.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:31.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:32.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:31.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:32.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:31.995+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:32.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:32.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:32.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:32.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:32.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:32.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:32.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:32.117+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:32.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:32.234+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:32.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:32.234+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:32.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:32.234+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:32.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:32.235+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:33.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:32.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:33.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:32.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:33.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:32.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:33.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:32.970+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:33.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:33.137+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:33.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:33.137+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:33.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:33.137+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:33.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:33.138+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:33.595 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:33.234+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:33.595 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:33.234+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:33.595 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:33.234+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:33.595 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:33.235+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:34.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:34.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:34.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:34.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:34.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:34.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:34.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:34.136+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:34.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:33.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:34.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:33.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:34.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:33.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:34.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:33.941+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:34.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:34.220+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:34.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:34.220+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:34.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:34.220+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:34.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:34.221+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:35.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:34.906+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:35.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:34.906+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:35.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:34.906+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:35.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:34.907+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:35.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:35.178+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:35.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:35.178+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:35.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:35.178+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:35.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:35.178+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:35.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:35.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:35.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:35.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:35.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:35.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:35.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:35.250+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:36.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:35.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:36.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:35.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:36.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:35.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:36.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:35.887+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:36.429 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:33:36.329+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25458 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:33:36.429 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:36.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:36.429 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:36.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:36.429 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:36.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:36.429 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:36.176+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:36.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:36.244+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:36.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:36.244+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:36.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:36.244+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:36.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:36.245+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:37.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:36.872+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:37.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:36.872+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:37.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:36.872+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:37.109 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:36.873+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:37.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:37.134+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:37.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:37.134+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:37.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:37.134+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:37.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:37.135+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:37.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:37.244+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:37.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:37.244+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:37.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:37.244+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:37.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:37.244+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:38.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:37.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:38.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:37.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:38.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:37.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:38.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:37.836+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:38.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:38.157+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:38.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:38.157+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:38.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:38.157+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:38.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:38.158+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:38.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:38.212+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:38.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:38.212+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:38.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:38.212+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:38.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:38.213+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:39.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:38.830+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:39.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:38.830+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:39.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:38.830+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:39.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:38.830+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:39.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:39.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:39.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:39.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:39.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:39.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:39.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:39.212+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:39.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:39.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:39.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:39.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:39.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:39.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:39.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:39.160+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:40.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:39.879+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:40.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:39.879+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:40.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:39.879+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:40.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:39.879+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:40.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:40.157+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:40.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:40.157+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:40.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:40.157+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:40.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:40.158+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:40.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:40.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:40.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:40.257+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:41.118 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:40.840+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:41.118 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:40.840+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:41.118 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:40.840+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:41.118 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:40.841+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:41.428 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:33:41.332+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25546 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:33:41.429 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:41.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:41.429 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:41.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:41.429 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:41.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:41.429 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:41.293+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:41.429 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:41.143+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:41.429 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:41.143+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:41.429 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:41.143+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:41.429 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:41.144+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:42.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:41.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:42.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:41.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:42.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:41.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:42.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:41.851+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:42.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:42.288+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:42.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:42.288+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:42.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:42.288+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:42.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:42.289+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:42.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:42.179+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:42.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:42.179+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:42.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:42.179+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:42.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:42.181+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:43.127 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:42.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:43.127 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:42.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:43.127 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:42.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:43.127 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:42.852+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:43.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:43.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:43.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:43.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:43.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:43.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:43.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:43.153+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:43.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:43.266+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:43.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:43.266+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:43.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:43.266+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:43.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:43.266+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:43.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:43.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:43.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:43.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:43.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:43.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:43.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:43.864+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:44.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:44.278+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:44.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:44.278+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:44.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:44.278+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:44.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:44.279+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:44.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:44.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:44.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:44.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:44.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:44.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:44.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:44.148+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:45.078 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:44.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:45.079 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:44.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:45.079 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:44.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:45.079 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:44.819+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:45.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:45.104+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:45.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:45.104+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:45.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:45.104+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:45.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:45.105+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:45.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:45.290+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:45.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:45.290+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:45.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:45.290+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:45.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:45.291+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:46.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:45.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:46.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:45.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:46.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:45.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:46.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:45.814+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:46.433 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:33:46.335+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25636 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:33:46.433 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:46.312+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:46.433 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:46.312+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:46.433 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:46.312+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:46.433 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:46.313+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:46.433 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:46.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:46.433 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:46.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:46.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:46.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:46.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:46.149+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:47.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:46.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:47.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:46.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:47.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:46.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:47.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:46.800+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:47.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:47.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:47.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:47.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:47.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:47.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:47.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:47.314+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:47.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:47.122+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:47.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:47.122+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:47.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:47.122+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:47.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:47.124+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:48.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:47.838+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:48.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:47.838+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:48.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:47.838+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:48.106 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:47.838+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:48.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:48.288+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:48.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:48.288+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:48.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:48.288+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:48.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:48.289+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:48.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:48.131+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:48.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:48.131+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:48.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:48.131+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:48.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:48.131+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:49.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:48.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:49.089 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:48.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:49.089 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:48.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:49.089 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:48.850+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:49.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:49.114+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:49.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:49.114+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:49.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:49.114+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:49.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:49.115+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:49.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:49.259+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:49.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:49.259+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:49.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:49.259+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:49.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:49.260+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:50.172 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:50.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:50.172 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:50.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:50.172 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:50.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:50.172 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:50.083+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:50.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:49.888+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:50.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:49.888+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:50.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:49.888+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:50.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:49.889+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:50.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:50.266+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:50.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:50.266+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:50.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:50.266+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:50.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:50.266+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:51.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:51.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:51.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:51.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:51.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:51.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:51.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:51.103+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:51.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:50.890+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:51.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:50.890+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:51.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:50.890+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:51.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:50.891+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:51.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:33:51.338+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25724 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:33:51.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:51.265+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:51.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:51.266+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:51.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:51.266+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:51.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:51.266+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:52.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:51.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:52.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:51.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:52.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:51.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:52.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:51.875+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:52.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:52.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:52.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:52.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:52.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:52.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:52.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:52.056+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:52.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:52.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:52.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:52.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:52.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:52.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:52.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:52.294+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:53.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:52.853+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:53.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:52.853+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:53.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:52.853+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:53.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:52.853+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:53.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:53.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:53.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:53.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:53.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:53.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:53.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:53.038+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:53.596 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:53.304+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:53.596 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:53.304+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:53.596 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:53.304+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:53.596 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:53.305+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:54.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:54.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:54.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:54.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:54.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:54.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:54.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:54.088+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:54.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:53.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:54.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:53.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:54.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:53.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:54.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:53.890+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:54.531 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:54.275+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:54.531 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:54.275+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:54.532 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:54.275+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:54.532 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:54.276+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:55.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:55.079+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:55.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:55.079+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:55.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:55.079+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:55.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:55.079+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:55.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:54.932+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:55.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:54.932+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:55.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:54.932+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:55.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:54.933+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:55.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:55.242+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:55.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:55.242+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:55.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:55.242+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:55.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:55.244+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:56.029 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:55.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:56.029 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:55.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:56.029 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:55.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:56.029 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:55.947+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:56.029 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:56.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:56.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:56.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:56.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:56.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:56.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:56.053+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:56.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:56.195+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:56.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:56.196+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:56.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:56.196+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:56.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:56.196+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:56.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:33:56.341+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25815 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:33:57.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:57.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:57.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:57.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:57.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:57.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:57.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:57.005+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:57.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:56.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:57.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:56.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:57.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:56.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:57.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:56.995+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:57.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:57.164+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:57.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:57.164+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:57.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:57.164+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:57.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:57.165+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:58.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:58.043+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:58.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:58.043+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:58.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:58.043+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:58.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:58.045+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:58.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:58.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:58.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:58.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:58.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:58.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:58.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:58.129+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:58.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:58.039+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:58.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:58.039+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:58.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:58.039+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:58.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:58.040+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:59.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:59.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:59.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:59.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:59.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:59.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:33:59.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:33:59.135+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:59.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:59.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:59.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:59.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:59.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:59.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:33:59.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:33:59.029+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:33:59.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:59.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:59.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:59.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:59.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:59.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:33:59.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:33:59.048+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:00.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:00.166+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:00.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:00.166+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:00.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:00.166+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:00.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:00.167+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:00.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:00.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:00.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:00.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:00.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:00.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:00.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:00.050+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:00.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:00.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:00.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:00.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:00.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:00.025+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:00.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:00.025+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:01.318 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:01.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:01.318 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:01.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:01.318 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:01.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:01.318 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:01.047+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:01.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:01.168+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:01.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:01.168+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:01.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:01.168+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:01.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:01.169+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:01.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:01.050+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:01.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:01.050+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:01.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:01.050+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:01.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:01.051+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:01.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:34:01.343+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25903 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:34:02.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:02.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:02.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:02.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:02.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:02.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:02.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:02.001+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:02.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:02.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:02.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:02.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:02.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:02.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:02.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:02.061+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:02.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:02.214+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:02.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:02.214+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:02.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:02.214+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:02.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:02.215+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:03.386 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:03.049+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:03.387 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:03.049+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:03.387 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:03.049+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:03.387 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:03.050+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:03.387 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:03.240+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:03.387 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:03.240+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:03.387 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:03.240+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:03.387 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:03.240+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:03.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:03.060+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:03.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:03.060+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:03.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:03.060+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:03.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:03.061+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:04.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:04.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:04.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:04.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:04.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:04.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:04.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:04.036+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:04.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:04.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:04.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:04.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:04.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:04.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:04.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:04.073+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:04.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:04.279+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:04.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:04.279+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:04.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:04.279+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:04.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:04.279+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:05.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:05.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:05.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:05.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:05.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:05.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:05.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:05.052+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:05.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:05.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:05.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:05.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:05.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:05.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:05.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:05.077+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:05.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:05.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:05.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:05.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:05.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:05.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:05.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:05.294+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:06.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:06.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:06.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:06.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:06.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:06.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:06.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:06.024+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:06.268 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:06.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:06.269 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:06.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:06.269 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:06.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:06.269 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:06.121+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:06.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:34:06.346+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 25993 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:34:06.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:06.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:06.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:06.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:06.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:06.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:06.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:06.294+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:07.083 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:07.003+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:07.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:07.003+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:07.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:07.003+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:07.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:07.004+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:07.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:07.259+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:07.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:07.259+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:07.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:07.259+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:07.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:07.260+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:07.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:07.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:07.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:07.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:07.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:07.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:07.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:07.110+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:08.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:08.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:08.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:08.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:08.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:08.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:08.428 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:08.066+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:08.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:07.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:08.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:07.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:08.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:07.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:08.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:07.975+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:08.428 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:08.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:08.428 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:08.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:08.428 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:08.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:08.428 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:08.212+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:09.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:09.079+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:09.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:09.079+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:09.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:09.079+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:09.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:09.080+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:09.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:08.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:09.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:08.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:09.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:08.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:09.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:08.989+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:09.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:09.187+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:09.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:09.187+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:09.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:09.187+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:09.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:09.188+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:10.172 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:10.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:10.172 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:10.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:10.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:10.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:10.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:10.123+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:10.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:09.997+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:10.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:09.997+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:10.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:09.997+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:10.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:09.997+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:10.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:10.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:10.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:10.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:10.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:10.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:10.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:10.223+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:11.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:11.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:11.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:11.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:11.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:11.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:11.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:11.106+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:11.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:10.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:11.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:10.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:11.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:10.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:11.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:10.966+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:11.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:34:11.349+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26081 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:34:11.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:11.229+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:11.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:11.229+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:11.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:11.229+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:11.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:11.230+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:12.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:12.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:12.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:12.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:12.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:12.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:12.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:12.109+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:12.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:11.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:12.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:11.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:12.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:11.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:12.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:11.937+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:12.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:12.249+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:12.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:12.249+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:12.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:12.249+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:12.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:12.249+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:13.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:13.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:13.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:13.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:13.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:13.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:13.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:13.092+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:13.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:12.933+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:13.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:12.933+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:13.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:12.933+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:13.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:12.934+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:13.597 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:13.282+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:13.597 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:13.282+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:13.598 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:13.282+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:13.598 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:13.283+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:14.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:13.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:14.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:13.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:14.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:13.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:14.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:13.913+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:14.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:14.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:14.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:14.063+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:14.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:14.063+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:14.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:14.063+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:14.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:14.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:14.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:14.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:14.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:14.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:14.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:14.257+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:15.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:14.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:15.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:14.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:15.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:14.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:15.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:14.874+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:15.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:15.025+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:15.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:15.025+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:15.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:15.025+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:15.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:15.026+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:15.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:15.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:15.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:15.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:15.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:15.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:15.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:15.298+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:16.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:15.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:16.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:15.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:16.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:15.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:16.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:15.835+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:16.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:16.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:16.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:16.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:16.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:16.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:16.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:16.073+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:16.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:34:16.352+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26172 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:34:16.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:16.337+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:16.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:16.337+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:16.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:16.337+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:16.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:16.337+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:17.124 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:16.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:17.124 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:16.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:17.124 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:16.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:17.124 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:16.799+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:17.125 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:17.031+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:17.125 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:17.031+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:17.125 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:17.031+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:17.125 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:17.031+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:17.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:17.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:17.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:17.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:17.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:17.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:17.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:17.367+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:18.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:18.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:18.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:18.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:18.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:18.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:18.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:18.014+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:18.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:17.809+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:18.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:17.809+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:18.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:17.809+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:18.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:17.809+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:18.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:18.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:18.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:18.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:18.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:18.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:18.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:18.396+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:19.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:18.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:19.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:18.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:19.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:18.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:19.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:18.787+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:19.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:18.980+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:19.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:18.980+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:19.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:18.980+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:19.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:18.981+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:19.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:19.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:19.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:19.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:19.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:19.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:19.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:19.347+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:20.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:20.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:20.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:20.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:20.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:20.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:20.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:20.018+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:20.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:19.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:20.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:19.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:20.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:19.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:20.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:19.833+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:20.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:20.318+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:20.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:20.318+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:20.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:20.318+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:20.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:20.319+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:21.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:20.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:21.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:20.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:21.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:20.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:21.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:20.972+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:21.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:20.808+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:21.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:20.808+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:21.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:20.808+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:21.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:20.809+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:21.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:34:21.355+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26260 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:34:21.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:21.345+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:21.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:21.345+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:21.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:21.345+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:21.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:21.346+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:22.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:22.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:22.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:22.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:22.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:22.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:22.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:22.009+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:22.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:21.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:22.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:21.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:22.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:21.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:22.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:21.829+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:22.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:22.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:22.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:22.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:22.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:22.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:22.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:22.366+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:23.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:22.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:23.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:22.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:23.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:22.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:23.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:22.964+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:23.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:22.789+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:23.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:22.789+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:23.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:22.789+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:23.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:22.790+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:23.598 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:23.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:23.598 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:23.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:23.598 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:23.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:23.598 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:23.363+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:23.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:23.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:23.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:23.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:23.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:23.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:23.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:23.754+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:24.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:23.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:24.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:23.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:24.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:23.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:24.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:23.924+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:24.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:24.402+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:24.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:24.402+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:24.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:24.402+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:24.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:24.402+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:25.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:24.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:25.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:24.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:25.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:24.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:25.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:24.939+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:25.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:24.736+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:25.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:24.736+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:25.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:24.736+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:25.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:24.737+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:25.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:25.415+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:25.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:25.415+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:25.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:25.415+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:25.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:25.416+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:25.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:25.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:25.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:25.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:25.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:25.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:25.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:25.926+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:25.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:25.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:25.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:25.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:25.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:25.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:25.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:25.710+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:26.683 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:34:26.357+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26350 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:34:26.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:26.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:26.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:26.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:26.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:26.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:26.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:26.389+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:26.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:26.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:26.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:26.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:26.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:26.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:26.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:26.934+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:26.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:26.707+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:26.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:26.707+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:26.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:26.707+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:26.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:26.708+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:27.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:27.391+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:27.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:27.391+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:27.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:27.391+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:27.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:27.392+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:28.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:27.908+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:28.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:27.908+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:28.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:27.908+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:28.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:27.909+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:28.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:27.721+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:28.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:27.721+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:28.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:27.721+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:28.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:27.722+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:28.592 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:28.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:28.592 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:28.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:28.592 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:28.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:28.592 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:28.438+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:28.866 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:28.679+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:28.867 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:28.679+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:28.867 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:28.679+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:28.867 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:28.679+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:29.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:28.891+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:29.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:28.891+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:29.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:28.891+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:29.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:28.892+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:29.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:29.408+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:29.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:29.408+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:29.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:29.408+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:29.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:29.408+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:29.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:29.635+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:29.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:29.635+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:29.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:29.635+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:29.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:29.635+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:30.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:29.940+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:30.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:29.940+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:30.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:29.940+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:30.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:29.941+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:30.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:30.416+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:30.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:30.416+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:30.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:30.416+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:30.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:30.416+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:30.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:30.668+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:30.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:30.668+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:30.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:30.668+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:30.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:30.669+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:31.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:30.957+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:31.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:30.957+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:31.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:30.957+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:31.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:30.958+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:31.677 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:34:31.360+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26438 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:34:31.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:31.372+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:31.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:31.372+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:31.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:31.372+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:31.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:31.373+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:31.942 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:31.701+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:31.942 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:31.701+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:31.942 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:31.701+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:31.942 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:31.702+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:32.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:31.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:32.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:31.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:32.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:31.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:32.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:31.968+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:32.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:32.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:32.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:32.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:32.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:32.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:32.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:32.401+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:32.996 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:32.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:32.996 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:32.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:32.996 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:32.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:32.996 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:32.730+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:32.996 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:32.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:32.997 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:32.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:32.997 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:32.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:32.997 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:32.922+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:33.636 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:33.382+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:33.636 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:33.382+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:33.636 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:33.382+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:33.636 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:33.383+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:33.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:33.773+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:33.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:33.775+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:33.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:33.775+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:33.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:33.775+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:33.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:33.895+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:33.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:33.895+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:33.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:33.895+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:33.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:33.896+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:34.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:34.399+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:34.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:34.399+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:34.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:34.399+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:34.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:34.399+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:35.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:34.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:35.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:34.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:35.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:34.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:35.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:34.777+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:35.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:34.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:35.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:34.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:35.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:34.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:35.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:34.922+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:35.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:35.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:35.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:35.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:35.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:35.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:35.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:35.408+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:36.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:35.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:36.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:35.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:36.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:35.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:36.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:35.751+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:36.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:35.923+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:36.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:35.923+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:36.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:35.923+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:36.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:35.924+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:36.682 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:34:36.364+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26529 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:34:36.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:36.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:36.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:36.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:36.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:36.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:36.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:36.438+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:36.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:36.707+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:36.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:36.707+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:36.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:36.707+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:36.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:36.708+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:36.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:36.956+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:36.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:36.956+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:36.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:36.956+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:36.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:36.957+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:37.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:37.424+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:37.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:37.424+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:37.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:37.424+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:37.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:37.424+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:38.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:37.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:38.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:37.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:38.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:37.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:38.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:37.731+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:38.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:37.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:38.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:37.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:38.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:37.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:38.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:37.969+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:38.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:38.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:38.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:38.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:38.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:38.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:38.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:38.395+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:38.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:38.749+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:38.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:38.749+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:38.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:38.749+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:38.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:38.749+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:39.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:38.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:39.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:38.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:39.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:38.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:39.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:38.965+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:39.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:39.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:39.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:39.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:39.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:39.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:39.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:39.442+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:40.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:39.955+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:40.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:39.955+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:40.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:39.955+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:40.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:39.956+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:40.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:39.782+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:40.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:39.782+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:40.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:39.782+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:40.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:39.783+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:40.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:40.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:40.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:40.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:40.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:40.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:40.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:40.434+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:41.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:40.982+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:41.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:40.982+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:41.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:40.982+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:41.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:40.983+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:41.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:40.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:41.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:40.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:41.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:40.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:41.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:40.821+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:41.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:34:41.367+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26617 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:34:41.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:41.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:41.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:41.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:41.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:41.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:41.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:41.465+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:42.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:41.946+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:42.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:41.946+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:42.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:41.946+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:42.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:41.946+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:42.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:41.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:42.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:41.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:42.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:41.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:42.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:41.800+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:42.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:42.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:42.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:42.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:42.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:42.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:42.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:42.462+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:43.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:42.950+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:43.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:42.950+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:43.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:42.950+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:43.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:42.951+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:43.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:42.789+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:43.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:42.789+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:43.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:42.789+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:43.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:42.789+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:43.909 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:43.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:43.909 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:43.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:43.909 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:43.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:43.909 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:43.745+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:43.909 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:43.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:43.909 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:43.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:43.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:43.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:43.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:43.502+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:44.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:43.934+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:44.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:43.935+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:44.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:43.935+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:44.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:43.935+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:44.756 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:44.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:44.756 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:44.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:44.757 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:44.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:44.757 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:44.503+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:45.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:44.782+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:45.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:44.782+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:45.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:44.782+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:45.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:44.783+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:45.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:44.892+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:45.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:44.892+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:45.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:44.892+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:45.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:44.893+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:45.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:45.517+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:45.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:45.517+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:45.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:45.517+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:45.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:45.518+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:45.891 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:45.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:45.891 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:45.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:45.891 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:45.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:45.891 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:45.756+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:46.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:45.916+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:46.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:45.916+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:46.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:45.916+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:46.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:45.916+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:46.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:34:46.370+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26707 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:34:46.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:46.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:46.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:46.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:46.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:46.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:46.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:46.503+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:47.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:46.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:47.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:46.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:47.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:46.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:47.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:46.923+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:47.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:46.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:47.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:46.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:47.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:46.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:47.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:46.775+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:47.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:47.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:47.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:47.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:47.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:47.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:47.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:47.467+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:48.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:47.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:48.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:47.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:48.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:47.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:48.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:47.973+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:48.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:47.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:48.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:47.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:48.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:47.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:48.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:47.762+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:48.757 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:48.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:48.758 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:48.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:48.758 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:48.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:48.758 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:48.514+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:48.758 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:48.713+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:48.758 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:48.713+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:48.758 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:48.713+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:48.758 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:48.714+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:49.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:48.955+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:49.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:48.955+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:49.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:48.955+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:49.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:48.956+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:49.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:49.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:49.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:49.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:49.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:49.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:49.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:49.551+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:49.895 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:49.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:49.895 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:49.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:49.895 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:49.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:49.895 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:49.742+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:50.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:49.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:50.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:49.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:50.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:49.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:50.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:49.921+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:50.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:50.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:50.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:50.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:50.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:50.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:50.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:50.599+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:50.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:50.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:50.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:50.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:50.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:50.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:50.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:50.773+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:51.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:50.958+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:51.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:50.958+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:51.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:50.958+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:51.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:50.959+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:51.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:34:51.373+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26795 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:34:51.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:51.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:51.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:51.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:51.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:51.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:51.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:51.595+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:52.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:51.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:52.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:51.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:52.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:51.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:52.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:51.966+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:52.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:51.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:52.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:51.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:52.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:51.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:52.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:51.735+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:52.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:52.702+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:52.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:52.702+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:52.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:52.702+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:52.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:52.702+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:52.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:52.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:52.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:52.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:52.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:52.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:52.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:52.625+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:53.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:52.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:53.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:52.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:53.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:52.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:53.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:52.923+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:53.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:53.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:53.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:53.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:53.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:53.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:53.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:53.580+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:53.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:53.655+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:53.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:53.655+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:53.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:53.655+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:53.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:53.656+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:54.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:53.944+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:54.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:53.944+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:54.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:53.944+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:54.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:53.945+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:54.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:54.609+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:54.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:54.609+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:54.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:54.609+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:54.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:54.610+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:54.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:54.658+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:54.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:54.658+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:54.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:54.658+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:54.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:54.659+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:55.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:54.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:55.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:54.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:55.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:54.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:55.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:54.977+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:55.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:55.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:55.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:55.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:55.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:55.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:55.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:55.621+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:55.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:55.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:55.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:55.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:55.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:55.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:55.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:55.664+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:56.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:55.956+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:56.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:55.956+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:56.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:55.956+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:56.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:55.957+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:56.474 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:34:56.376+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26886 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:34:56.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:56.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:56.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:56.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:56.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:56.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:56.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:56.587+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:56.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:56.663+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:56.922 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:56.663+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:56.922 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:56.663+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:56.922 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:56.664+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:57.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:56.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:57.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:56.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:57.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:56.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:57.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:56.948+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:57.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:57.616+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:57.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:57.616+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:57.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:57.616+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:57.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:57.617+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:57.879 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:57.638+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:57.879 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:57.638+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:57.879 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:57.638+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:57.879 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:57.638+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:58.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:57.904+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:58.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:57.904+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:58.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:57.904+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:58.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:57.905+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:58.907 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:58.683+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:58.907 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:58.683+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:58.907 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:58.683+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:58.907 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:58.684+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:58.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:58.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:58.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:58.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:58.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:58.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:58.908 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:58.656+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:59.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:58.932+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:59.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:58.932+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:59.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:58.932+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:59.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:58.934+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:59.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:59.690+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:59.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:59.690+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:59.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:59.690+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:34:59.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:34:59.691+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:59.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:59.898+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:59.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:59.898+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:59.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:59.898+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:34:59.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:34:59.898+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:34:59.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:59.725+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:59.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:59.725+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:59.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:59.725+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:34:59.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:34:59.725+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:00.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:00.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:00.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:00.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:00.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:00.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:00.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:00.645+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:00.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:00.877+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:00.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:00.877+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:00.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:00.877+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:00.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:00.878+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:00.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:00.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:00.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:00.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:00.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:00.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:00.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:00.742+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:01.607 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:35:01.379+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 26974 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:35:01.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:01.836+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:01.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:01.836+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:01.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:01.836+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:01.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:01.838+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:01.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:01.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:01.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:01.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:01.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:01.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:01.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:01.634+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:01.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:01.716+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:01.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:01.716+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:01.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:01.716+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:01.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:01.717+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:02.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:02.842+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:02.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:02.842+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:02.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:02.842+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:02.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:02.842+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:02.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:02.639+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:02.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:02.639+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:02.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:02.639+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:02.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:02.640+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:02.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:02.747+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:02.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:02.747+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:02.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:02.747+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:02.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:02.748+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:03.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:03.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:03.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:03.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:03.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:03.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:03.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:03.595+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:03.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:03.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:03.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:03.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:03.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:03.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:03.884 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:03.838+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:03.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:03.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:03.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:03.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:03.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:03.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:03.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:03.761+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:04.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:04.601+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:04.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:04.601+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:04.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:04.601+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:04.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:04.602+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:04.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:04.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:04.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:04.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:04.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:04.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:04.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:04.742+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:04.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:04.834+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:04.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:04.834+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:04.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:04.834+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:04.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:04.835+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:05.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:05.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:05.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:05.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:05.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:05.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:05.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:05.632+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:05.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:05.844+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:05.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:05.844+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:05.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:05.844+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:05.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:05.845+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:05.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:05.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:05.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:05.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:05.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:05.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:05.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:05.774+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:06.599 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:35:06.382+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27064 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:35:06.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:06.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:06.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:06.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:06.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:06.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:06.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:06.625+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:06.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:06.743+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:06.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:06.743+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:06.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:06.743+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:06.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:06.743+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:06.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:06.852+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:06.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:06.852+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:06.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:06.852+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:06.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:06.852+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:07.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:07.743+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:07.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:07.743+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:07.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:07.743+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:07.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:07.744+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:07.858 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:07.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:07.858 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:07.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:07.858 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:07.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:07.858 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:07.583+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:08.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:07.883+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:08.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:07.883+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:08.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:07.883+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:08.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:07.884+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:08.862 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:08.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:08.862 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:08.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:08.862 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:08.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:08.862 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:08.775+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:08.862 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:08.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:08.862 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:08.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:08.863 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:08.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:08.863 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:08.632+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:09.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:08.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:09.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:08.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:09.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:08.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:09.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:08.888+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:09.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:09.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:09.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:09.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:09.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:09.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:09.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:09.860+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:09.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:09.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:09.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:09.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:09.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:09.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:09.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:09.632+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:09.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:09.786+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:09.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:09.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:09.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:09.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:09.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:09.787+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:10.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:10.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:10.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:10.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:10.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:10.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:10.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:10.594+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:10.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:10.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:10.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:10.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:10.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:10.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:10.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:10.838+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:10.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:10.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:10.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:10.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:10.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:10.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:10.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:10.798+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:11.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:35:11.385+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27152 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:35:11.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:11.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:11.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:11.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:11.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:11.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:11.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:11.594+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:12.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:11.843+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:12.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:11.843+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:12.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:11.843+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:12.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:11.844+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:12.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:11.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:12.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:11.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:12.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:11.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:12.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:11.819+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:12.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:12.856+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:12.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:12.856+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:12.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:12.856+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:12.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:12.856+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:12.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:12.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:12.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:12.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:12.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:12.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:12.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:12.643+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:12.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:12.781+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:12.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:12.781+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:12.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:12.781+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:12.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:12.782+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:13.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:13.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:13.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:13.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:13.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:13.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:13.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:13.663+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:13.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:13.854+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:13.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:13.854+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:13.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:13.854+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:13.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:13.854+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:13.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:13.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:13.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:13.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:13.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:13.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:13.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:13.822+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:14.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:14.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:14.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:14.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:14.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:14.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:14.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:14.844+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:14.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:14.882+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:14.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:14.882+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:14.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:14.882+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:14.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:14.883+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:14.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:14.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:14.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:14.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:14.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:14.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:14.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:14.706+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:15.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:15.852+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:15.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:15.852+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:15.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:15.852+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:15.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:15.853+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:15.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:15.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:15.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:15.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:15.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:15.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:15.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:15.868+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:15.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:15.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:15.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:15.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:15.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:15.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:15.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:15.706+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:16.665 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:35:16.389+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27243 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:35:16.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:16.842+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:16.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:16.842+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:16.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:16.842+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:16.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:16.843+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:16.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:16.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:16.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:16.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:16.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:16.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:16.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:16.692+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:16.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:16.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:16.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:16.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:16.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:16.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:16.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:16.850+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:17.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:17.891+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:17.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:17.891+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:17.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:17.891+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:17.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:17.891+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:17.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:17.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:17.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:17.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:17.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:17.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:17.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:17.665+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:17.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:17.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:17.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:17.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:17.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:17.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:17.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:17.875+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:18.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:18.882+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:18.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:18.882+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:18.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:18.882+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:18.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:18.882+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:18.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:18.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:18.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:18.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:18.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:18.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:18.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:18.885+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:18.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:18.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:18.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:18.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:18.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:18.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:18.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:18.694+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:19.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:19.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:19.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:19.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:19.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:19.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:19.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:19.703+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:20.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:19.922+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:20.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:19.922+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:20.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:19.922+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:20.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:19.923+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:20.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:19.910+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:20.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:19.910+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:20.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:19.910+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:20.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:19.911+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:20.865 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:20.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:20.865 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:20.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:20.865 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:20.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:20.865 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:20.660+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:21.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:20.892+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:21.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:20.892+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:21.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:20.892+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:21.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:20.892+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:21.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:20.891+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:21.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:20.891+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:21.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:20.891+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:21.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:20.891+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:21.634 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:35:21.392+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27331 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:35:21.886 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:21.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:21.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:21.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:21.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:21.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:21.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:21.660+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:22.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:21.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:22.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:21.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:22.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:21.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:22.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:21.921+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:22.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:21.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:22.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:21.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:22.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:21.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:22.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:21.913+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:22.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:22.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:22.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:22.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:22.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:22.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:22.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:22.626+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:23.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:22.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:23.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:22.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:23.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:22.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:23.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:22.921+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:23.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:22.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:23.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:22.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:23.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:22.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:23.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:22.951+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:23.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:23.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:23.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:23.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:23.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:23.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:23.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:23.629+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:24.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:23.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:24.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:23.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:24.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:23.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:24.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:23.972+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:24.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:23.952+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:24.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:23.952+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:24.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:23.952+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:24.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:23.953+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:24.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:24.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:24.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:24.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:24.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:24.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:24.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:24.643+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:25.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:24.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:25.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:24.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:25.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:24.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:25.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:24.968+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:25.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:24.913+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:25.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:24.913+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:25.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:24.913+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:25.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:24.914+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:25.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:25.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:25.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:25.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:25.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:25.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:25.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:25.665+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:26.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:25.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:26.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:25.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:26.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:25.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:26.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:25.954+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:26.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:25.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:26.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:25.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:26.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:25.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:26.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:25.929+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:26.492 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:35:26.394+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27421 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:35:26.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:26.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:26.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:26.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:26.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:26.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:26.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:26.707+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:27.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:26.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:27.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:26.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:27.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:26.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:27.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:26.960+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:27.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:26.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:27.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:26.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:27.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:26.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:27.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:26.914+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:28.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:27.975+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:28.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:27.975+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:28.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:27.975+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:28.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:27.975+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:28.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:27.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:28.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:27.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:28.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:27.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:28.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:27.753+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:28.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:27.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:28.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:27.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:28.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:27.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:28.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:27.867+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:28.998 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:28.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:28.999 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:28.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:28.999 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:28.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:28.999 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:28.910+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:28.999 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:28.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:28.999 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:28.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:28.999 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:28.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:28.999 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:28.748+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:29.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:29.023+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:29.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:29.023+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:29.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:29.023+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:29.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:29.024+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:29.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:29.703+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:29.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:29.703+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:29.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:29.703+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:29.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:29.704+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:29.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:29.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:29.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:29.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:29.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:29.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:29.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:29.934+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:30.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:30.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:30.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:30.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:30.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:30.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:30.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:30.040+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:30.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:30.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:30.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:30.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:30.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:30.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:30.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:30.886+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:30.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:30.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:30.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:30.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:30.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:30.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:30.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:30.698+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:31.371 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:31.073+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:31.371 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:31.073+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:31.371 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:31.073+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:31.371 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:31.073+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:31.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:35:31.397+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27509 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:35:32.011 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:31.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:32.011 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:31.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:32.012 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:31.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:32.012 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:31.929+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:32.012 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:31.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:32.012 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:31.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:32.012 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:31.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:32.012 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:31.729+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:32.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:32.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:32.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:32.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:32.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:32.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:32.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:32.037+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:32.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:32.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:32.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:32.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:32.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:32.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:32.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:32.692+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:33.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:32.964+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:33.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:32.964+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:33.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:32.964+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:33.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:32.964+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:33.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:33.084+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:33.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:33.084+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:33.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:33.084+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:33.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:33.084+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:33.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:33.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:33.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:33.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:33.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:33.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:33.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:33.656+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:34.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:34.110+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:34.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:34.110+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:34.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:34.110+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:34.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:34.112+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:34.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:33.982+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:34.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:33.982+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:34.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:33.982+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:34.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:33.982+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:34.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:34.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:34.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:34.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:34.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:34.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:34.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:34.700+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:35.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:35.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:35.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:35.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:35.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:35.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:35.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:35.128+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:35.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:34.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:35.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:34.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:35.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:34.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:35.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:34.995+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:35.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:35.670+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:35.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:35.670+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:35.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:35.670+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:35.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:35.671+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:36.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:36.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:36.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:36.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:36.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:36.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:36.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:36.154+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:36.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:35.997+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:36.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:35.997+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:36.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:35.997+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:36.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:35.997+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:36.684 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:35:36.402+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27600 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:35:36.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:36.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:36.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:36.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:36.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:36.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:36.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:36.711+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:37.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:36.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:37.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:36.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:37.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:36.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:37.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:36.993+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:37.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:37.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:37.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:37.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:37.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:37.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:37.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:37.194+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:38.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:37.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:38.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:37.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:38.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:37.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:38.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:37.722+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:38.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:38.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:38.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:38.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:38.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:38.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:38.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:38.205+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:38.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:38.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:38.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:38.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:38.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:38.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:38.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:38.033+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:38.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:38.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:38.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:38.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:38.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:38.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:38.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:38.754+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:39.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:39.199+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:39.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:39.199+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:39.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:39.199+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:39.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:39.200+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:39.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:39.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:39.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:39.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:39.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:39.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:39.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:39.038+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:40.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:39.765+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:40.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:39.765+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:40.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:39.765+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:40.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:39.766+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:40.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:40.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:40.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:40.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:40.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:40.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:40.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:40.054+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:40.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:40.205+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:40.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:40.205+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:40.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:40.205+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:40.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:40.206+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:41.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:40.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:41.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:40.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:41.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:40.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:41.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:40.755+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:41.379 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:41.162+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:41.379 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:41.162+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:41.379 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:41.162+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:41.379 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:41.163+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:41.380 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:41.097+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:41.380 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:41.097+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:41.380 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:41.097+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:41.380 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:41.097+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:41.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:35:41.405+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27688 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:35:42.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:41.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:42.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:41.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:42.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:41.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:42.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:41.741+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:42.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:42.171+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:42.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:42.171+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:42.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:42.171+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:42.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:42.171+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:42.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:42.114+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:42.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:42.114+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:42.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:42.114+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:42.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:42.115+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:43.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:42.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:43.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:42.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:43.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:42.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:43.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:42.782+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:43.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:43.149+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:43.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:43.149+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:43.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:43.149+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:43.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:43.150+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:43.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:43.079+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:43.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:43.079+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:43.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:43.079+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:43.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:43.079+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:43.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:43.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:43.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:43.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:43.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:43.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:43.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:43.743+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:44.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:44.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:44.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:44.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:44.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:44.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:44.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:44.195+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:44.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:44.034+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:44.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:44.034+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:44.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:44.034+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:44.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:44.035+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:44.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:44.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:44.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:44.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:44.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:44.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:44.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:44.718+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:45.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:45.226+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:45.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:45.226+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:45.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:45.226+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:45.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:45.226+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:45.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:45.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:45.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:45.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:45.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:45.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:45.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:45.031+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:45.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:45.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:45.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:45.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:45.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:45.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:45.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:45.692+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:46.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:46.191+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:46.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:46.191+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:46.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:46.191+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:46.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:46.192+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:46.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:46.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:46.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:46.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:46.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:46.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:46.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:46.055+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:46.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:35:46.408+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27778 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:35:47.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:46.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:47.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:46.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:47.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:46.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:47.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:46.740+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:47.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:47.057+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:47.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:47.057+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:47.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:47.057+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:47.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:47.057+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:47.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:47.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:47.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:47.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:47.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:47.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:47.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:47.165+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:47.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:47.704+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:47.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:47.704+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:47.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:47.704+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:47.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:47.705+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:48.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:48.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:48.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:48.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:48.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:48.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:48.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:48.129+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:48.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:48.026+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:48.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:48.026+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:48.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:48.026+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:48.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:48.026+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:49.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:48.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:49.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:48.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:49.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:48.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:49.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:48.730+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:49.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:49.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:49.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:49.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:49.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:49.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:49.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:49.165+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:49.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:49.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:49.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:49.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:49.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:49.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:49.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:49.058+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:50.092 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:49.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:50.092 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:49.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:50.092 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:49.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:50.092 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:49.777+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:50.092 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:50.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:50.093 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:50.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:50.093 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:50.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:50.093 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:50.009+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:50.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:50.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:50.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:50.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:50.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:50.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:50.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:50.120+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:51.118 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:50.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:51.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:50.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:51.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:50.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:51.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:50.805+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:51.119 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:51.007+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:51.119 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:51.007+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:51.119 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:51.007+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:51.119 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:51.007+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:51.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:51.144+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:51.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:51.144+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:51.386 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:51.144+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:51.386 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:51.145+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:51.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:35:51.411+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27865 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:35:52.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:52.019+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:52.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:52.019+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:52.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:52.019+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:52.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:52.019+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:52.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:51.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:52.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:51.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:52.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:51.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:52.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:51.774+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:52.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:52.160+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:52.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:52.160+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:52.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:52.160+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:52.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:52.161+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:53.169 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:52.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:53.169 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:52.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:53.169 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:52.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:53.169 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:52.771+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:53.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:52.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:53.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:52.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:53.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:52.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:53.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:52.979+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:53.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:53.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:53.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:53.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:53.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:53.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:53.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:53.195+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:53.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:53.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:53.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:53.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:53.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:53.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:53.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:53.740+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:54.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:54.235+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:54.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:54.235+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:54.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:54.235+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:54.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:54.235+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:54.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:54.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:54.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:54.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:54.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:54.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:54.300 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:54.029+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:55.023 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:54.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:55.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:54.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:55.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:54.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:55.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:54.786+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:55.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:55.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:55.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:55.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:55.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:55.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:55.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:55.050+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:55.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:55.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:55.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:55.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:55.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:55.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:55.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:55.213+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:56.000 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:55.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:56.001 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:55.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:56.001 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:55.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:56.001 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:55.762+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:56.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:56.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:56.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:56.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:56.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:56.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:56.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:56.027+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:56.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:56.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:56.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:56.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:56.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:56.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:56.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:56.205+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:56.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:35:56.415+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 27956 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:35:57.006 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:56.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:57.006 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:56.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:57.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:56.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:57.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:56.753+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:57.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:57.245+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:57.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:57.245+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:57.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:57.245+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:57.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:57.246+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:57.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:57.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:57.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:57.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:57.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:57.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:57.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:57.033+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:58.019 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:57.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:58.019 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:57.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:58.019 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:57.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:58.019 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:57.770+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:58.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:58.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:58.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:58.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:58.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:58.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:58.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:58.211+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:58.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:58.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:58.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:58.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:58.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:58.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:58.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:58.045+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:59.037 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:58.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:59.037 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:58.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:59.037 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:58.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:35:59.037 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:58.723+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:59.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:59.062+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:59.338 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:59.062+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:59.338 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:59.062+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:35:59.338 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:35:59.063+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:35:59.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:59.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:59.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:59.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:59.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:59.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:35:59.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:35:59.212+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:00.080 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:59.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:00.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:59.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:00.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:59.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:00.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:35:59.730+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:00.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:00.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:00.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:00.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:00.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:00.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:00.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:00.172+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:00.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:00.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:00.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:00.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:00.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:00.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:00.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:00.107+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:01.098 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:00.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:01.098 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:00.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:01.098 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:00.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:01.098 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:00.733+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:01.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:01.124+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:01.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:01.124+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:01.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:01.124+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:01.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:01.125+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:01.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:01.129+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:01.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:01.129+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:01.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:01.129+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:01.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:01.130+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:01.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:36:01.418+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28044 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:36:02.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:01.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:02.055 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:01.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:02.055 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:01.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:02.055 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:01.734+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:02.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:02.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:02.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:02.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:02.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:02.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:02.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:02.081+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:02.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:02.133+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:02.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:02.133+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:02.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:02.133+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:02.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:02.135+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:02.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:02.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:02.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:02.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:02.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:02.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:02.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:02.711+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:03.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:03.075+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:03.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:03.075+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:03.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:03.075+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:03.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:03.076+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:03.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:03.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:03.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:03.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:03.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:03.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:03.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:03.155+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:03.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:03.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:03.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:03.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:03.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:03.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:03.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:03.684+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:04.376 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:04.203+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:04.376 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:04.203+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:04.376 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:04.203+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:04.376 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:04.204+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:04.376 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:04.050+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:04.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:04.050+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:04.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:04.050+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:04.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:04.050+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:04.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:04.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:04.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:04.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:04.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:04.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:04.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:04.697+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:05.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:05.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:05.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:05.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:05.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:05.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:05.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:05.024+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:05.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:05.199+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:05.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:05.199+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:05.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:05.199+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:05.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:05.199+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:05.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:05.684+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:05.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:05.684+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:05.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:05.684+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:05.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:05.685+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:06.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:06.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:06.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:06.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:06.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:06.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:06.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:06.234+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:06.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:06.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:06.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:06.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:06.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:06.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:06.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:06.050+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:06.611 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:36:06.421+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28134 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:36:06.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:06.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:06.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:06.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:06.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:06.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:06.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:06.639+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:07.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:07.206+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:07.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:07.206+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:07.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:07.206+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:07.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:07.207+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:07.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:07.093+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:07.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:07.093+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:07.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:07.093+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:07.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:07.094+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:07.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:07.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:07.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:07.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:07.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:07.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:07.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:07.611+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:08.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:08.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:08.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:08.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:08.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:08.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:08.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:08.243+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:08.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:08.063+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:08.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:08.063+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:08.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:08.063+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:08.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:08.064+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:08.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:08.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:08.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:08.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:08.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:08.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:08.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:08.600+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:09.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:09.016+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:09.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:09.016+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:09.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:09.016+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:09.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:09.016+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:09.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:09.273+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:09.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:09.273+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:09.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:09.273+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:09.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:09.274+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:09.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:09.609+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:09.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:09.609+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:09.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:09.609+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:09.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:09.610+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:10.254 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:09.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:10.254 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:09.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:10.255 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:09.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:10.255 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:09.991+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:10.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:10.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:10.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:10.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:10.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:10.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:10.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:10.281+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:10.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:10.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:10.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:10.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:10.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:10.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:10.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:10.640+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:11.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:10.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:11.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:10.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:11.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:10.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:11.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:10.948+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:11.522 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:36:11.424+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28222 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:36:11.522 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:11.251+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:11.522 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:11.251+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:11.522 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:11.251+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:11.522 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:11.252+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:11.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:11.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:11.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:11.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:11.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:11.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:11.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:11.606+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:12.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:11.909+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:12.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:11.909+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:12.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:11.909+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:12.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:11.909+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:12.555 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:12.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:12.555 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:12.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:12.555 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:12.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:12.555 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:12.283+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:12.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:12.581+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:12.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:12.581+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:12.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:12.581+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:12.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:12.582+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:13.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:12.909+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:13.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:12.909+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:13.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:12.909+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:13.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:12.910+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:13.572 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:13.307+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:13.572 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:13.307+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:13.572 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:13.307+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:13.572 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:13.308+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:13.866 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:13.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:13.866 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:13.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:13.866 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:13.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:13.866 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:13.599+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:14.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:13.892+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:14.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:13.892+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:14.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:13.892+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:14.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:13.892+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:14.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:14.326+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:14.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:14.326+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:14.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:14.326+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:14.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:14.327+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:14.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:14.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:14.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:14.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:14.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:14.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:14.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:14.649+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:15.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:14.919+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:15.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:14.919+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:15.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:14.919+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:15.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:14.919+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:15.652 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:15.376+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:15.653 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:15.376+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:15.653 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:15.376+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:15.653 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:15.377+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:15.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:15.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:15.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:15.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:15.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:15.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:15.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:15.679+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:16.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:15.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:16.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:15.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:16.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:15.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:16.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:15.966+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:16.681 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:36:16.427+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28313 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:36:16.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:16.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:16.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:16.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:16.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:16.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:16.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:16.415+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:16.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:16.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:16.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:16.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:16.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:16.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:16.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:16.708+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:16.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:16.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:16.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:16.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:16.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:16.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:16.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:16.937+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:17.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:17.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:17.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:17.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:17.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:17.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:17.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:17.450+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:17.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:17.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:17.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:17.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:17.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:17.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:17.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:17.665+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:18.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:17.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:18.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:17.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:18.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:17.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:18.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:17.890+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:18.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:18.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:18.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:18.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:18.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:18.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:18.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:18.478+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:18.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:18.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:18.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:18.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:18.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:18.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:18.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:18.648+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:19.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:18.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:19.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:18.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:19.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:18.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:19.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:18.937+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:19.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:19.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:19.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:19.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:19.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:19.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:19.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:19.697+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:19.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:19.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:19.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:19.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:19.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:19.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:19.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:19.489+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:20.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:19.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:20.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:19.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:20.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:19.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:20.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:19.924+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:20.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:20.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:20.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:20.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:20.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:20.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:20.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:20.505+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:20.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:20.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:20.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:20.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:20.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:20.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:20.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:20.700+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:21.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:20.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:21.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:20.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:21.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:20.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:21.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:20.931+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:21.446 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:36:21.431+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28401 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:36:21.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:21.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:21.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:21.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:21.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:21.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:21.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:21.474+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:21.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:21.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:21.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:21.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:21.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:21.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:21.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:21.687+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:22.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:21.977+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:22.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:21.977+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:22.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:21.977+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:22.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:21.977+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:22.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:22.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:22.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:22.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:22.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:22.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:22.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:22.495+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:22.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:22.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:22.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:22.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:22.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:22.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:22.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:22.735+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:23.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:23.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:23.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:23.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:23.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:23.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:23.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:23.005+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:23.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:23.492+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:23.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:23.492+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:23.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:23.492+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:23.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:23.493+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:23.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:23.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:23.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:23.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:23.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:23.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:23.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:23.725+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:24.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:24.047+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:24.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:24.047+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:24.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:24.047+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:24.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:24.047+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:24.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:24.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:24.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:24.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:24.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:24.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:24.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:24.515+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:24.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:24.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:24.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:24.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:24.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:24.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:24.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:24.732+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:25.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:25.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:25.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:25.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:25.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:25.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:25.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:25.034+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:25.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:25.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:25.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:25.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:25.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:25.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:25.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:25.467+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:26.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:25.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:26.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:25.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:26.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:25.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:26.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:25.733+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:26.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:26.063+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:26.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:26.063+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:26.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:26.063+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:26.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:26.064+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:26.688 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:36:26.433+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28491 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:36:26.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:26.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:26.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:26.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:26.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:26.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:26.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:26.433+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:26.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:26.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:26.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:26.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:26.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:26.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:26.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:26.714+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:27.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:27.016+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:27.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:27.016+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:27.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:27.016+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:27.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:27.016+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:27.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:27.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:27.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:27.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:27.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:27.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:27.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:27.480+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:28.015 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:27.742+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:28.015 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:27.742+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:28.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:27.742+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:28.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:27.743+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:28.405 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:28.040+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:28.405 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:28.040+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:28.406 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:28.040+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:28.406 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:28.041+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:28.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:28.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:28.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:28.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:28.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:28.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:28.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:28.431+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:29.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:28.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:29.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:28.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:29.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:28.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:29.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:28.741+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:29.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:29.064+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:29.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:29.064+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:29.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:29.064+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:29.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:29.064+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:29.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:29.393+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:29.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:29.393+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:29.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:29.393+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:29.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:29.394+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:29.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:29.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:29.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:29.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:29.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:29.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:29.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:29.692+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:30.326 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:30.094+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:30.327 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:30.094+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:30.327 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:30.094+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:30.327 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:30.094+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:30.657 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:30.351+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:30.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:30.351+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:30.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:30.351+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:30.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:30.352+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:30.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:30.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:30.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:30.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:30.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:30.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:30.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:30.684+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:31.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:31.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:31.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:31.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:31.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:31.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:31.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:31.135+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:31.632 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:36:31.436+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28579 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:36:31.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:31.392+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:31.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:31.392+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:31.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:31.392+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:31.633 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:31.393+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:31.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:31.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:31.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:31.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:31.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:31.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:31.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:31.658+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:32.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:32.350+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:32.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:32.350+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:32.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:32.350+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:32.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:32.351+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:32.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:32.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:32.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:32.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:32.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:32.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:32.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:32.159+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:32.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:32.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:32.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:32.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:32.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:32.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:32.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:32.648+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:33.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:33.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:33.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:33.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:33.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:33.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:33.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:33.346+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:33.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:33.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:33.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:33.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:33.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:33.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:33.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:33.205+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:33.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:33.601+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:33.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:33.601+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:33.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:33.601+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:33.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:33.602+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:34.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:34.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:34.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:34.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:34.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:34.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:34.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:34.365+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:34.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:34.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:34.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:34.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:34.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:34.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:34.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:34.203+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:34.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:34.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:34.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:34.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:34.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:34.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:34.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:34.560+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:35.383 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:35.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:35.383 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:35.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:35.384 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:35.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:35.384 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:35.155+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:35.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:35.409+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:35.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:35.409+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:35.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:35.409+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:35.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:35.411+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:35.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:35.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:35.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:35.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:35.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:35.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:35.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:35.588+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:36.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:36.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:36.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:36.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:36.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:36.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:36.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:36.114+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:36.607 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:36:36.441+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28670 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:36:36.607 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:36.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:36.607 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:36.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:36.607 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:36.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:36.607 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:36.451+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:36.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:36.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:36.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:36.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:36.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:36.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:36.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:36.634+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:37.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:37.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:37.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:37.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:37.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:37.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:37.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:37.109+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:37.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:37.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:37.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:37.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:37.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:37.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:37.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:37.440+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:37.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:37.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:37.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:37.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:37.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:37.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:37.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:37.647+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:38.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:38.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:38.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:38.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:38.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:38.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:38.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:38.159+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:38.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:38.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:38.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:38.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:38.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:38.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:38.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:38.487+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:38.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:38.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:38.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:38.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:38.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:38.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:38.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:38.684+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:39.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:39.185+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:39.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:39.185+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:39.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:39.185+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:39.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:39.185+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:39.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:39.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:39.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:39.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:39.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:39.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:39.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:39.460+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:39.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:39.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:39.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:39.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:39.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:39.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:39.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:39.650+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:40.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:40.195+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:40.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:40.195+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:40.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:40.195+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:40.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:40.196+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:40.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:40.484+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:40.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:40.484+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:40.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:40.484+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:40.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:40.484+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:40.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:40.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:40.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:40.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:40.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:40.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:40.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:40.616+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:41.429 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:36:41.444+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28758 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:36:41.429 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:41.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:41.430 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:41.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:41.430 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:41.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:41.430 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:41.203+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:41.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:41.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:41.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:41.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:41.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:41.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:41.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:41.483+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:41.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:41.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:41.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:41.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:41.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:41.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:41.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:41.637+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:42.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:42.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:42.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:42.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:42.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:42.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:42.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:42.190+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:42.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:42.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:42.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:42.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:42.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:42.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:42.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:42.488+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:42.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:42.656+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:42.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:42.656+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:42.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:42.656+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:42.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:42.657+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:43.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:43.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:43.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:43.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:43.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:43.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:43.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:43.170+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:43.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:43.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:43.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:43.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:43.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:43.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:43.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:43.507+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:43.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:43.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:43.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:43.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:43.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:43.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:43.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:43.683+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:44.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:44.123+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:44.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:44.123+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:44.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:44.123+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:44.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:44.123+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:44.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:44.703+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:44.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:44.703+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:44.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:44.703+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:44.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:44.703+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:44.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:44.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:44.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:44.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:44.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:44.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:44.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:44.508+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:45.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:45.102+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:45.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:45.102+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:45.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:45.102+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:45.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:45.103+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:45.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:45.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:45.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:45.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:45.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:45.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:45.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:45.672+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:45.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:45.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:45.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:45.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:45.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:45.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:45.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:45.476+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:46.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:46.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:46.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:46.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:46.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:46.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:46.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:46.056+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:36:46.627+0000 7f770c428700 0 monclient(hunting): authenticate timed out after 300 2024-04-27T02:36:46.601 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:36:46.447+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28848 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:36:46.602 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:46.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:46.602 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:46.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:46.602 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:46.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:46.602 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:46.471+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:46.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:46.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:46.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:46.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:46.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:46.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:46.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:46.662+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:47.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:47.063+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:47.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:47.063+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:47.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:47.063+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:47.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:47.063+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:47.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:47.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:47.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:47.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:47.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:47.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:47.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:47.465+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:47.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:47.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:47.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:47.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:47.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:47.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:47.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:47.684+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:48.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:48.079+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:48.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:48.079+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:48.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:48.079+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:48.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:48.079+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:48.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:48.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:48.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:48.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:48.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:48.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:48.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:48.466+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:48.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:48.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:48.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:48.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:48.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:48.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:48.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:48.656+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:49.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:49.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:49.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:49.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:49.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:49.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:49.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:49.118+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:49.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:49.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:49.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:49.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:49.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:49.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:49.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:49.643+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:49.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:49.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:49.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:49.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:49.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:49.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:49.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:49.438+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:50.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:50.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:50.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:50.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:50.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:50.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:50.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:50.100+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:50.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:50.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:50.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:50.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:50.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:50.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:50.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:50.608+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:50.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:50.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:50.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:50.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:50.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:50.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:50.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:50.468+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:51.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:51.102+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:51.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:51.102+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:51.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:51.102+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:51.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:51.102+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:51.693 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:36:51.450+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 28936 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:36:51.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:51.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:51.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:51.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:51.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:51.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:51.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:51.424+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:51.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:51.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:51.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:51.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:51.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:51.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:51.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:51.622+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:52.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:52.126+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:52.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:52.126+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:52.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:52.126+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:52.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:52.126+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:52.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:52.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:52.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:52.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:52.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:52.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:52.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:52.441+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:52.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:52.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:52.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:52.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:52.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:52.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:52.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:52.605+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:53.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:53.088+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:53.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:53.088+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:53.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:53.088+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:53.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:53.088+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:53.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:53.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:53.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:53.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:53.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:53.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:53.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:53.564+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:53.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:53.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:53.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:53.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:53.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:53.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:53.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:53.460+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:54.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:54.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:54.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:54.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:54.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:54.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:54.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:54.113+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:54.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:54.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:54.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:54.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:54.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:54.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:54.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:54.550+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:54.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:54.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:54.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:54.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:54.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:54.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:54.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:54.465+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:55.430 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:55.082+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:55.430 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:55.082+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:55.430 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:55.082+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:55.430 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:55.082+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:55.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:55.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:55.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:55.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:55.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:55.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:55.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:55.457+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:55.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:55.584+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:55.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:55.584+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:55.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:55.584+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:55.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:55.585+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:56.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:56.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:56.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:56.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:56.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:56.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:56.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:56.114+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:56.693 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:36:56.454+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 29027 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:36:56.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:56.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:56.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:56.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:56.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:56.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:56.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:56.468+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:56.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:56.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:56.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:56.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:56.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:56.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:56.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:56.578+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:57.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:57.141+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:57.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:57.141+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:57.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:57.141+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:57.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:57.141+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:57.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:57.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:57.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:57.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:57.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:57.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:57.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:57.604+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:57.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:57.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:57.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:57.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:57.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:57.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:57.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:57.443+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:58.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:58.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:58.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:58.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:58.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:58.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:58.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:58.162+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:58.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:58.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:58.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:58.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:58.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:58.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:58.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:58.443+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:58.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:58.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:58.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:58.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:58.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:58.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:58.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:58.638+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:59.405 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:59.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:59.405 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:59.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:59.405 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:59.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:36:59.405 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:36:59.175+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:59.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:59.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:59.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:59.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:59.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:59.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:36:59.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:36:59.432+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:36:59.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:59.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:59.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:59.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:59.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:59.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:36:59.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:36:59.622+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:00.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:00.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:00.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:00.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:00.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:00.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:00.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:00.411+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:00.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:00.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:00.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:00.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:00.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:00.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:00.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:00.179+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:00.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:00.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:00.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:00.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:00.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:00.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:00.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:00.635+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:01.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:01.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:01.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:01.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:01.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:01.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:01.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:01.397+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:01.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:01.165+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:01.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:01.165+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:01.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:01.165+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:01.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:01.165+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:01.693 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:37:01.457+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 29115 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:37:01.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:01.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:01.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:01.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:01.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:01.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:01.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:01.684+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:02.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:02.134+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:02.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:02.134+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:02.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:02.134+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:02.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:02.135+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:02.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:02.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:02.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:02.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:02.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:02.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:02.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:02.444+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:02.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:02.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:02.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:02.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:02.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:02.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:02.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:02.673+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:03.421 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:03.089+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:03.421 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:03.089+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:03.421 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:03.089+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:03.421 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:03.090+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:03.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:03.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:03.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:03.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:03.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:03.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:03.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:03.447+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:03.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:03.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:03.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:03.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:03.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:03.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:03.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:03.706+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:04.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:04.137+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:04.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:04.137+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:04.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:04.137+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:04.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:04.138+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:04.852 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:04.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:04.852 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:04.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:04.852 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:04.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:04.852 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:04.716+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:04.853 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:04.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:04.853 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:04.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:04.853 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:04.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:04.853 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:04.497+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:05.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:05.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:05.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:05.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:05.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:05.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:05.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:05.155+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:05.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:05.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:05.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:05.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:05.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:05.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:05.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:05.764+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:05.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:05.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:05.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:05.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:05.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:05.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:05.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:05.545+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:06.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:06.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:06.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:06.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:06.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:06.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:06.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:06.161+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:06.693 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:37:06.460+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 29205 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:37:06.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:06.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:06.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:06.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:06.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:06.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:06.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:06.535+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:07.132 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:06.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:07.132 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:06.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:07.132 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:06.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:07.132 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:06.753+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:07.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:07.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:07.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:07.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:07.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:07.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:07.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:07.159+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:07.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:07.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:07.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:07.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:07.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:07.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:07.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:07.771+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:07.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:07.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:07.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:07.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:07.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:07.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:07.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:07.527+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:08.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:08.128+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:08.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:08.128+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:08.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:08.128+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:08.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:08.128+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:08.754 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:08.499+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:08.754 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:08.499+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:08.754 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:08.499+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:08.754 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:08.499+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:09.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:08.780+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:09.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:08.780+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:09.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:08.780+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:09.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:08.781+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:09.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:09.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:09.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:09.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:09.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:09.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:09.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:09.151+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:09.892 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:09.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:09.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:09.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:09.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:09.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:09.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:09.761+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:09.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:09.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:09.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:09.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:09.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:09.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:09.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:09.541+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:10.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:10.152+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:10.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:10.152+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:10.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:10.152+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:10.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:10.152+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:10.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:10.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:10.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:10.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:10.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:10.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:10.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:10.535+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:10.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:10.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:10.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:10.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:10.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:10.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:10.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:10.713+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:11.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:11.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:11.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:11.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:11.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:11.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:11.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:11.191+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:11.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:37:11.464+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 29293 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:37:11.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:11.681+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:11.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:11.681+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:11.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:11.681+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:11.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:11.682+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:11.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:11.577+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:11.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:11.577+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:11.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:11.577+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:11.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:11.577+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:12.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:12.200+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:12.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:12.200+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:12.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:12.200+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:12.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:12.201+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:12.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:12.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:12.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:12.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:12.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:12.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:12.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:12.605+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:12.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:12.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:12.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:12.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:12.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:12.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:12.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:12.732+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:13.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:13.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:13.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:13.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:13.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:13.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:13.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:13.179+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:13.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:13.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:13.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:13.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:13.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:13.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:13.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:13.655+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:13.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:13.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:13.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:13.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:13.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:13.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:13.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:13.752+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:14.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:14.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:14.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:14.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:14.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:14.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:14.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:14.178+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:14.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:14.630+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:14.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:14.631+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:14.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:14.631+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:14.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:14.631+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:14.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:14.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:14.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:14.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:14.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:14.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:14.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:14.727+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:15.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:15.186+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:15.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:15.187+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:15.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:15.187+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:15.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:15.187+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:15.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:15.589+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:15.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:15.589+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:15.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:15.589+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:15.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:15.589+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:15.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:15.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:15.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:15.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:15.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:15.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:15.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:15.739+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:16.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:16.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:16.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:16.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:16.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:16.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:16.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:16.161+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:16.693 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:37:16.467+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 29382 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:37:16.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:16.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:16.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:16.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:16.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:16.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:16.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:16.598+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:17.103 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:16.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:17.103 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:16.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:17.103 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:16.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:17.103 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:16.782+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:17.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:17.129+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:17.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:17.129+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:17.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:17.129+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:17.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:17.129+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:17.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:17.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:17.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:17.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:17.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:17.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:17.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:17.795+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:17.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:17.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:17.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:17.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:17.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:17.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:17.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:17.575+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:18.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:18.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:18.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:18.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:18.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:18.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:18.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:18.143+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:18.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:18.570+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:18.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:18.570+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:18.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:18.570+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:18.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:18.571+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:18.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:18.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:18.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:18.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:18.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:18.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:18.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:18.755+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:19.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:19.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:19.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:19.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:19.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:19.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:19.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:19.174+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:19.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:19.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:19.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:19.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:19.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:19.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:19.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:19.598+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:19.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:19.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:19.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:19.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:19.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:19.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:19.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:19.754+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:20.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:20.204+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:20.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:20.204+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:20.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:20.204+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:20.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:20.205+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:20.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:20.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:20.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:20.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:20.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:20.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:20.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:20.586+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:20.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:20.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:20.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:20.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:20.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:20.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:20.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:20.706+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:21.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:21.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:21.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:21.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:21.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:21.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:21.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:21.209+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:21.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:37:21.470+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 29469 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:37:21.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:21.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:21.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:21.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:21.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:21.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:21.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:21.664+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:21.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:21.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:21.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:21.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:21.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:21.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:21.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:21.608+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:22.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:22.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:22.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:22.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:22.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:22.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:22.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:22.199+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:22.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:22.589+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:22.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:22.589+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:22.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:22.589+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:22.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:22.590+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:22.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:22.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:22.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:22.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:22.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:22.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:22.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:22.711+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:23.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:23.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:23.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:23.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:23.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:23.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:23.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:23.189+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:23.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:23.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:23.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:23.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:23.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:23.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:23.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:23.580+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:24.126 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:23.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:24.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:23.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:24.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:23.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:24.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:23.740+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:24.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:24.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:24.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:24.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:24.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:24.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:24.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:24.152+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:24.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:24.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:24.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:24.559+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:24.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:24.559+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:24.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:24.560+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:24.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:24.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:24.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:24.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:24.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:24.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:24.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:24.692+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:25.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:25.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:25.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:25.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:25.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:25.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:25.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:25.114+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:25.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:25.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:25.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:25.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:25.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:25.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:25.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:25.673+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:25.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:25.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:25.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:25.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:25.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:25.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:25.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:25.549+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:26.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:26.119+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:26.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:26.119+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:26.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:26.119+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:26.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:26.120+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:26.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:37:26.473+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 29559 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:37:26.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:26.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:26.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:26.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:26.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:26.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:26.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:26.534+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:26.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:26.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:26.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:26.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:26.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:26.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:26.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:26.633+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:27.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:27.154+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:27.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:27.154+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:27.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:27.154+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:27.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:27.154+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:27.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:27.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:27.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:27.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:27.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:27.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:27.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:27.585+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:27.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:27.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:27.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:27.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:27.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:27.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:27.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:27.621+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:28.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:28.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:28.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:28.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:28.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:28.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:28.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:28.148+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:28.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:28.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:28.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:28.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:28.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:28.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:28.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:28.628+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:28.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:28.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:28.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:28.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:28.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:28.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:28.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:28.654+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:29.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:29.165+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:29.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:29.165+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:29.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:29.165+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:29.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:29.165+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:29.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:29.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:29.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:29.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:29.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:29.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:29.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:29.599+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:29.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:29.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:29.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:29.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:29.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:29.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:29.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:29.673+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:30.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:30.116+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:30.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:30.116+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:30.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:30.116+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:30.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:30.116+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:30.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:30.577+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:30.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:30.577+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:30.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:30.577+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:30.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:30.577+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:30.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:30.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:30.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:30.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:30.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:30.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:30.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:30.721+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:31.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:31.075+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:31.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:31.075+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:31.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:31.075+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:31.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:31.075+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:31.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:37:31.477+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 29645 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:37:31.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:31.566+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:31.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:31.566+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:31.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:31.566+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:31.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:31.567+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:31.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:31.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:31.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:31.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:31.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:31.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:31.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:31.683+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:32.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:32.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:32.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:32.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:32.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:32.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:32.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:32.066+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:32.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:32.589+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:32.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:32.589+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:32.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:32.589+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:32.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:32.590+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:32.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:32.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:32.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:32.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:32.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:32.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:32.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:32.635+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:33.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:33.095+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:33.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:33.095+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:33.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:33.095+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:33.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:33.096+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:33.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:33.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:33.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:33.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:33.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:33.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:33.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:33.629+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:33.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:33.631+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:33.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:33.631+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:33.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:33.631+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:33.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:33.632+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:34.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:34.112+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:34.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:34.112+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:34.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:34.112+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:34.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:34.113+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:34.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:34.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:34.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:34.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:34.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:34.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:34.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:34.676+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:34.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:34.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:34.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:34.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:34.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:34.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:34.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:34.655+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:35.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:35.095+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:35.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:35.095+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:35.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:35.095+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:35.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:35.096+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:35.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:35.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:35.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:35.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:35.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:35.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:35.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:35.660+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:35.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:35.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:35.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:35.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:35.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:35.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:35.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:35.670+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:36.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:36.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:36.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:36.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:36.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:36.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:36.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:36.125+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:36.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:37:36.481+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 29736 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:37:36.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:36.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:36.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:36.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:36.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:36.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:36.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:36.698+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:36.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:36.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:36.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:36.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:36.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:36.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:36.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:36.659+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:37.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:37.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:37.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:37.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:37.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:37.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:37.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:37.171+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:37.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:37.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:37.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:37.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:37.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:37.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:37.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:37.658+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:37.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:37.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:37.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:37.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:37.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:37.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:37.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:37.700+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:38.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:38.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:38.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:38.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:38.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:38.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:38.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:38.222+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:38.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:38.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:38.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:38.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:38.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:38.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:38.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:38.686+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:38.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:38.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:38.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:38.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:38.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:38.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:38.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:38.746+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:39.624 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:39.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:39.625 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:39.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:39.625 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:39.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:39.625 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:39.249+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:39.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:39.732+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:39.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:39.732+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:39.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:39.732+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:39.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:39.733+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:39.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:39.651+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:39.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:39.651+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:39.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:39.651+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:39.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:39.651+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:40.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:40.231+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:40.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:40.231+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:40.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:40.231+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:40.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:40.232+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:40.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:40.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:40.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:40.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:40.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:40.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:40.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:40.763+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:40.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:40.635+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:40.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:40.635+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:40.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:40.635+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:40.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:40.636+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:41.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:41.191+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:41.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:41.191+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:41.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:41.191+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:41.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:41.191+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:41.729 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:37:41.484+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 29823 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:37:41.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:41.669+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:41.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:41.669+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:41.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:41.669+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:41.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:41.670+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:42.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:41.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:42.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:41.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:42.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:41.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:42.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:41.756+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:42.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:42.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:42.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:42.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:42.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:42.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:42.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:42.204+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:42.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:42.644+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:42.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:42.644+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:42.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:42.644+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:42.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:42.644+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:42.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:42.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:42.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:42.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:42.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:42.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:42.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:42.797+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:43.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:43.243+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:43.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:43.243+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:43.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:43.243+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:43.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:43.243+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:43.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:43.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:43.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:43.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:43.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:43.644+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:43.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:43.644+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:43.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:43.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:43.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:43.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:43.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:43.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:43.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:43.813+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:44.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:44.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:44.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:44.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:44.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:44.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:44.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:44.292+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:44.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:44.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:44.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:44.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:44.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:44.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:44.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:44.676+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:44.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:44.819+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:44.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:44.819+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:44.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:44.819+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:44.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:44.820+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:45.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:45.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:45.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:45.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:45.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:45.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:45.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:45.260+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:46.193 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:45.721+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:46.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:45.721+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:46.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:45.721+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:46.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:45.722+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:46.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:45.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:46.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:45.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:46.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:45.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:46.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:45.870+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:46.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:46.223+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:46.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:46.223+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:46.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:46.223+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:46.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:46.224+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:46.828 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:37:46.488+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 29913 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:37:46.828 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:46.714+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:46.828 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:46.714+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:46.828 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:46.714+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:46.828 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:46.714+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:47.180 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:46.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:47.180 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:46.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:47.180 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:46.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:47.180 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:46.855+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:47.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:47.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:47.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:47.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:47.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:47.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:47.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:47.206+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:48.146 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:47.760+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:48.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:47.760+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:48.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:47.760+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:48.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:47.761+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:48.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:47.809+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:48.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:47.809+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:48.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:47.809+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:48.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:47.809+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:48.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:48.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:48.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:48.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:48.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:48.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:48.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:48.173+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:48.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:48.859+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:48.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:48.859+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:48.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:48.859+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:48.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:48.859+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:48.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:48.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:48.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:48.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:48.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:48.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:48.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:48.718+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:49.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:49.200+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:49.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:49.200+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:49.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:49.200+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:49.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:49.200+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:50.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:49.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:50.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:49.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:50.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:49.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:50.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:49.891+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:50.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:49.755+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:50.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:49.755+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:50.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:49.755+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:50.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:49.755+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:50.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:50.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:50.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:50.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:50.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:50.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:50.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:50.236+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:51.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:50.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:51.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:50.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:51.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:50.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:51.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:50.893+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:51.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:50.765+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:51.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:50.765+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:51.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:50.765+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:51.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:50.766+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:51.587 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:37:51.491+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 30000 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:37:51.588 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:51.272+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:51.588 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:51.272+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:51.588 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:51.272+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:51.588 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:51.272+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:51.872 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:51.810+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:51.872 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:51.810+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:51.872 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:51.810+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:51.872 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:51.811+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:52.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:51.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:52.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:51.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:52.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:51.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:52.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:51.899+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:52.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:52.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:52.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:52.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:52.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:52.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:52.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:52.276+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:53.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:52.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:53.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:52.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:53.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:52.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:16.212935+0000 (oldest deadline 2024-04-27T02:27:36.212935+0000) 2024-04-27T02:37:53.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:52.852+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:53.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:52.778+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:53.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:52.778+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:53.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:52.778+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:53.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:52.778+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:53.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:53.250+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:53.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:53.250+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:53.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:53.250+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:53.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:53.251+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:53.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:53.807+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:53.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:53.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:53.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:53.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:53.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:53.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:53.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:53.788+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:54.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:54.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:54.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:54.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:54.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:54.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:54.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:54.270+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:55.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:54.815+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:55.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:54.761+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:55.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:54.761+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:55.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:54.761+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:55.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:54.762+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:55.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:55.266+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:55.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:55.266+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:55.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:55.266+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:55.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:55.267+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:55.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:55.848+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:55.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:55.717+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:55.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:55.717+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:55.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:55.717+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:55.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:55.717+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:56.593 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:37:56.495+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 30091 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:37:56.593 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:56.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:56.594 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:56.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:56.594 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:56.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:56.594 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:56.315+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:56.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:56.830+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:56.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:56.752+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:56.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:56.752+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:56.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:56.752+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:56.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:56.752+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:57.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:57.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:57.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:57.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:57.680 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:57.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:57.680 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:57.364+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:57.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:57.870+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:57.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:57.705+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:57.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:57.705+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:57.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:57.705+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:57.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:57.706+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:58.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:58.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:58.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:58.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:58.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:58.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:58.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:58.413+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:59.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:58.878+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:59.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:58.732+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:59.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:58.732+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:59.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:58.732+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:59.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:58.733+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:59.672 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:59.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:59.672 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:59.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:59.672 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:59.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:37:59.672 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:37:59.373+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:59.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:37:59.879+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:37:59.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:59.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:59.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:59.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:59.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:59.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:37:59.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:37:59.699+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:00.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:00.362+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:38:00.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:00.362+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:38:00.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:00.362+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:27.060986+0000 (oldest deadline 2024-04-27T02:27:47.060986+0000) 2024-04-27T02:38:00.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:00.363+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:00.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:00.835+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:00.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:00.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:38:00.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:00.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:38:00.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:00.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:38:00.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:00.687+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:01.693 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:38:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:38:01.498+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 30178 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:38:01.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:01.403+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:01.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:01.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:38:01.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:01.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:38:01.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:01.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:38:01.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:01.644+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:02.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:01.788+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:02.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:02.384+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:02.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:02.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:38:02.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:02.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:38:02.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:02.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:38:02.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:02.620+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:03.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:02.798+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:03.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:03.421+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:03.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:03.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:38:03.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:03.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:38:03.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:03.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:38:03.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:03.611+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:04.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:03.832+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:04.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:04.415+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:04.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:04.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:38:04.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:04.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:38:04.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:04.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:27:35.307818+0000 (oldest deadline 2024-04-27T02:27:55.307818+0000) 2024-04-27T02:38:04.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:04.602+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:05.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:04.829+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:05.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:05.603+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:05.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:05.416+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:06.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:05.820+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:06.599 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:38:06.502+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 30268 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:38:06.599 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:06.387+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:06.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:06.576+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:06.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:06.802+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:07.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:07.597+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:07.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:07.373+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:08.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:07.835+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:08.606 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:08.326+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:08.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:08.633+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:08.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:08.876+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:09.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:09.285+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:09.827 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:09.658+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:10.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:09.853+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:10.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:10.281+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:10.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:10.698+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:10.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:10.814+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:11.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:11.280+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:11.603 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:38:11.505+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 30356 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:38:11.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:11.662+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:11.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:11.772+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:12.579 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:12.235+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:12.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:12.640+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:12.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:12.779+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:13.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:13.186+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:13.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:13.673+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:13.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:13.802+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:14.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:14.144+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:14.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:14.652+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:14.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:14.763+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:15.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:15.139+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:15.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:15.623+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:16.128 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:15.728+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:16.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:16.155+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:16.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:38:16.509+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 30447 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:38:16.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:16.642+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:16.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:16.705+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:17.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:17.159+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:17.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:17.626+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:17.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:17.742+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:18.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:18.199+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:18.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:18.671+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:18.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:18.752+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:19.599 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:19.222+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:19.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:19.626+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:19.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:19.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:19.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:19.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:19.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:19.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:19.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:19.711+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:20.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:20.263+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:20.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:20.644+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:20.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:20.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:20.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:20.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:20.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:20.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:20.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:20.667+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:21.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:21.214+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:21.795 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:38:21.511+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 30534 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:38:21.795 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:21.684+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:21.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:21.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:21.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:21.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:21.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:21.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:21.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:21.660+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:22.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:22.234+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:22.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:22.702+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:22.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:22.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:22.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:22.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:22.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:22.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:22.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:22.617+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:23.590 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:23.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:23.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:23.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:23.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:23.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:23.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:23.239+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:23.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:23.661+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:23.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:23.616+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:23.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:23.616+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:23.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:23.616+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:23.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:23.616+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:24.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:24.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:24.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:24.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:24.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:24.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:24.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:24.240+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:24.866 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:24.636+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:24.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:24.654+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:24.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:24.654+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:24.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:24.654+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:24.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:24.654+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:25.505 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:25.244+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:25.506 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:25.244+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:25.506 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:25.244+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:25.506 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:25.244+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:25.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:25.625+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:25.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:25.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:25.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:25.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:25.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:25.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:25.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:25.608+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:26.490 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:26.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:26.490 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:26.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:26.490 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:26.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:26.490 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:26.257+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:26.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:38:26.515+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 30624 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:38:26.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:26.640+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:26.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:26.639+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:26.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:26.639+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:26.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:26.639+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:26.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:26.640+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:27.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:27.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:27.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:27.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:27.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:27.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:27.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:27.215+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:27.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:27.622+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:27.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:27.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:27.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:27.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:27.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:27.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:27.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:27.639+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:28.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:28.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:28.557 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:28.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:28.557 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:28.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:28.557 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:28.261+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:28.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:28.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:28.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:28.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:28.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:28.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:28.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:28.583+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:28.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:28.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:28.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:28.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:28.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:28.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:28.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:28.643+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:29.535 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:29.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:29.535 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:29.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:29.535 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:29.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:29.535 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:29.255+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:29.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:29.560+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:29.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:29.560+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:29.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:29.560+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:29.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:29.561+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:29.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:29.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:29.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:29.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:29.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:29.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:29.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:29.620+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:30.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:30.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:30.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:30.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:30.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:30.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:30.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:30.248+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:30.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:30.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:30.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:30.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:30.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:30.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:30.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:30.605+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:30.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:30.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:30.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:30.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:30.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:30.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:30.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:30.643+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:31.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:31.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:31.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:31.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:31.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:31.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:31.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:31.208+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:31.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:38:31.519+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 30711 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:38:31.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:31.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:31.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:31.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:31.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:31.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:31.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:31.590+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:31.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:31.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:31.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:31.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:31.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:31.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:31.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:31.634+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:32.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:32.175+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:32.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:32.175+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:32.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:32.175+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:32.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:32.176+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:32.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:32.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:32.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:32.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:32.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:32.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:32.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:32.630+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:32.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:32.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:32.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:32.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:32.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:32.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:32.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:32.592+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:33.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:33.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:33.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:33.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:33.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:33.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:33.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:33.206+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:33.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:33.553+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:33.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:33.553+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:33.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:33.553+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:33.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:33.553+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:33.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:33.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:33.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:33.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:33.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:33.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:33.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:33.634+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:34.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:34.177+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:34.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:34.177+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:34.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:34.177+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:34.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:34.177+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:34.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:34.616+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:34.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:34.616+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:34.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:34.616+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:34.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:34.617+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:34.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:34.528+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:34.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:34.528+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:34.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:34.528+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:34.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:34.529+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:35.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:35.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:35.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:35.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:35.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:35.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:35.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:35.181+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:35.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:35.660+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:35.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:35.660+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:35.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:35.660+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:35.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:35.661+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:35.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:35.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:35.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:35.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:35.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:35.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:35.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:35.512+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:36.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:36.131+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:36.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:36.131+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:36.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:36.131+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:36.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:36.132+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:36.693 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:38:36.524+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 30802 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:38:36.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:36.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:36.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:36.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:36.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:36.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:36.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:36.468+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:36.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:36.691+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:36.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:36.691+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:36.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:36.691+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:36.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:36.691+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:37.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:37.176+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:37.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:37.176+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:37.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:37.176+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:37.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:37.176+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:37.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:37.672+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:37.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:37.672+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:37.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:37.672+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:37.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:37.673+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:37.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:37.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:37.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:37.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:37.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:37.507+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:37.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:37.507+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:38.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:38.153+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:38.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:38.153+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:38.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:38.153+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:38.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:38.154+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:38.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:38.656+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:38.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:38.656+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:38.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:38.656+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:38.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:38.657+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:38.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:38.539+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:38.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:38.539+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:38.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:38.539+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:38.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:38.540+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:39.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:39.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:39.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:39.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:39.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:39.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:39.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:39.141+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:39.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:39.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:39.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:39.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:39.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:39.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:39.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:39.613+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:39.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:39.563+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:39.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:39.563+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:39.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:39.563+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:39.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:39.564+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:40.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:40.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:40.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:40.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:40.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:40.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:40.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:40.133+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:40.803 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:40.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:40.803 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:40.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:40.803 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:40.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:40.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:40.629+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:40.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:40.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:40.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:40.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:40.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:40.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:40.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:40.535+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:41.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:41.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:41.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:41.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:41.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:41.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:41.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:41.141+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:41.944 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:38:41.528+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 30890 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:38:41.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:41.665+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:41.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:41.665+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:41.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:41.665+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:41.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:41.666+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:41.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:41.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:41.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:41.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:41.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:41.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:41.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:41.571+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:42.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:42.130+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:42.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:42.130+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:42.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:42.130+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:42.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:42.130+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:42.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:42.671+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:42.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:42.671+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:42.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:42.671+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:42.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:42.672+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:42.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:42.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:42.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:42.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:42.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:42.560+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:42.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:42.561+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:43.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:43.168+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:43.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:43.168+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:43.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:43.168+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:43.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:43.169+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:43.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:43.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:43.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:43.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:43.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:43.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:43.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:43.667+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:43.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:43.516+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:43.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:43.516+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:43.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:43.516+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:43.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:43.517+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:44.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:44.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:44.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:44.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:44.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:44.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:44.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:44.206+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:44.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:44.657+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:44.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:44.657+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:44.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:44.657+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:44.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:44.657+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:44.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:44.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:44.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:44.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:44.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:44.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:44.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:44.499+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:45.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:45.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:45.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:45.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:45.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:45.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:45.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:45.215+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:45.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:45.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:45.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:45.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:45.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:45.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:45.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:45.699+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:45.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:45.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:45.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:45.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:45.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:45.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:45.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:45.502+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:46.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:46.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:46.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:46.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:46.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:46.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:46.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:46.206+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:46.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:38:46.532+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 30980 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:38:46.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:46.685+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:46.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:46.685+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:46.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:46.685+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:46.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:46.685+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:46.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:46.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:46.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:46.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:46.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:46.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:46.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:46.525+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:47.538 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:47.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:47.538 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:47.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:47.538 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:47.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:47.538 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:47.229+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:47.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:47.648+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:47.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:47.648+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:47.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:47.648+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:47.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:47.649+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:47.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:47.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:47.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:47.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:47.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:47.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:47.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:47.565+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:48.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:48.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:48.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:48.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:48.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:48.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:48.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:48.248+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:48.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:48.603+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:48.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:48.603+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:48.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:48.603+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:48.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:48.605+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:48.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:48.601+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:48.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:48.601+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:48.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:48.601+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:48.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:48.601+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:49.551 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:49.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:49.551 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:49.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:49.551 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:49.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:49.551 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:49.289+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:49.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:49.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:49.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:49.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:49.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:49.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:49.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:49.583+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:49.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:49.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:49.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:49.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:49.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:49.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:49.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:49.579+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:50.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:50.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:50.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:50.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:50.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:50.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:50.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:50.254+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:50.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:50.585+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:50.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:50.585+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:50.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:50.585+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:50.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:50.586+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:50.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:50.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:50.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:50.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:50.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:50.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:50.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:50.584+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:51.509 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:51.226+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:51.509 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:51.226+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:51.509 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:51.226+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:51.509 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:51.227+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:51.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:38:51.535+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 31068 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:38:51.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:51.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:51.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:51.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:51.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:51.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:51.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:51.596+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:51.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:51.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:51.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:51.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:51.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:51.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:51.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:51.628+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:52.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:52.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:52.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:52.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:52.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:52.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:52.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:52.219+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:52.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:52.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:52.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:52.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:52.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:52.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:52.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:52.580+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:52.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:52.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:52.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:52.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:52.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:52.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:52.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:52.595+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:53.548 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:53.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:53.548 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:53.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:53.548 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:53.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:53.548 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:53.232+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:53.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:53.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:53.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:53.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:53.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:53.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:53.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:53.575+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:53.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:53.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:53.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:53.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:53.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:53.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:53.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:53.630+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:54.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:54.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:54.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:54.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:54.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:54.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:54.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:54.194+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:54.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:54.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:54.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:54.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:54.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:54.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:54.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:54.621+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:54.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:54.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:54.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:54.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:54.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:54.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:54.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:54.625+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:55.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:55.192+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:55.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:55.192+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:55.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:55.192+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:55.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:55.192+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:55.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:55.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:55.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:55.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:55.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:55.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:55.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:55.587+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:55.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:55.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:55.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:55.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:55.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:55.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:55.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:55.630+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:56.512 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:56.238+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:56.513 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:56.238+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:56.513 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:56.238+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:56.513 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:56.238+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:56.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:38:56.539+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 31159 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:38:56.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:56.556+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:56.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:56.556+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:56.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:56.556+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:56.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:56.557+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:56.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:56.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:56.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:56.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:56.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:56.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:56.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:56.621+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:57.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:57.275+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:57.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:57.275+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:57.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:57.275+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:57.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:57.276+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:57.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:57.570+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:57.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:57.570+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:57.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:57.570+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:57.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:57.571+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:57.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:57.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:57.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:57.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:57.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:57.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:57.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:57.624+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:58.514 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:58.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:58.514 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:58.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:58.514 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:58.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:58.514 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:58.280+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:58.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:58.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:58.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:58.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:58.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:58.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:58.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:58.593+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:58.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:58.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:58.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:58.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:58.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:58.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:58.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:58.541+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:59.532 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:59.230+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:59.533 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:59.230+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:59.533 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:59.230+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:38:59.533 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:38:59.231+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:59.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:59.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:59.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:59.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:59.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:59.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:38:59.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:38:59.630+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:38:59.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:59.559+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:59.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:59.559+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:59.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:59.559+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:38:59.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:38:59.560+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:00.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:00.213+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:00.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:00.213+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:00.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:00.213+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:00.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:00.213+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:00.786 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:00.650+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:00.787 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:00.650+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:00.787 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:00.650+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:00.787 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:00.651+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:00.787 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:00.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:00.787 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:00.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:00.787 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:00.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:00.787 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:00.539+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:01.485 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:01.227+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:01.485 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:01.227+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:01.485 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:01.227+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:01.485 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:01.227+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:01.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:39:01.543+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 31247 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:39:01.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:01.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:01.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:01.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:01.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:01.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:01.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:01.660+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:01.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:01.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:01.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:01.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:01.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:01.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:01.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:01.513+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:02.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:02.235+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:02.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:02.235+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:02.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:02.235+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:02.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:02.235+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:02.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:02.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:02.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:02.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:02.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:02.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:02.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:02.505+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:02.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:02.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:02.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:02.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:02.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:02.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:02.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:02.699+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:03.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:03.199+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:03.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:03.199+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:03.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:03.199+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:03.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:03.200+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:03.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:03.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:03.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:03.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:03.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:03.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:03.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:03.554+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:03.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:03.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:03.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:03.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:03.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:03.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:03.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:03.678+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:04.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:04.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:04.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:04.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:04.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:04.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:04.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:04.168+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:04.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:04.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:04.803 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:04.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:04.803 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:04.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:04.803 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:04.511+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:04.803 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:04.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:04.803 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:04.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:04.803 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:04.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:04.803 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:04.725+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:05.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:05.157+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:05.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:05.157+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:05.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:05.157+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:05.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:05.157+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:05.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:05.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:05.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:05.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:05.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:05.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:05.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:05.467+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:05.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:05.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:05.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:05.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:05.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:05.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:05.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:05.688+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:06.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:06.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:06.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:06.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:06.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:06.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:06.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:06.175+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:06.693 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:39:06.547+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 31336 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:39:06.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:06.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:06.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:06.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:06.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:06.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:06.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:06.672+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:06.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:06.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:06.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:06.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:06.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:06.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:06.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:06.455+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:07.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:07.204+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:07.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:07.204+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:07.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:07.204+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:07.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:07.204+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:07.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:07.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:07.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:07.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:07.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:07.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:07.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:07.465+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:07.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:07.714+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:07.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:07.714+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:07.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:07.714+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:07.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:07.714+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:08.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:08.189+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:08.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:08.189+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:08.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:08.189+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:08.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:08.190+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:08.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:08.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:08.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:08.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:08.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:08.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:08.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:08.437+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:09.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:08.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:09.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:08.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:09.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:08.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:09.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:08.740+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:09.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:09.226+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:09.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:09.226+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:09.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:09.226+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:09.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:09.226+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:09.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:09.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:09.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:09.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:09.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:09.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:09.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:09.443+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:10.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:09.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:10.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:09.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:10.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:09.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:10.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:09.767+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:10.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:10.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:10.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:10.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:10.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:10.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:10.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:10.240+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:10.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:10.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:10.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:10.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:10.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:10.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:10.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:10.464+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:11.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:10.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:11.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:10.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:11.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:10.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:11.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:10.806+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:11.524 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:11.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:11.525 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:11.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:11.525 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:11.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:11.525 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:11.498+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:11.525 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:11.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:11.525 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:11.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:11.526 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:11.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:11.526 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:11.290+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:11.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:39:11.550+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 31423 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:39:11.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:11.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:11.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:11.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:11.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:11.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:11.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:11.786+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:12.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:12.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:12.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:12.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:12.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:12.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:12.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:12.293+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:12.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:12.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:12.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:12.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:12.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:12.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:12.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:12.449+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:13.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:12.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:13.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:12.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:13.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:12.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:13.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:12.808+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:13.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:13.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:13.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:13.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:13.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:13.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:13.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:13.486+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:13.616 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:13.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:13.616 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:13.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:13.616 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:13.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:13.616 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:13.278+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:13.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:13.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:13.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:13.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:13.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:13.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:13.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:13.813+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:14.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:14.280+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:14.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:14.280+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:14.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:14.280+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:14.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:14.280+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:14.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:14.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:14.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:14.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:14.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:14.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:14.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:14.455+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:15.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:14.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:15.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:14.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:15.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:14.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:15.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:14.805+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:15.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:15.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:15.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:15.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:15.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:15.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:15.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:15.497+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:15.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:15.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:15.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:15.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:15.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:15.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:15.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:15.295+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:16.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:15.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:16.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:15.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:16.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:15.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:16.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:15.842+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:16.538 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:39:16.555+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 31514 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:39:16.539 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:16.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:16.539 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:16.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:16.539 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:16.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:16.539 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:16.308+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:16.539 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:16.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:16.539 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:16.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:16.539 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:16.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:16.540 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:16.512+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:17.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:16.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:17.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:16.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:17.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:16.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:17.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:16.826+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:17.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:17.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:17.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:17.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:17.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:17.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:17.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:17.521+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:17.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:17.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:17.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:17.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:17.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:17.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:17.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:17.319+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:18.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:17.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:18.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:17.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:18.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:17.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:18.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:17.812+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:18.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:18.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:18.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:18.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:18.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:18.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:18.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:18.478+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:18.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:18.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:18.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:18.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:18.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:18.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:18.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:18.308+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:19.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:18.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:19.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:18.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:19.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:18.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:19.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:18.802+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:19.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:19.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:19.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:19.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:19.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:19.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:19.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:19.463+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:19.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:19.327+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:19.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:19.327+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:19.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:19.327+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:19.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:19.327+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:20.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:19.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:20.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:19.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:20.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:19.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:20.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:19.802+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:20.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:20.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:20.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:20.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:20.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:20.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:20.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:20.477+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:20.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:20.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:20.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:20.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:20.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:20.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:20.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:20.281+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:21.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:20.848+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:21.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:20.848+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:21.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:20.848+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:21.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:20.849+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:21.543 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:39:21.557+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 31602 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:39:21.543 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:21.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:21.543 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:21.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:21.543 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:21.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:21.543 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:21.501+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:21.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:21.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:21.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:21.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:21.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:21.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:21.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:21.311+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:22.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:21.835+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:22.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:21.835+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:22.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:21.835+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:22.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:21.835+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:22.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:22.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:22.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:22.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:22.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:22.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:22.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:22.307+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:22.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:22.473+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:22.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:22.473+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:22.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:22.473+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:22.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:22.473+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:23.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:22.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:23.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:22.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:23.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:22.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:23.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:22.861+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:23.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:23.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:23.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:23.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:23.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:23.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:23.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:23.433+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:23.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:23.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:23.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:23.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:23.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:23.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:23.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:23.293+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:23.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:23.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:23.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:23.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:23.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:23.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:23.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:23.857+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:24.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:24.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:24.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:24.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:24.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:24.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:24.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:24.475+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:24.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:24.265+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:24.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:24.265+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:24.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:24.265+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:24.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:24.265+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:25.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:24.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:25.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:24.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:25.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:24.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:25.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:24.898+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:25.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:25.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:25.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:25.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:25.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:25.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:25.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:25.475+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:25.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:25.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:25.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:25.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:25.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:25.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:25.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:25.270+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:26.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:25.917+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:26.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:25.917+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:26.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:25.917+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:26.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:25.917+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:26.492 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:26.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:26.492 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:26.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:26.492 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:26.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:26.492 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:26.303+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:26.857 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:39:26.561+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 31692 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:39:26.857 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:26.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:26.857 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:26.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:26.857 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:26.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:26.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:26.519+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:27.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:26.883+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:27.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:26.883+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:27.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:26.883+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:27.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:26.883+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:27.491 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:27.318+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:27.491 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:27.318+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:27.491 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:27.318+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:27.491 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:27.319+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:27.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:27.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:27.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:27.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:27.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:27.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:27.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:27.517+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:28.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:27.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:28.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:27.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:28.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:27.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:28.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:27.930+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:28.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:28.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:28.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:28.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:28.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:28.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:28.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:28.350+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:28.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:28.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:28.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:28.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:28.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:28.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:28.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:28.497+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:29.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:28.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:29.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:28.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:29.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:28.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:29.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:28.969+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:29.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:29.305+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:29.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:29.305+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:29.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:29.305+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:29.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:29.305+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:29.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:29.531+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:29.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:29.531+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:29.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:29.531+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:29.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:29.532+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:30.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:29.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:30.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:29.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:30.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:29.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:30.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:29.929+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:30.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:30.333+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:30.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:30.333+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:30.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:30.333+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:30.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:30.334+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:30.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:30.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:30.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:30.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:30.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:30.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:30.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:30.525+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:31.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:30.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:31.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:30.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:31.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:30.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:31.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:30.938+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:31.553 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:39:31.566+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 31780 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:39:31.553 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:31.327+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:31.553 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:31.327+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:31.553 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:31.327+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:31.553 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:31.328+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:31.554 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:31.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:31.554 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:31.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:31.554 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:31.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:31.554 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:31.492+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:32.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:31.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:32.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:31.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:32.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:31.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:32.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:31.968+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:32.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:32.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:32.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:32.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:32.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:32.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:32.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:32.502+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:32.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:32.300+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:32.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:32.300+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:32.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:32.300+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:32.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:32.300+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:33.313 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:33.008+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:33.313 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:33.008+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:33.313 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:33.008+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:33.313 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:33.009+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:33.617 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:33.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:33.617 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:33.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:33.617 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:33.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:33.617 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:33.456+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:33.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:33.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:33.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:33.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:33.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:33.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:33.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:33.340+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:34.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:34.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:34.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:34.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:34.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:34.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:34.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:34.011+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:34.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:34.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:34.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:34.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:34.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:34.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:34.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:34.497+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:34.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:34.309+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:34.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:34.309+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:34.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:34.309+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:34.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:34.310+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:35.310 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:35.018+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:35.311 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:35.018+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:35.311 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:35.018+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:35.311 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:35.019+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:35.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:35.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:35.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:35.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:35.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:35.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:35.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:35.453+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:35.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:35.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:35.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:35.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:35.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:35.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:35.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:35.338+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:36.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:36.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:36.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:36.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:36.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:36.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:36.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:36.017+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:36.667 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:39:36.570+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 31871 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:39:36.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:36.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:36.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:36.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:36.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:36.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:36.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:36.375+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:36.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:36.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:36.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:36.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:36.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:36.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:36.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:36.429+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:37.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:37.064+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:37.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:37.064+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:37.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:37.064+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:37.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:37.065+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:37.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:37.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:37.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:37.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:37.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:37.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:37.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:37.404+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:37.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:37.346+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:37.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:37.346+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:37.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:37.346+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:37.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:37.346+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:38.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:38.044+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:38.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:38.044+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:38.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:38.044+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:38.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:38.045+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:38.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:38.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:38.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:38.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:38.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:38.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:38.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:38.384+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:38.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:38.421+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:38.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:38.421+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:38.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:38.421+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:38.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:38.422+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:39.354 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:39.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:39.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:39.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:39.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:39.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:39.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:39.094+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:39.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:39.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:39.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:39.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:39.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:39.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:39.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:39.381+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:39.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:39.393+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:39.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:39.393+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:39.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:39.393+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:39.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:39.394+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:40.370 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:40.119+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:40.370 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:40.119+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:40.370 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:40.119+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:40.370 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:40.120+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:40.370 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:40.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:40.370 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:40.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:40.370 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:40.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:40.370 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:40.351+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:40.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:40.396+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:40.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:40.396+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:40.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:40.396+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:40.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:40.397+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:41.349 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:41.114+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:41.349 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:41.114+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:41.349 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:41.114+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:41.349 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:41.114+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:41.672 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:39:41.574+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 31959 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:39:41.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:41.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:41.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:41.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:41.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:41.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:41.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:41.376+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:41.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:41.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:41.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:41.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:41.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:41.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:41.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:41.398+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:42.383 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:42.357+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:42.383 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:42.357+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:42.383 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:42.357+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:42.383 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:42.357+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:42.383 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:42.113+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:42.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:42.113+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:42.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:42.113+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:42.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:42.114+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:42.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:42.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:42.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:42.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:42.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:42.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:42.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:42.411+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:43.328 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:43.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:43.328 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:43.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:43.328 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:43.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:43.328 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:43.081+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:43.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:43.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:43.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:43.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:43.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:43.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:43.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:43.452+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:43.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:43.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:43.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:43.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:43.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:43.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:43.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:43.355+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:44.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:44.032+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:44.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:44.032+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:44.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:44.032+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:44.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:44.033+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:44.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:44.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:44.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:44.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:44.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:44.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:44.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:44.348+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:44.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:44.412+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:44.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:44.412+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:44.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:44.412+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:44.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:44.413+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:45.350 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:45.034+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:45.350 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:45.034+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:45.350 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:45.034+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:45.350 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:45.035+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:45.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:45.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:45.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:45.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:45.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:45.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:45.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:45.377+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:45.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:45.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:45.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:45.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:45.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:45.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:45.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:45.443+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:46.340 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:46.034+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:46.340 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:46.034+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:46.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:46.034+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:46.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:46.035+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:46.677 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:39:46.578+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 32048 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:39:46.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:46.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:46.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:46.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:46.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:46.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:46.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:46.459+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:46.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:46.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:46.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:46.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:46.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:46.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:46.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:46.391+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:47.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:47.012+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:47.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:47.012+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:47.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:47.012+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:47.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:47.013+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:47.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:47.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:47.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:47.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:47.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:47.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:47.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:47.372+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:47.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:47.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:47.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:47.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:47.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:47.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:47.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:47.451+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:48.329 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:47.973+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:48.329 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:47.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:48.329 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:47.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:48.329 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:47.974+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:48.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:48.355+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:48.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:48.355+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:48.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:48.355+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:48.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:48.355+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:48.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:48.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:48.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:48.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:48.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:48.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:48.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:48.472+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:49.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:48.984+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:49.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:48.984+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:49.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:48.984+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:49.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:48.985+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:49.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:49.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:49.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:49.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:49.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:49.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:49.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:49.350+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:49.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:49.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:49.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:49.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:49.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:49.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:49.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:49.446+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:50.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:49.959+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:50.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:49.959+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:50.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:49.959+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:50.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:49.960+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:50.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:50.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:50.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:50.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:50.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:50.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:50.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:50.387+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:50.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:50.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:50.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:50.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:50.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:50.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:50.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:50.449+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:51.186 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:50.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:51.187 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:50.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:51.187 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:50.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:51.187 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:50.939+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:51.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:51.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:51.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:51.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:51.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:51.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:51.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:51.351+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:51.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:51.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:51.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:51.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:51.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:51.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:51.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:51.447+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:51.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:39:51.582+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 32136 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:39:52.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:51.976+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:52.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:51.976+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:52.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:51.976+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:52.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:51.977+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:52.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:52.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:52.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:52.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:52.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:52.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:52.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:52.475+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:52.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:52.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:52.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:52.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:52.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:52.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:52.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:52.311+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:53.249 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:53.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:53.249 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:53.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:53.249 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:53.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:53.249 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:53.025+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:53.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:53.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:53.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:53.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:53.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:53.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:53.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:53.276+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:53.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:53.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:53.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:53.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:53.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:53.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:53.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:53.521+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:54.211 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:54.064+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:54.211 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:54.064+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:54.211 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:54.064+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:54.211 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:54.065+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:54.495 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:54.238+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:54.495 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:54.238+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:54.495 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:54.238+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:54.495 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:54.238+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:54.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:54.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:54.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:54.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:54.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:54.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:54.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:54.522+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:55.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:55.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:55.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:55.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:55.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:55.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:55.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:55.097+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:55.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:55.241+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:55.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:55.241+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:55.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:55.241+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:55.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:55.241+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:55.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:55.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:55.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:55.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:55.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:55.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:55.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:55.569+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:56.340 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:56.104+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:56.340 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:56.104+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:56.340 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:56.104+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:56.340 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:56.105+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:56.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:56.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:56.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:56.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:56.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:56.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:56.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:56.228+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:56.684 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:39:56.586+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 32223 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:39:56.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:56.532+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:56.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:56.532+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:56.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:56.532+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:56.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:56.533+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:57.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:57.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:57.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:57.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:57.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:57.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:57.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:57.092+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:57.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:57.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:57.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:57.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:57.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:57.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:57.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:57.208+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:57.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:57.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:57.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:57.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:57.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:57.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:57.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:57.498+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:58.422 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:58.175+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:58.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:58.175+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:58.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:58.175+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:58.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:58.175+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:58.423 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:58.136+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:58.423 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:58.136+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:58.423 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:58.136+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:58.423 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:58.137+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:58.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:58.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:58.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:58.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:58.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:58.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:58.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:58.450+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:59.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:59.090+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:59.399 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:59.090+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:59.399 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:59.090+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:39:59.399 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:39:59.091+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:59.399 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:59.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:59.399 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:59.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:59.399 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:59.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:39:59.399 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:39:59.162+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:39:59.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:59.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:59.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:59.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:59.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:59.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:39:59.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:39:59.426+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:00.354 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:00.068+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:00.354 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:00.068+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:00.354 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:00.068+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:00.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:00.068+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:00.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:00.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:00.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:00.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:00.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:00.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:00.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:00.179+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:00.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:00.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:00.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:00.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:00.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:00.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:00.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:00.380+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:01.310 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:01.048+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:01.311 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:01.048+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:01.311 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:01.048+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:01.311 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:01.049+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:01.311 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:01.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:01.312 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:01.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:01.312 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:01.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:01.312 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:01.215+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:01.564 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:01.337+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:01.564 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:01.337+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:01.564 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:01.337+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:01.564 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:01.338+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:01.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:40:01.590+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 32305 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:40:02.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:02.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:02.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:02.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:02.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:02.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:02.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:02.001+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:02.284 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:02.245+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:02.284 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:02.245+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:02.284 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:02.245+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:02.284 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:02.246+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:02.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:02.309+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:02.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:02.309+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:02.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:02.309+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:02.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:02.311+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:03.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:03.017+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:03.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:03.017+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:03.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:03.017+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:03.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:03.017+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:03.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:03.234+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:03.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:03.234+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:03.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:03.234+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:03.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:03.234+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:03.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:03.335+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:03.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:03.335+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:03.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:03.335+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:03.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:03.336+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:04.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:04.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:04.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:04.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:04.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:04.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:04.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:04.185+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:04.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:03.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:04.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:03.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:04.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:03.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:04.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:03.969+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:04.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:04.356+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:04.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:04.356+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:04.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:04.356+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:04.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:04.357+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:05.296 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:05.007+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:05.296 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:05.007+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:05.296 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:05.007+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:05.297 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:05.008+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:05.297 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:05.233+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:05.297 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:05.233+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:05.297 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:05.233+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:05.297 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:05.233+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:05.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:05.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:05.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:05.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:05.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:05.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:05.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:05.324+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:06.245 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:05.975+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:06.245 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:05.975+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:06.245 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:05.975+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:06.245 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:05.976+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:06.246 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:06.191+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:06.246 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:06.191+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:06.246 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:06.191+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:06.246 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:06.192+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:06.567 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:06.324+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:06.568 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:06.324+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:06.568 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:06.324+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:06.568 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:06.325+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:06.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:40:06.594+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 32386 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:40:07.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:06.962+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:07.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:06.962+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:07.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:06.962+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:07.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:06.963+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:07.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:07.318+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:07.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:07.318+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:07.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:07.319+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:07.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:07.319+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:07.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:07.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:07.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:07.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:07.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:07.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:07.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:07.237+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:08.333 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:07.997+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:08.334 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:07.997+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:08.334 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:07.997+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:08.334 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:07.998+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:08.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:08.238+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:08.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:08.238+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:08.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:08.238+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:08.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:08.238+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:08.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:08.360+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:08.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:08.360+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:08.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:08.360+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:08.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:08.361+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:09.235 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:09.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:09.236 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:09.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:09.236 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:09.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:09.236 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:09.001+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:09.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:09.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:09.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:09.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:09.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:09.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:09.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:09.263+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:09.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:09.337+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:09.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:09.337+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:09.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:09.337+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:09.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:09.337+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:10.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:09.951+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:10.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:09.951+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:10.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:09.951+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:10.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:09.952+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:10.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:10.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:10.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:10.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:10.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:10.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:10.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:10.301+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:10.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:10.266+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:10.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:10.266+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:10.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:10.266+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:10.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:10.266+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:11.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:10.907+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:11.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:10.907+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:11.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:10.907+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:11.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:10.908+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:11.572 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:11.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:11.572 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:11.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:11.572 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:11.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:11.572 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:11.333+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:11.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:11.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:11.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:11.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:11.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:11.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:11.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:11.254+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:11.846 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:40:11.598+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 32465 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:40:12.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:11.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:12.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:11.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:12.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:11.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:12.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:11.874+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:12.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:12.376+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:12.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:12.376+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:12.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:12.376+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:12.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:12.377+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:12.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:12.288+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:12.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:12.288+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:12.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:12.288+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:12.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:12.288+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:13.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:12.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:13.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:12.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:13.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:12.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:13.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:12.912+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:13.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:13.360+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:13.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:13.360+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:13.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:13.360+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:13.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:13.361+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:13.621 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:13.252+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:13.621 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:13.252+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:13.621 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:13.252+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:13.621 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:13.252+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:14.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:13.936+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:14.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:13.936+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:14.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:13.936+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:14.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:13.936+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:14.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:14.380+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:14.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:14.380+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:14.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:14.380+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:14.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:14.381+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:14.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:14.247+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:14.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:14.247+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:14.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:14.247+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:14.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:14.248+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:15.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:14.947+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:15.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:14.947+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:15.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:14.947+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:15.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:14.948+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:15.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:15.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:15.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:15.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:15.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:15.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:15.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:15.221+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:15.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:15.393+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:15.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:15.393+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:15.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:15.393+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:15.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:15.394+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:16.186 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:15.902+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:16.186 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:15.902+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:16.186 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:15.902+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:16.186 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:15.902+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:16.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:16.355+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:16.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:16.355+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:16.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:16.355+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:16.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:16.356+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:16.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:16.213+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:16.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:16.213+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:16.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:16.213+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:16.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:16.213+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:16.866 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:40:16.603+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 32546 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:40:17.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:16.893+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:17.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:16.893+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:17.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:16.893+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:17.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:16.894+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:17.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:17.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:17.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:17.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:17.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:17.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:17.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:17.359+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:17.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:17.242+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:17.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:17.242+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:17.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:17.242+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:17.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:17.242+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:18.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:17.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:18.186 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:17.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:18.186 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:17.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:18.186 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:17.909+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:18.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:18.212+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:18.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:18.212+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:18.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:18.212+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:18.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:18.212+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:18.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:18.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:18.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:18.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:18.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:18.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:18.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:18.336+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:19.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:18.943+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:19.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:18.943+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:19.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:18.943+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:19.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:18.945+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:19.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:19.172+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:19.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:19.172+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:19.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:19.172+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:19.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:19.173+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:19.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:19.287+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:19.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:19.287+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:19.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:19.287+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:19.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:19.289+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:20.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:19.987+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:20.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:19.987+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:20.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:19.987+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:20.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:19.989+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:20.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:20.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:20.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:20.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:20.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:20.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:20.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:20.217+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:20.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:20.303+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:20.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:20.303+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:20.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:20.303+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:20.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:20.304+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:21.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:20.964+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:21.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:20.964+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:21.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:20.964+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:21.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:20.965+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:21.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:21.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:21.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:21.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:21.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:21.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:21.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:21.172+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:21.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:21.328+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:21.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:21.328+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:21.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:21.328+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:21.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:21.328+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:21.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:40:21.606+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 32628 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:40:22.262 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:22.007+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:22.262 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:22.007+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:22.262 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:22.007+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:22.262 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:22.007+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:22.262 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:22.152+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:22.262 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:22.152+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:22.262 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:22.152+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:22.263 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:22.152+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:22.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:22.287+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:22.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:22.287+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:22.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:22.287+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:22.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:22.288+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:23.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:23.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:23.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:23.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:23.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:23.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:23.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:23.108+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:23.307 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:23.028+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:23.307 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:23.028+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:23.307 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:23.028+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:23.307 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:23.029+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:23.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:23.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:23.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:23.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:23.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:23.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:23.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:23.333+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:24.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:24.067+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:24.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:24.067+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:24.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:24.067+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:24.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:24.068+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:24.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:24.064+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:24.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:24.064+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:24.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:24.064+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:24.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:24.065+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:24.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:24.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:24.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:24.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:24.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:24.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:24.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:24.305+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:25.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:25.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:25.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:25.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:25.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:25.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:25.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:25.057+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:25.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:25.088+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:25.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:25.088+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:25.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:25.088+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:25.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:25.089+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:25.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:25.329+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:25.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:25.329+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:25.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:25.329+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:25.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:25.330+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:26.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:26.132+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:26.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:26.132+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:26.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:26.132+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:26.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:26.133+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:26.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:26.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:26.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:26.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:26.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:26.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:26.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:26.010+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:26.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:26.347+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:26.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:26.347+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:26.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:26.347+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:26.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:26.347+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:26.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:40:26.610+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 32715 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:40:27.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:27.141+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:27.310 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:27.141+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:27.310 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:27.141+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:27.310 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:27.142+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:27.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:27.007+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:27.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:27.007+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:27.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:27.007+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:27.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:27.008+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:27.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:27.335+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:27.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:27.335+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:27.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:27.335+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:27.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:27.336+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:28.106 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:27.977+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:28.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:27.977+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:28.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:27.977+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:28.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:27.977+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:28.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:28.132+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:28.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:28.132+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:28.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:28.132+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:28.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:28.133+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:28.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:28.344+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:28.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:28.344+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:28.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:28.344+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:28.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:28.345+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:29.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:29.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:29.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:29.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:29.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:29.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:29.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:29.097+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:29.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:28.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:29.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:28.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:29.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:28.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:29.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:28.939+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:29.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:29.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:29.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:29.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:29.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:29.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:29.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:29.369+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:30.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:30.090+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:30.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:30.090+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:30.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:30.090+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:30.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:30.091+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:30.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:29.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:30.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:29.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:30.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:29.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:30.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:29.921+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:30.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:30.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:30.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:30.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:30.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:30.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:30.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:30.386+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:31.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:31.080+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:31.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:31.080+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:31.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:31.080+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:31.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:31.080+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:31.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:30.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:31.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:30.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:31.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:30.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:31.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:30.956+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:31.693 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:40:31.615+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 32803 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:40:31.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:31.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:31.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:31.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:31.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:31.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:31.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:31.399+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:32.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:31.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:32.098 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:31.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:32.098 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:31.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:32.098 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:31.975+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:32.367 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:32.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:32.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:32.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:32.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:32.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:32.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:32.124+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:32.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:32.394+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:32.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:32.394+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:32.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:32.394+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:32.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:32.395+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:33.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:32.984+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:33.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:32.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:33.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:32.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:33.331 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:32.985+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:33.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:33.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:33.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:33.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:33.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:33.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:33.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:33.171+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:33.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:33.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:33.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:33.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:33.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:33.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:33.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:33.358+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:34.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:34.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:34.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:34.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:34.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:34.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:34.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:34.162+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:34.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:34.000+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:34.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:34.000+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:34.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:34.000+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:34.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:34.001+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:34.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:34.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:34.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:34.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:34.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:34.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:34.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:34.389+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:35.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:35.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:35.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:35.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:35.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:35.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:35.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:35.119+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:35.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:34.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:35.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:34.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:35.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:34.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:35.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:34.967+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:35.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:35.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:35.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:35.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:35.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:35.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:35.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:35.388+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:36.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:36.100+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:36.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:36.100+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:36.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:36.100+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:36.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:36.101+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:36.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:35.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:36.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:35.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:36.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:35.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:36.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:35.940+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:36.591 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:36.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:36.591 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:36.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:36.591 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:36.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:36.591 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:36.438+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:36.913 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:40:36.621+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 32894 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:40:37.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:37.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:37.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:37.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:37.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:37.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:37.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:37.130+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:37.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:36.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:37.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:36.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:37.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:36.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:37.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:36.940+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:37.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:37.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:37.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:37.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:37.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:37.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:37.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:37.413+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:38.434 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:38.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:38.434 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:38.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:38.434 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:38.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:38.434 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:38.162+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:38.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:37.979+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:38.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:37.979+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:38.435 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:37.979+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:38.435 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:37.979+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:38.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:38.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:38.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:38.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:38.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:38.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:38.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:38.461+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:39.391 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:39.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:39.392 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:39.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:39.392 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:39.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:39.392 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:39.202+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:39.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:38.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:39.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:38.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:39.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:38.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:39.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:38.980+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:39.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:39.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:39.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:39.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:39.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:39.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:39.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:39.419+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:40.410 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:40.221+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:40.410 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:40.221+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:40.411 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:40.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:40.411 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:40.222+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:40.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:39.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:40.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:39.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:40.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:39.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:40.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:39.986+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:40.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:40.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:40.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:40.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:40.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:40.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:40.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:40.438+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:41.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:41.245+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:41.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:41.245+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:41.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:41.245+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:41.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:41.246+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:41.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:41.000+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:41.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:41.000+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:41.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:41.000+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:41.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:41.001+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:41.723 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:40:41.625+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 32982 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:40:41.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:41.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:41.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:41.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:41.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:41.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:41.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:41.475+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:42.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:41.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:42.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:41.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:42.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:41.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:42.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:41.994+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:42.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:42.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:42.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:42.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:42.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:42.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:42.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:42.476+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:42.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:42.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:42.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:42.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:42.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:42.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:42.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:42.259+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:43.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:43.039+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:43.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:43.039+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:43.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:43.039+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:43.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:43.039+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:43.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:43.288+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:43.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:43.288+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:43.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:43.288+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:43.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:43.288+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:43.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:43.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:43.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:43.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:43.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:43.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:43.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:43.473+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:44.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:43.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:44.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:43.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:44.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:43.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:44.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:43.999+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:44.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:44.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:44.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:44.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:44.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:44.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:44.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:44.453+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:44.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:44.275+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:44.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:44.275+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:44.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:44.275+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:44.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:44.275+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:45.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:44.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:45.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:44.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:45.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:44.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:45.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:44.959+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:45.455 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:45.232+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:45.455 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:45.232+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:45.455 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:45.232+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:45.455 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:45.232+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:45.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:45.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:45.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:45.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:45.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:45.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:45.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:45.483+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:46.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:45.949+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:46.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:45.949+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:46.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:45.949+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:46.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:45.949+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:46.438 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:46.229+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:46.438 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:46.229+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:46.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:46.229+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:46.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:46.230+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:46.693 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:40:46.629+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 33072 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:40:46.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:46.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:46.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:46.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:46.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:46.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:46.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:46.466+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:47.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:46.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:47.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:46.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:47.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:46.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:47.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:46.978+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:47.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:47.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:47.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:47.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:47.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:47.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:47.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:47.210+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:47.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:47.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:47.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:47.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:47.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:47.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:47.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:47.427+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:48.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:47.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:48.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:47.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:48.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:47.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:48.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:47.943+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:48.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:48.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:48.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:48.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:48.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:48.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:48.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:48.415+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:48.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:48.249+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:48.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:48.249+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:48.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:48.249+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:48.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:48.250+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:49.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:48.962+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:49.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:48.962+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:49.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:48.962+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:49.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:48.963+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:49.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:49.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:49.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:49.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:49.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:49.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:49.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:49.414+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:49.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:49.298+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:49.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:49.298+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:49.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:49.298+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:49.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:49.298+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:50.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:49.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:50.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:49.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:50.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:49.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:50.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:49.944+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:50.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:50.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:50.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:50.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:50.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:50.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:50.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:50.388+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:50.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:50.251+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:50.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:50.251+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:50.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:50.251+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:50.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:50.252+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:51.380 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:51.231+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:51.381 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:51.231+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:51.381 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:51.231+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:51.381 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:51.232+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:51.381 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:50.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:51.381 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:50.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:51.381 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:50.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:51.381 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:50.993+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:51.689 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:40:51.633+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 33160 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:40:51.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:51.407+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:51.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:51.407+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:51.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:51.407+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:51.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:51.408+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:52.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:52.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:52.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:52.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:52.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:52.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:52.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:52.258+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:52.333 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:52.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:52.333 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:52.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:52.333 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:52.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:52.333 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:52.010+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:52.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:52.359+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:52.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:52.359+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:52.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:52.359+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:52.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:52.360+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:53.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:52.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:53.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:52.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:53.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:52.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:53.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:52.970+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:53.624 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:53.396+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:53.624 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:53.396+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:53.624 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:53.396+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:53.624 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:53.397+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:53.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:53.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:53.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:53.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:53.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:53.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:53.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:53.265+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:54.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:53.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:54.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:53.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:54.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:53.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:54.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:53.969+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:54.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:54.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:54.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:54.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:54.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:54.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:54.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:54.218+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:54.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:54.356+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:54.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:54.356+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:54.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:54.356+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:54.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:54.356+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:55.303 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:55.216+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:55.303 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:55.216+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:55.303 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:55.216+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:55.303 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:55.217+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:55.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:54.990+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:55.304 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:54.990+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:55.304 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:54.990+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:55.304 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:54.990+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:55.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:55.329+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:55.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:55.329+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:55.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:55.329+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:55.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:55.330+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:56.291 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:56.176+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:56.291 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:56.176+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:56.291 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:56.176+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:56.291 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:56.177+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:56.291 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:56.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:56.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:56.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:56.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:56.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:56.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:56.032+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:56.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:56.317+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:56.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:56.317+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:56.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:56.317+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:56.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:56.319+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:56.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:40:56.637+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 33251 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:40:57.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:57.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:57.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:57.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:57.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:57.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:57.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:57.208+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:57.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:57.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:57.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:57.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:57.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:57.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:57.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:57.011+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:57.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:57.344+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:57.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:57.344+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:57.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:57.344+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:57.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:57.345+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:58.270 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:58.168+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:58.271 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:58.168+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:58.271 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:58.168+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:58.271 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:58.169+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:58.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:58.019+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:58.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:58.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:58.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:58.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:58.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:58.020+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:58.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:58.297+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:58.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:58.297+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:58.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:58.297+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:58.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:58.297+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:59.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:59.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:59.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:59.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:59.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:59.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:40:59.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:40:59.277+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:59.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:59.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:59.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:59.034+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:59.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:59.034+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:40:59.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:40:59.034+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:40:59.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:59.203+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:59.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:59.203+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:59.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:59.203+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:40:59.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:40:59.204+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:00.273 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:00.187+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:00.273 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:00.187+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:00.273 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:00.187+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:00.273 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:00.188+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:00.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:00.039+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:00.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:00.039+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:00.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:00.039+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:00.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:00.040+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:00.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:00.299+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:00.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:00.299+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:00.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:00.299+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:00.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:00.300+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:01.303 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:01.207+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:01.303 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:01.207+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:01.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:01.207+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:01.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:01.208+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:01.304 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:01.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:01.304 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:01.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:01.304 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:01.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:01.304 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:01.041+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:01.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:01.330+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:01.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:01.330+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:01.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:01.330+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:01.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:01.330+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:01.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:41:01.641+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 33339 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:41:02.338 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:02.082+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:02.338 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:02.082+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:02.338 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:02.082+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:02.338 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:02.082+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:02.338 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:02.203+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:02.338 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:02.203+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:02.339 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:02.203+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:02.339 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:02.204+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:02.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:02.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:02.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:02.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:02.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:02.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:02.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:02.365+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:03.359 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:03.203+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:03.359 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:03.203+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:03.359 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:03.203+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:03.359 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:03.204+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:03.360 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:03.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:03.360 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:03.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:03.360 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:03.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:03.360 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:03.077+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:03.625 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:03.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:03.625 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:03.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:03.626 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:03.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:03.626 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:03.386+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:04.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:04.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:04.389 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:04.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:04.389 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:04.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:04.389 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:04.239+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:04.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:04.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:04.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:04.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:04.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:04.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:04.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:04.084+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:04.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:04.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:04.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:04.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:04.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:04.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:04.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:04.415+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:05.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:05.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:05.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:05.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:05.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:05.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:05.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:05.248+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:05.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:05.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:05.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:05.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:05.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:05.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:05.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:05.122+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:05.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:05.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:05.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:05.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:05.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:05.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:05.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:05.430+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:06.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:06.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:06.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:06.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:06.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:06.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:06.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:06.144+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:06.424 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:06.277+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:06.424 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:06.277+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:06.424 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:06.277+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:06.424 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:06.278+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:06.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:41:06.645+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 33429 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:41:06.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:06.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:06.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:06.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:06.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:06.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:06.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:06.450+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:07.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:07.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:07.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:07.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:07.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:07.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:07.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:07.180+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:07.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:07.259+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:07.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:07.259+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:07.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:07.259+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:07.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:07.260+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:07.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:07.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:07.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:07.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:07.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:07.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:07.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:07.486+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:08.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:08.240+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:08.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:08.240+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:08.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:08.240+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:08.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:08.241+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:08.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:08.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:08.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:08.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:08.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:08.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:08.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:08.175+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:08.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:08.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:08.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:08.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:08.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:08.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:08.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:08.473+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:09.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:09.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:09.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:09.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:09.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:09.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:09.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:09.180+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:09.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:09.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:09.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:09.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:09.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:09.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:09.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:09.262+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:09.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:09.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:09.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:09.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:09.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:09.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:09.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:09.494+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:10.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:10.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:10.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:10.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:10.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:10.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:10.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:10.160+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:10.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:10.257+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:10.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:10.257+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:10.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:10.257+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:10.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:10.258+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:10.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:10.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:10.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:10.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:10.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:10.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:10.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:10.465+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:11.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:11.296+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:11.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:11.296+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:11.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:11.296+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:11.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:11.297+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:11.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:11.131+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:11.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:11.131+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:11.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:11.131+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:11.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:11.132+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:11.695 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:41:11.650+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 33517 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:41:11.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:11.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:11.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:11.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:11.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:11.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:11.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:11.463+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:12.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:12.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:12.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:12.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:12.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:12.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:12.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:12.251+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:12.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:12.120+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:12.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:12.120+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:12.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:12.120+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:12.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:12.120+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:12.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:12.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:12.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:12.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:12.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:12.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:12.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:12.462+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:13.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:13.221+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:13.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:13.221+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:13.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:13.221+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:13.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:13.222+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:13.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:13.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:13.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:13.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:13.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:13.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:13.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:13.151+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:13.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:13.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:13.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:13.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:13.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:13.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:13.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:13.421+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:14.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:14.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:14.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:14.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:14.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:14.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:14.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:14.169+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:14.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:14.234+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:14.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:14.234+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:14.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:14.234+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:14.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:14.235+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:14.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:14.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:14.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:14.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:14.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:14.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:14.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:14.471+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:15.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:15.204+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:15.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:15.204+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:15.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:15.204+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:15.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:15.205+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:15.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:15.131+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:15.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:15.131+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:15.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:15.131+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:15.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:15.131+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:15.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:15.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:15.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:15.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:15.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:15.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:15.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:15.520+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:16.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:16.095+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:16.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:16.095+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:16.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:16.095+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:16.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:16.095+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:16.342 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:16.241+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:16.342 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:16.241+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:16.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:16.241+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:16.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:16.242+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:16.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:16.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:16.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:16.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:16.629 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:16.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:16.629 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:16.479+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:16.900 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:41:16.653+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 33607 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:41:17.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:17.089+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:17.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:17.089+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:17.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:17.089+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:17.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:17.089+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:17.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:17.228+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:17.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:17.228+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:17.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:17.228+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:17.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:17.229+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:17.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:17.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:17.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:17.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:17.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:17.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:17.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:17.475+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:18.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:18.225+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:18.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:18.225+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:18.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:18.225+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:18.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:18.225+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:18.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:18.116+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:18.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:18.116+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:18.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:18.116+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:18.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:18.116+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:18.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:18.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:18.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:18.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:18.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:18.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:18.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:18.523+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:19.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:19.178+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:19.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:19.178+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:19.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:19.178+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:19.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:19.179+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:19.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:19.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:19.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:19.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:19.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:19.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:19.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:19.091+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:19.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:19.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:19.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:19.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:19.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:19.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:19.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:19.573+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:20.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:20.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:20.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:20.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:20.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:20.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:20.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:20.227+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:20.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:20.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:20.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:20.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:20.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:20.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:20.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:20.105+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:20.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:20.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:20.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:20.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:20.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:20.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:20.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:20.591+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:21.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:21.265+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:21.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:21.265+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:21.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:21.265+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:21.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:21.266+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:21.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:21.129+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:21.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:21.129+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:21.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:21.129+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:21.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:21.129+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:21.944 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:41:21.657+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 33694 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:41:21.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:21.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:21.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:21.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:21.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:21.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:21.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:21.594+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:22.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:22.086+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:22.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:22.086+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:22.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:22.086+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:22.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:22.086+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:22.588 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:22.245+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:22.588 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:22.245+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:22.588 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:22.245+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:22.588 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:22.246+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:22.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:22.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:22.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:22.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:22.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:22.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:22.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:22.613+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:23.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:23.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:23.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:23.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:23.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:23.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:23.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:23.294+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:23.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:23.125+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:23.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:23.125+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:23.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:23.125+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:23.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:23.125+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:23.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:23.603+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:23.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:23.603+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:23.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:23.603+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:23.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:23.604+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:24.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:24.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:24.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:24.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:24.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:24.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:24.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:24.274+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:24.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:24.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:24.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:24.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:24.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:24.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:24.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:24.086+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:24.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:24.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:24.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:24.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:24.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:24.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:24.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:24.579+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:25.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:25.102+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:25.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:25.102+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:25.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:25.102+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:25.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:25.102+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:25.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:25.242+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:25.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:25.242+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:25.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:25.242+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:25.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:25.243+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:25.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:25.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:25.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:25.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:25.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:25.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:25.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:25.626+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:26.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:26.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:26.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:26.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:26.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:26.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:26.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:26.134+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:26.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:26.227+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:26.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:26.227+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:26.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:26.227+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:26.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:26.227+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:26.762 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:41:26.661+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 33784 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:41:26.762 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:26.630+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:26.762 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:26.630+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:26.762 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:26.630+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:26.762 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:26.631+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:27.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:27.185+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:27.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:27.185+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:27.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:27.185+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:27.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:27.185+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:27.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:27.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:27.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:27.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:27.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:27.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:27.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:27.135+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:27.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:27.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:27.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:27.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:27.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:27.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:27.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:27.630+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:28.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:28.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:28.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:28.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:28.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:28.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:28.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:28.171+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:28.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:28.166+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:28.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:28.166+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:28.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:28.166+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:28.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:28.166+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:28.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:28.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:28.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:28.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:28.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:28.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:28.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:28.628+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:29.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:29.192+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:29.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:29.192+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:29.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:29.192+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:29.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:29.192+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:29.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:29.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:29.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:29.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:29.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:29.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:29.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:29.156+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:29.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:29.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:29.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:29.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:29.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:29.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:29.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:29.648+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:30.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:30.204+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:30.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:30.204+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:30.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:30.204+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:30.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:30.204+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:30.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:30.123+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:30.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:30.123+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:30.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:30.123+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:30.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:30.124+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:30.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:30.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:30.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:30.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:30.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:30.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:30.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:30.620+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:31.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:31.163+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:31.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:31.163+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:31.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:31.163+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:31.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:31.164+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:31.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:31.163+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:31.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:31.163+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:31.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:31.163+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:31.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:31.163+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:31.945 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:41:31.666+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 33872 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:41:31.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:31.599+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:31.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:31.599+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:31.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:31.599+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:31.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:31.600+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:32.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:32.152+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:32.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:32.152+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:32.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:32.152+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:32.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:32.153+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:32.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:32.168+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:32.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:32.168+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:32.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:32.168+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:32.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:32.169+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:32.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:32.560+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:32.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:32.560+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:32.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:32.560+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:32.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:32.561+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:33.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:33.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:33.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:33.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:33.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:33.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:33.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:33.165+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:33.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:33.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:33.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:33.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:33.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:33.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:33.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:33.146+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:33.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:33.570+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:33.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:33.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:33.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:33.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:33.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:33.571+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:34.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:34.172+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:34.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:34.172+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:34.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:34.172+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:34.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:34.172+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:34.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:34.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:34.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:34.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:34.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:34.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:34.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:34.162+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:34.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:34.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:34.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:34.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:34.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:34.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:34.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:34.605+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:35.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:35.152+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:35.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:35.152+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:35.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:35.152+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:35.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:35.153+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:35.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:35.154+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:35.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:35.154+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:35.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:35.154+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:35.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:35.154+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:35.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:35.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:35.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:35.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:35.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:35.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:35.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:35.613+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:36.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:36.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:36.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:36.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:36.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:36.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:36.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:36.159+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:36.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:36.109+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:36.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:36.109+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:36.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:36.109+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:36.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:36.110+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:36.945 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:41:36.670+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 33963 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:41:36.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:36.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:36.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:36.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:36.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:36.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:36.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:36.647+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:37.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:37.107+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:37.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:37.107+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:37.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:37.107+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:37.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:37.108+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:37.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:37.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:37.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:37.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:37.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:37.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:37.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:37.188+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:37.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:37.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:37.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:37.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:37.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:37.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:37.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:37.675+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:38.406 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:38.121+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:38.406 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:38.121+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:38.406 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:38.121+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:38.406 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:38.122+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:38.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:38.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:38.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:38.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:38.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:38.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:38.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:38.150+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:38.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:38.635+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:38.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:38.635+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:38.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:38.635+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:38.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:38.636+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:39.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:39.153+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:39.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:39.153+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:39.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:39.153+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:39.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:39.153+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:39.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:39.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:39.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:39.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:39.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:39.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:39.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:39.158+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:39.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:39.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:39.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:39.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:39.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:39.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:39.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:39.682+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:40.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:40.123+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:40.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:40.123+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:40.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:40.123+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:40.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:40.123+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:40.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:40.172+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:40.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:40.172+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:40.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:40.172+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:40.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:40.172+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:40.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:40.653+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:40.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:40.653+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:40.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:40.653+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:40.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:40.654+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:41.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:41.145+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:41.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:41.145+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:41.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:41.145+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:41.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:41.145+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:41.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:41.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:41.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:41.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:41.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:41.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:41.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:41.203+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:41.946 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:41:41.674+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 34051 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:41:41.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:41.623+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:41.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:41.623+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:41.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:41.623+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:41.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:41.624+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:42.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:42.123+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:42.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:42.123+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:42.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:42.123+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:42.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:42.124+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:42.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:42.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:42.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:42.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:42.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:42.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:42.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:42.233+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:42.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:42.599+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:42.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:42.599+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:42.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:42.599+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:42.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:42.599+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:43.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:43.141+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:43.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:43.141+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:43.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:43.141+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:43.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:43.142+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:43.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:43.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:43.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:43.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:43.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:43.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:43.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:43.229+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:43.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:43.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:43.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:43.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:43.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:43.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:43.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:43.598+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:44.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:44.185+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:44.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:44.185+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:44.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:44.185+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:44.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:44.186+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:44.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:44.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:44.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:44.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:44.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:44.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:44.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:44.183+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:44.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:44.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:44.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:44.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:44.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:44.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:44.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:44.561+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:45.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:45.147+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:45.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:45.147+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:45.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:45.147+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:45.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:45.147+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:45.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:45.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:45.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:45.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:45.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:45.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:45.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:45.144+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:45.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:45.543+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:45.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:45.543+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:45.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:45.543+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:45.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:45.544+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:46.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:46.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:46.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:46.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:46.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:46.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:46.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:46.130+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:46.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:46.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:46.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:46.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:46.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:46.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:46.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:46.159+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:46.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:41:46.628+0000 7f770c428700 0 monclient(hunting): authenticate timed out after 300 2024-04-27T02:41:46.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:41:46.678+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 34141 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:41:46.779 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:46.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:46.779 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:46.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:46.779 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:46.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:46.779 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:46.521+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:47.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:47.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:47.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:47.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:47.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:47.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:47.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:47.119+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:47.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:47.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:47.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:47.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:47.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:47.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:47.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:47.171+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:47.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:47.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:47.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:47.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:47.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:47.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:47.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:47.492+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:48.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:48.140+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:48.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:48.140+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:48.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:48.140+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:48.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:48.140+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:48.431 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:48.152+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:48.431 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:48.152+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:48.431 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:48.152+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:48.431 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:48.153+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:48.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:48.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:48.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:48.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:48.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:48.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:48.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:48.454+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:49.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:49.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:49.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:49.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:49.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:49.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:49.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:49.180+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:49.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:49.178+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:49.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:49.178+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:49.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:49.178+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:49.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:49.179+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:49.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:49.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:49.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:49.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:49.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:49.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:49.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:49.451+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:50.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:50.164+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:50.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:50.164+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:50.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:50.164+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:50.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:50.166+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:50.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:50.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:50.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:50.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:50.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:50.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:50.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:50.143+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:50.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:50.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:50.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:50.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:50.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:50.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:50.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:50.484+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:51.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:51.141+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:51.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:51.141+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:51.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:51.141+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:51.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:51.142+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:51.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:51.166+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:51.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:51.166+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:51.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:51.166+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:51.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:51.167+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:51.680 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:41:51.683+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 34230 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:41:51.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:51.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:51.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:51.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:51.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:51.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:51.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:51.450+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:52.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:52.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:52.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:52.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:52.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:52.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:52.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:52.209+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:52.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:52.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:52.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:52.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:52.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:52.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:52.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:52.179+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:52.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:52.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:52.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:52.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:52.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:52.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:52.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:52.440+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:53.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:53.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:53.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:53.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:53.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:53.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:53.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:53.446+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:53.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:53.217+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:53.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:53.217+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:53.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:53.217+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:53.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:53.218+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:53.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:53.253+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:53.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:53.253+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:53.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:53.253+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:53.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:53.254+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:54.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:54.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:54.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:54.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:54.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:54.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:54.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:54.478+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:54.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:54.300+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:54.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:54.300+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:54.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:54.300+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:54.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:54.301+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:54.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:54.263+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:54.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:54.263+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:54.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:54.263+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:54.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:54.264+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:55.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:55.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:55.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:55.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:55.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:55.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:55.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:55.480+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:55.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:55.251+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:55.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:55.251+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:55.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:55.251+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:55.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:55.252+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:55.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:55.241+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:55.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:55.241+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:55.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:55.241+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:55.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:55.242+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:56.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:56.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:56.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:56.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:56.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:56.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:56.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:56.239+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:56.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:56.263+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:56.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:56.263+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:56.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:56.263+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:56.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:56.264+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:56.788 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:41:56.687+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 34322 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:41:56.789 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:56.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:56.789 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:56.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:56.789 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:56.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:56.789 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:56.497+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:57.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:57.291+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:57.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:57.291+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:57.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:57.291+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:57.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:57.292+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:57.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:57.217+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:57.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:57.217+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:57.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:57.217+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:57.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:57.218+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:57.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:57.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:57.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:57.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:57.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:57.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:57.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:57.506+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:58.519 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:58.263+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:58.519 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:58.263+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:58.519 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:58.263+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:58.520 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:58.263+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:58.520 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:58.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:58.520 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:58.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:58.520 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:58.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:58.520 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:58.313+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:58.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:58.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:58.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:58.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:58.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:58.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:58.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:58.543+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:59.517 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:59.281+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:59.517 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:59.281+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:59.517 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:59.281+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:41:59.518 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:41:59.282+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:59.518 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:59.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:59.518 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:59.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:59.518 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:59.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:41:59.518 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:41:59.278+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:41:59.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:59.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:59.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:59.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:59.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:59.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:41:59.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:41:59.541+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:00.521 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:00.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:00.521 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:00.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:00.522 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:00.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:00.522 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:00.258+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:00.522 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:00.301+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:00.522 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:00.301+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:00.522 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:00.301+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:00.522 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:00.302+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:00.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:00.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:00.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:00.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:00.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:00.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:00.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:00.545+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:01.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:01.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:01.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:01.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:01.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:01.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:01.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:01.514+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:01.669 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:01.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:01.669 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:01.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:01.669 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:01.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:01.669 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:01.337+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:01.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:01.306+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:01.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:01.306+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:01.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:01.306+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:01.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:01.307+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:01.947 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:42:01.691+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 34411 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:42:02.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:02.287+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:02.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:02.287+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:02.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:02.287+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:02.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:02.287+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:02.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:02.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:02.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:02.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:02.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:02.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:02.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:02.465+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:02.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:02.304+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:02.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:02.304+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:02.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:02.304+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:02.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:02.305+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:03.631 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:03.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:03.632 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:03.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:03.632 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:03.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:03.632 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:03.310+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:03.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:03.438+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:03.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:03.438+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:03.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:03.438+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:03.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:03.439+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:03.633 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:03.322+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:03.633 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:03.322+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:03.633 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:03.322+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:03.633 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:03.323+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:04.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:04.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:04.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:04.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:04.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:04.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:04.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:04.345+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:04.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:04.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:04.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:04.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:04.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:04.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:04.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:04.411+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:04.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:04.301+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:04.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:04.301+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:04.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:04.301+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:04.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:04.302+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:05.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:05.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:05.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:05.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:05.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:05.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:05.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:05.345+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:05.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:05.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:05.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:05.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:05.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:05.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:05.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:05.445+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:05.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:05.315+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:05.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:05.315+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:05.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:05.315+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:05.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:05.316+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:06.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:06.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:06.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:06.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:06.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:06.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:06.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:06.401+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:06.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:06.349+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:06.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:06.349+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:06.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:06.349+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:06.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:06.350+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:06.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:06.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:06.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:06.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:06.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:06.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:06.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:06.339+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:06.947 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:42:06.695+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 34502 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:42:07.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:07.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:07.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:07.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:07.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:07.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:07.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:07.332+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:07.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:07.343+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:07.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:07.343+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:07.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:07.343+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:07.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:07.344+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:07.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:07.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:07.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:07.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:07.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:07.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:07.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:07.430+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:08.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:08.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:08.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:08.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:08.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:08.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:08.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:08.325+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:08.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:08.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:08.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:08.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:08.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:08.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:08.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:08.421+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:08.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:08.372+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:08.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:08.372+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:08.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:08.372+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:08.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:08.372+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:09.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:09.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:09.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:09.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:09.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:09.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:09.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:09.308+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:09.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:09.387+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:09.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:09.387+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:09.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:09.387+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:09.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:09.387+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:09.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:09.380+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:09.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:09.380+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:09.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:09.380+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:09.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:09.381+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:10.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:10.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:10.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:10.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:10.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:10.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:10.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:10.276+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:10.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:10.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:10.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:10.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:10.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:10.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:10.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:10.426+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:10.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:10.394+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:10.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:10.394+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:10.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:10.394+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:10.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:10.395+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:11.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:11.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:11.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:11.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:11.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:11.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:11.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:11.460+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:11.678 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:11.439+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:11.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:11.439+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:11.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:11.439+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:11.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:11.440+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:11.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:11.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:11.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:11.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:11.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:11.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:11.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:11.240+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:11.948 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:42:11.700+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 34591 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:42:12.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:12.404+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:12.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:12.404+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:12.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:12.404+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:12.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:12.405+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:12.481 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:12.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:12.481 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:12.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:12.481 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:12.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:12.481 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:12.282+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:12.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:12.502+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:12.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:12.502+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:12.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:12.502+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:12.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:12.503+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:13.526 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:13.283+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:13.527 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:13.283+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:13.527 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:13.283+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:13.527 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:13.284+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:13.527 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:13.371+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:13.527 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:13.371+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:13.527 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:13.371+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:13.527 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:13.372+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:13.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:13.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:13.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:13.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:13.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:13.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:13.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:13.549+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:14.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:14.354+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:14.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:14.354+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:14.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:14.354+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:14.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:14.355+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:14.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:14.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:14.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:14.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:14.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:14.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:14.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:14.514+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:14.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:14.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:14.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:14.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:14.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:14.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:14.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:14.265+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:15.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:15.327+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:15.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:15.327+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:15.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:15.327+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:15.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:15.328+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:15.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:15.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:15.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:15.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:15.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:15.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:15.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:15.531+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:15.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:15.298+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:15.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:15.298+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:15.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:15.298+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:15.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:15.298+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:16.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:16.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:16.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:16.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:16.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:16.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:16.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:16.518+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:16.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:16.345+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:16.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:16.345+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:16.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:16.345+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:16.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:16.346+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:16.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:16.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:16.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:16.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:16.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:16.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:16.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:16.345+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:16.947 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:42:16.704+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 34683 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:42:17.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:17.389+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:17.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:17.389+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:17.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:17.389+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:17.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:17.389+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:17.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:17.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:17.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:17.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:17.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:17.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:17.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:17.389+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:17.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:17.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:17.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:17.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:17.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:17.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:17.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:17.520+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:18.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:18.404+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:18.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:18.404+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:18.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:18.404+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:18.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:18.405+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:18.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:18.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:18.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:18.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:18.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:18.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:18.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:18.422+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:18.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:18.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:18.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:18.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:18.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:18.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:18.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:18.510+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:19.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:19.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:19.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:19.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:19.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:19.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:19.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:19.550+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:19.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:19.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:19.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:19.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:19.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:19.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:19.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:19.389+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:19.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:19.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:19.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:19.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:19.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:19.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:19.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:19.394+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:20.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:20.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:20.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:20.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:20.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:20.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:20.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:20.530+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:20.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:20.386+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:20.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:20.386+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:20.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:20.386+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:20.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:20.387+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:20.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:20.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:20.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:20.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:20.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:20.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:20.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:20.370+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:21.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:21.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:21.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:21.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:21.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:21.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:21.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:21.513+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:21.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:21.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:21.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:21.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:21.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:21.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:21.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:21.365+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:21.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:21.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:21.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:21.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:21.688 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:21.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:21.688 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:21.340+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:21.948 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:42:21.707+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 34772 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:42:22.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:22.402+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:22.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:22.402+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:22.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:22.402+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:22.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:22.402+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:22.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:22.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:22.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:22.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:22.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:22.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:22.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:22.376+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:22.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:22.522+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:22.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:22.522+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:22.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:22.522+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:22.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:22.522+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:23.633 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:23.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:23.634 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:23.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:23.634 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:23.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:23.634 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:23.510+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:23.634 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:23.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:23.634 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:23.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:23.634 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:23.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:23.634 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:23.450+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:23.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:23.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:23.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:23.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:23.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:23.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:23.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:23.388+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:24.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:24.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:24.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:24.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:24.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:24.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:24.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:24.526+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:24.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:24.459+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:24.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:24.459+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:24.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:24.459+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:24.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:24.460+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:24.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:24.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:24.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:24.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:24.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:24.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:24.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:24.339+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:25.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:25.468+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:25.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:25.468+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:25.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:25.468+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:25.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:25.469+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:25.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:25.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:25.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:25.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:25.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:25.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:25.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:25.364+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:25.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:25.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:25.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:25.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:25.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:25.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:25.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:25.576+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:26.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:26.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:26.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:26.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:26.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:26.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:26.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:26.533+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:26.600 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:26.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:26.600 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:26.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:26.600 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:26.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:26.600 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:26.503+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:26.601 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:26.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:26.601 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:26.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:26.601 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:26.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:26.601 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:26.339+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:26.948 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:42:26.712+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 34863 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:42:27.549 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:27.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:27.550 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:27.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:27.550 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:27.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:27.550 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:27.296+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:27.550 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:27.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:27.550 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:27.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:27.550 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:27.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:27.550 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:27.537+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:27.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:27.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:27.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:27.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:27.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:27.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:27.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:27.571+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:28.542 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:28.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:28.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:28.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:28.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:28.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:28.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:28.248+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:28.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:28.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:28.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:28.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:28.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:28.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:28.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:28.563+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:28.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:28.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:28.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:28.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:28.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:28.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:28.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:28.587+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:29.535 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:29.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:29.535 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:29.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:29.535 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:29.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:29.535 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:29.526+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:29.536 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:29.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:29.536 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:29.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:29.536 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:29.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:29.536 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:29.284+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:29.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:29.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:29.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:29.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:29.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:29.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:29.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:29.557+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:30.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:30.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:30.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:30.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:30.633 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:30.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:30.633 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:30.531+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:30.633 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:30.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:30.633 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:30.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:30.633 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:30.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:30.633 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:30.522+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:30.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:30.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:30.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:30.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:30.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:30.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:30.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:30.328+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:31.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:31.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:31.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:31.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:31.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:31.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:31.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:31.555+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:31.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:31.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:31.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:31.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:31.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:31.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:31.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:31.374+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:31.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:31.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:31.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:31.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:31.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:31.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:31.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:31.548+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:31.948 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:42:31.717+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 34952 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:42:32.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:32.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:32.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:32.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:32.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:32.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:32.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:32.519+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:32.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:32.553+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:32.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:32.553+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:32.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:32.553+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:32.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:32.554+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:32.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:32.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:32.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:32.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:32.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:32.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:32.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:32.336+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:33.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:33.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:33.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:33.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:33.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:33.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:33.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:33.308+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:33.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:33.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:33.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:33.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:33.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:33.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:33.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:33.525+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:33.636 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:33.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:33.636 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:33.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:33.636 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:33.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:33.636 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:33.523+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:34.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:34.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:34.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:34.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:34.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:34.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:34.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:34.483+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:34.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:34.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:34.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:34.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:34.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:34.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:34.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:34.546+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:34.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:34.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:34.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:34.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:34.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:34.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:34.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:34.330+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:35.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:35.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:35.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:35.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:35.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:35.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:35.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:35.508+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:35.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:35.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:35.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:35.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:35.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:35.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:35.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:35.573+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:35.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:35.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:35.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:35.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:35.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:35.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:35.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:35.379+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:36.601 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:36.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:36.602 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:36.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:36.602 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:36.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:36.602 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:36.409+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:36.602 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:36.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:36.602 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:36.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:36.602 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:36.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:36.602 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:36.541+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:36.949 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:42:36.721+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 35044 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:42:36.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:36.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:36.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:36.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:36.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:36.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:36.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:36.623+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:37.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:37.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:37.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:37.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:37.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:37.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:37.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:37.388+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:37.627 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:37.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:37.627 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:37.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:37.627 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:37.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:37.627 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:37.515+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:37.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:37.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:37.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:37.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:37.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:37.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:37.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:37.648+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:38.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:38.612+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:38.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:38.612+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:38.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:38.612+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:38.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:38.612+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:38.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:38.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:38.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:38.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:38.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:38.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:38.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:38.410+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:38.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:38.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:38.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:38.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:38.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:38.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:38.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:38.539+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:39.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:39.656+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:39.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:39.656+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:39.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:39.656+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:39.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:39.656+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:39.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:39.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:39.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:39.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:39.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:39.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:39.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:39.414+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:39.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:39.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:39.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:39.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:39.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:39.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:39.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:39.568+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:40.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:40.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:40.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:40.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:40.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:40.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:40.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:40.416+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:40.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:40.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:40.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:40.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:40.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:40.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:40.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:40.525+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:40.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:40.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:40.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:40.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:40.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:40.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:40.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:40.680+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:41.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:41.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:41.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:41.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:41.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:41.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:41.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:41.384+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:41.646 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:41.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:41.646 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:41.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:41.646 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:41.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:41.646 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:41.488+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:41.949 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:42:41.726+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 35133 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:42:41.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:41.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:41.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:41.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:41.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:41.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:41.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:41.666+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:42.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:42.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:42.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:42.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:42.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:42.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:42.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:42.526+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:42.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:42.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:42.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:42.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:42.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:42.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:42.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:42.397+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:42.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:42.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:42.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:42.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:42.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:42.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:42.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:42.717+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:43.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:43.510+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:43.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:43.510+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:43.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:43.510+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:43.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:43.511+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:43.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:43.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:43.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:43.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:43.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:43.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:43.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:43.433+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:44.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:43.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:44.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:43.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:44.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:43.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:44.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:43.730+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:44.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:44.532+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:44.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:44.532+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:44.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:44.532+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:44.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:44.532+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:44.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:44.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:44.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:44.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:44.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:44.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:44.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:44.399+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:44.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:44.704+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:44.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:44.704+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:44.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:44.704+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:44.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:44.705+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:45.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:45.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:45.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:45.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:45.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:45.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:45.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:45.575+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:45.665 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:45.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:45.665 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:45.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:45.665 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:45.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:45.665 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:45.390+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:45.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:45.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:45.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:45.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:45.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:45.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:45.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:45.686+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:46.629 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:46.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:46.629 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:46.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:46.629 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:46.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:46.630 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:46.413+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:46.630 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:46.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:46.630 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:46.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:46.630 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:46.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:46.630 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:46.604+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:46.949 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:42:46.730+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 35223 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:42:46.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:46.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:46.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:46.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:46.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:46.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:46.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:46.650+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:47.638 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:47.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:47.638 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:47.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:47.638 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:47.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:47.639 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:47.631+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:47.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:47.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:47.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:47.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:47.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:47.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:47.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:47.599+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:47.639 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:47.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:47.639 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:47.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:47.640 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:47.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:47.640 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:47.384+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:48.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:48.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:48.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:48.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:48.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:48.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:48.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:48.386+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:48.618 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:48.635+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:48.618 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:48.635+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:48.619 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:48.635+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:48.619 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:48.636+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:48.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:48.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:48.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:48.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:48.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:48.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:48.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:48.639+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:49.666 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:49.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:49.666 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:49.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:49.666 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:49.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:49.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:49.616+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:49.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:49.421+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:49.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:49.421+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:49.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:49.421+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:49.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:49.421+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:49.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:49.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:49.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:49.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:49.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:49.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:49.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:49.687+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:50.664 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:50.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:50.664 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:50.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:50.664 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:50.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:50.664 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:50.415+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:50.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:50.587+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:50.665 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:50.587+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:50.665 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:50.587+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:50.665 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:50.589+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:50.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:50.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:50.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:50.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:50.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:50.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:50.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:50.685+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:51.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:51.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:51.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:51.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:51.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:51.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:51.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:51.549+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:51.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:51.653+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:51.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:51.653+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:51.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:51.653+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:51.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:51.654+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:51.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:51.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:51.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:51.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:51.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:51.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:51.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:51.444+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:52.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:42:51.734+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 35312 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:42:52.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:52.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:52.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:52.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:52.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:52.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:52.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:52.567+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:52.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:52.489+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:52.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:52.489+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:52.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:52.489+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:52.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:52.490+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:52.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:52.680+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:52.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:52.680+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:52.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:52.680+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:52.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:52.681+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:53.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:53.613+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:53.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:53.613+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:53.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:53.613+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:53.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:53.614+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:53.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:53.715+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:53.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:53.715+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:53.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:53.715+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:53.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:53.717+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:53.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:53.527+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:53.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:53.527+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:53.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:53.527+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:53.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:53.527+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:54.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:54.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:54.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:54.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:54.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:54.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:54.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:54.608+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:54.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:54.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:54.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:54.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:54.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:54.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:54.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:54.682+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:54.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:54.559+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:54.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:54.559+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:54.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:54.559+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:54.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:54.560+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:55.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:55.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:55.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:55.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:55.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:55.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:55.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:55.625+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:55.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:55.650+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:55.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:55.650+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:55.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:55.650+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:55.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:55.651+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:55.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:55.595+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:55.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:55.595+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:55.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:55.595+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:55.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:55.595+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:56.644 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:56.578+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:56.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:56.578+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:56.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:56.578+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:56.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:56.579+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:56.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:42:56.739+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 35404 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:42:56.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:56.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:56.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:56.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:56.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:56.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:56.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:56.675+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:56.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:56.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:56.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:56.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:56.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:56.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:56.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:56.665+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:57.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:57.700+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:57.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:57.700+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:57.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:57.700+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:57.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:57.701+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:57.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:57.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:57.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:57.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:57.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:57.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:57.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:57.664+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:57.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:57.604+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:57.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:57.604+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:57.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:57.604+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:57.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:57.604+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:58.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:58.663+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:58.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:58.663+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:58.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:58.663+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:58.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:58.665+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:58.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:58.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:58.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:58.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:58.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:58.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:58.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:58.645+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:58.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:58.565+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:58.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:58.565+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:58.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:58.565+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:58.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:58.565+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:59.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:59.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:59.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:59.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:59.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:59.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:42:59.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:42:59.667+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:59.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:59.524+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:59.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:59.524+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:59.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:59.524+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:42:59.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:42:59.525+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:42:59.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:59.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:59.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:59.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:59.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:59.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:42:59.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:42:59.648+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:00.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:00.484+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:00.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:00.484+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:00.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:00.484+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:00.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:00.484+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:00.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:00.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:00.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:00.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:00.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:00.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:00.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:00.684+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:00.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:00.670+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:00.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:00.670+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:00.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:00.670+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:00.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:00.671+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:01.744 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:43:01.743+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 35493 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:43:01.745 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:01.717+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:01.745 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:01.717+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:01.745 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:01.717+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:01.745 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:01.718+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:01.745 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:01.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:01.745 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:01.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:01.745 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:01.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:01.746 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:01.659+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:01.746 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:01.510+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:01.746 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:01.510+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:01.746 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:01.510+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:01.746 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:01.510+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:02.755 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:02.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:02.756 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:02.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:02.756 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:02.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:02.756 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:02.517+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:02.756 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:02.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:02.756 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:02.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:02.756 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:02.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:02.756 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:02.701+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:02.756 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:02.708+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:02.757 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:02.708+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:02.757 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:02.708+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:02.757 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:02.708+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:03.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:03.680+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:03.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:03.680+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:03.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:03.680+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:03.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:03.681+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:03.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:03.530+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:03.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:03.530+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:03.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:03.530+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:03.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:03.531+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:03.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:03.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:03.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:03.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:03.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:03.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:03.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:03.670+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:04.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:04.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:04.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:04.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:04.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:04.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:04.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:04.655+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:04.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:04.519+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:04.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:04.519+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:04.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:04.519+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:04.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:04.520+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:04.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:04.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:04.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:04.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:04.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:04.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:04.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:04.706+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:05.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:05.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:05.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:05.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:05.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:05.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:05.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:05.516+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:05.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:05.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:05.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:05.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:05.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:05.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:05.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:05.687+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:05.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:05.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:05.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:05.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:05.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:05.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:05.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:05.630+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:06.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:06.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:06.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:06.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:06.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:06.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:06.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:06.470+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:06.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:06.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:06.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:06.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:06.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:06.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:06.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:06.606+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:07.200 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:43:06.747+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 35584 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:43:07.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:06.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:07.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:06.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:07.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:06.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:07.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:06.729+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:07.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:07.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:07.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:07.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:07.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:07.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:07.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:07.441+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:07.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:07.645+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:07.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:07.645+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:07.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:07.645+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:07.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:07.646+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:08.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:07.778+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:08.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:07.778+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:08.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:07.778+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:08.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:07.779+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:08.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:08.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:08.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:08.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:08.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:08.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:08.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:08.445+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:08.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:08.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:08.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:08.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:08.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:08.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:08.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:08.644+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:09.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:08.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:09.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:08.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:09.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:08.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:09.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:08.735+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:09.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:09.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:09.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:09.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:09.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:09.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:09.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:09.441+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:09.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:09.680+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:09.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:09.680+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:09.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:09.680+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:09.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:09.681+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:10.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:09.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:10.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:09.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:10.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:09.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:10.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:09.739+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:10.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:10.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:10.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:10.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:10.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:10.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:10.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:10.405+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:10.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:10.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:10.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:10.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:10.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:10.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:10.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:10.687+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:10.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:10.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:10.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:10.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:10.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:10.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:10.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:10.752+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:11.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:11.400+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:11.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:11.400+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:11.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:11.400+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:11.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:11.401+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:11.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:43:11.752+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 35673 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:43:11.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:11.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:11.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:11.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:11.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:11.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:11.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:11.683+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:11.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:11.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:11.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:11.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:11.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:11.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:11.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:11.772+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:12.624 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:12.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:12.624 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:12.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:12.624 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:12.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:12.624 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:12.387+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:12.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:12.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:12.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:12.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:12.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:12.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:12.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:12.644+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:12.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:12.789+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:12.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:12.789+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:12.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:12.789+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:12.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:12.790+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:13.641 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:13.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:13.641 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:13.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:13.641 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:13.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:13.641 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:13.599+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:13.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:13.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:13.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:13.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:13.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:13.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:13.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:13.364+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:13.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:13.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:13.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:13.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:13.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:13.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:13.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:13.760+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:14.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:14.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:14.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:14.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:14.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:14.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:14.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:14.363+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:14.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:14.630+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:14.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:14.630+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:14.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:14.630+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:14.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:14.632+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:14.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:14.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:14.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:14.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:14.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:14.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:14.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:14.792+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:15.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:15.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:15.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:15.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:15.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:15.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:15.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:15.594+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:15.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:15.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:15.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:15.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:15.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:15.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:15.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:15.391+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:16.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:15.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:16.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:15.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:16.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:15.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:16.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:15.799+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:16.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:16.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:16.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:16.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:16.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:16.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:16.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:16.419+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:16.863 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:43:16.756+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 35765 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:43:16.863 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:16.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:16.863 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:16.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:16.863 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:16.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:16.863 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:16.626+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:16.864 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:16.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:16.864 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:16.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:16.864 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:16.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:16.864 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:16.755+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:17.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:17.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:17.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:17.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:17.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:17.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:17.648 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:17.383+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:17.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:17.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:17.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:17.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:17.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:17.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:17.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:17.773+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:17.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:17.665+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:17.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:17.665+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:17.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:17.665+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:17.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:17.666+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:18.648 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:18.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:18.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:18.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:18.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:18.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:18.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:18.379+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:18.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:18.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:18.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:18.767+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:18.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:18.767+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:18.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:18.767+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:18.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:18.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:18.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:18.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:18.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:18.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:18.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:18.667+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:19.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:19.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:19.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:19.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:19.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:19.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:19.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:19.407+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:19.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:19.767+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:19.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:19.767+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:19.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:19.767+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:19.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:19.767+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:19.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:19.678+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:19.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:19.678+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:19.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:19.678+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:19.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:19.679+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:20.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:20.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:20.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:20.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:20.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:20.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:20.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:20.424+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:20.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:20.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:20.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:20.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:20.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:20.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:20.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:20.674+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:21.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:20.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:21.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:20.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:21.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:20.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:21.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:20.774+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:21.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:21.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:21.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:21.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:21.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:21.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:21.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:21.403+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:21.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:21.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:21.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:21.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:21.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:21.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:21.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:21.634+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:22.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:43:21.760+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 35854 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:43:22.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:21.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:22.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:21.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:22.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:21.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:22.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:21.773+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:22.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:22.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:22.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:22.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:22.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:22.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:22.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:22.392+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:22.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:22.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:22.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:22.626+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:22.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:22.626+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:22.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:22.627+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:23.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:22.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:23.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:22.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:23.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:22.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:23.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:22.821+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:23.641 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:23.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:23.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:23.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:23.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:23.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:23.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:23.587+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:23.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:23.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:23.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:23.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:23.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:23.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:23.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:23.354+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:23.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:23.819+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:23.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:23.819+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:23.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:23.819+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:23.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:23.820+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:24.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:24.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:24.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:24.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:24.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:24.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:24.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:24.336+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:24.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:24.828+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:24.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:24.828+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:24.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:24.828+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:24.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:24.829+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:24.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:24.635+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:24.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:24.635+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:24.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:24.635+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:24.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:24.636+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:25.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:25.362+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:25.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:25.362+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:25.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:25.362+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:25.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:25.362+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:25.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:25.637+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:25.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:25.637+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:25.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:25.637+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:25.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:25.637+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:25.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:25.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:25.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:25.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:25.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:25.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:25.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:25.796+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:26.625 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:26.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:26.625 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:26.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:26.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:26.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:26.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:26.393+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:26.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:43:26.765+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 35945 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:43:26.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:26.774+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:26.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:26.774+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:26.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:26.774+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:26.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:26.775+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:26.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:26.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:26.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:26.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:26.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:26.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:26.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:26.644+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:27.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:27.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:27.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:27.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:27.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:27.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:27.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:27.415+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:27.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:27.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:27.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:27.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:27.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:27.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:27.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:27.609+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:28.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:27.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:28.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:27.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:28.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:27.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:28.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:27.794+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:28.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:28.615+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:28.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:28.615+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:28.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:28.615+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:28.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:28.616+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:28.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:28.398+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:28.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:28.398+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:28.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:28.398+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:28.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:28.398+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:29.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:28.803+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:29.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:28.803+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:29.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:28.803+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:29.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:28.804+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:29.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:29.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:29.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:29.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:29.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:29.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:29.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:29.582+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:29.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:29.357+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:29.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:29.357+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:29.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:29.357+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:29.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:29.358+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:30.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:29.836+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:30.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:29.836+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:30.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:29.836+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:30.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:29.837+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:30.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:30.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:30.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:30.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:30.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:30.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:30.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:30.555+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:30.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:30.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:30.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:30.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:30.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:30.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:30.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:30.383+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:31.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:30.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:31.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:30.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:31.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:30.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:31.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:30.886+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:31.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:31.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:31.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:31.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:31.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:31.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:31.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:31.585+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:31.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:31.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:31.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:31.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:31.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:31.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:31.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:31.388+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:32.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:43:31.771+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 36034 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:43:32.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:31.847+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:32.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:31.848+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:32.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:31.848+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:32.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:31.848+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:32.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:32.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:32.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:32.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:32.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:32.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:32.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:32.606+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:32.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:32.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:32.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:32.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:32.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:32.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:32.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:32.396+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:33.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:32.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:33.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:32.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:33.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:32.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:33.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:32.812+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:33.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:33.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:33.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:33.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:33.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:33.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:33.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:33.577+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:33.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:33.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:33.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:33.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:33.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:33.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:33.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:33.398+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:33.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:33.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:33.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:33.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:33.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:33.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:33.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:33.771+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:34.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:34.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:34.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:34.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:34.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:34.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:34.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:34.594+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:34.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:34.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:34.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:34.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:34.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:34.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:34.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:34.380+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:35.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:34.774+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:35.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:34.774+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:35.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:34.774+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:35.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:34.776+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:35.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:35.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:35.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:35.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:35.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:35.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:35.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:35.396+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:35.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:35.809+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:35.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:35.809+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:35.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:35.809+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:35.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:35.810+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:35.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:35.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:35.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:35.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:35.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:35.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:35.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:35.613+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:36.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:36.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:36.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:36.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:36.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:36.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:36.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:36.410+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:36.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:36.614+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:36.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:36.614+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:36.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:36.614+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:36.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:36.615+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:37.202 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:43:36.775+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 36126 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:43:37.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:36.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:37.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:36.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:37.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:36.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:37.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:36.767+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:37.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:37.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:37.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:37.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:37.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:37.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:37.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:37.382+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:37.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:37.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:37.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:37.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:37.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:37.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:37.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:37.633+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:37.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:37.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:37.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:37.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:37.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:37.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:37.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:37.802+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:38.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:38.417+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:38.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:38.417+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:38.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:38.417+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:38.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:38.417+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:38.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:38.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:38.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:38.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:38.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:38.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:38.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:38.845+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:38.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:38.674+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:38.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:38.674+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:38.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:38.674+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:38.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:38.675+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:39.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:39.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:39.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:39.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:39.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:39.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:39.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:39.445+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:39.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:39.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:39.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:39.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:39.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:39.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:39.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:39.685+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:40.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:39.843+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:40.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:39.843+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:40.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:39.843+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:40.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:39.844+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:40.794 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:40.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:40.795 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:40.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:40.795 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:40.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:40.795 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:40.702+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:40.795 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:40.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:40.795 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:40.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:40.795 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:40.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:40.795 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:40.494+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:41.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:40.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:41.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:40.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:41.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:40.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:41.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:40.813+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:41.688 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:41.452+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:41.688 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:41.452+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:41.688 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:41.452+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:41.688 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:41.453+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:41.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:43:41.780+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 36215 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:43:41.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:41.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:41.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:41.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:41.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:41.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:41.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:41.822+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:41.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:41.706+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:41.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:41.706+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:41.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:41.706+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:41.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:41.707+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:42.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:42.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:42.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:42.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:42.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:42.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:42.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:42.457+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:42.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:42.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:42.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:42.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:42.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:42.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:42.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:42.698+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:43.079 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:42.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:43.079 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:42.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:43.079 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:42.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:43.079 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:42.799+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:43.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:43.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:43.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:43.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:43.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:43.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:43.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:43.442+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:43.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:43.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:43.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:43.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:43.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:43.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:43.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:43.702+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:43.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:43.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:43.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:43.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:43.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:43.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:43.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:43.811+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:44.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:44.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:44.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:44.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:44.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:44.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:44.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:44.431+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:44.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:44.710+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:44.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:44.710+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:44.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:44.710+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:44.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:44.711+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:44.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:44.780+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:44.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:44.780+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:44.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:44.780+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:44.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:44.781+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:45.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:45.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:45.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:45.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:45.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:45.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:45.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:45.409+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:45.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:45.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:45.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:45.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:45.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:45.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:45.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:45.818+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:45.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:45.672+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:45.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:45.672+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:45.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:45.672+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:45.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:45.673+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:46.640 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:46.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:46.641 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:46.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:46.641 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:46.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:46.641 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:46.388+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:46.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:43:46.784+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 36306 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:43:46.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:46.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:46.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:46.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:46.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:46.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:46.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:46.659+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:46.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:46.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:46.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:46.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:46.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:46.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:46.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:46.814+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:47.632 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:47.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:47.632 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:47.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:47.632 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:47.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:47.632 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:47.340+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:47.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:47.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:47.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:47.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:47.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:47.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:47.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:47.799+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:47.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:47.649+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:47.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:47.649+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:47.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:47.649+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:47.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:47.650+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:48.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:48.296+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:48.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:48.296+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:48.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:48.296+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:48.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:48.297+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:48.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:48.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:48.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:48.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:48.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:48.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:48.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:48.751+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:48.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:48.672+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:48.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:48.672+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:48.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:48.672+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:48.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:48.673+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:49.680 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:49.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:49.680 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:49.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:49.680 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:49.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:49.680 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:49.265+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:49.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:49.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:49.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:49.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:49.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:49.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:49.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:49.756+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:49.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:49.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:49.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:49.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:49.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:49.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:49.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:49.698+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:50.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:50.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:50.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:50.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:50.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:50.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:50.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:50.294+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:51.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:50.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:51.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:50.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:51.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:50.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:51.085 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:50.726+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:51.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:50.726+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:51.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:50.726+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:51.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:50.726+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:51.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:50.727+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:51.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:51.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:51.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:51.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:51.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:51.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:51.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:51.335+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:51.952 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:43:51.789+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 36395 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:43:51.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:51.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:51.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:51.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:51.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:51.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:51.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:51.774+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:51.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:51.688+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:51.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:51.688+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:51.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:51.688+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:51.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:51.689+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:52.655 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:52.357+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:52.655 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:52.357+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:52.655 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:52.357+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:52.655 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:52.358+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:52.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:52.672+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:52.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:52.672+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:52.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:52.672+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:52.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:52.673+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:52.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:52.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:52.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:52.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:52.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:52.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:52.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:52.774+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:53.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:53.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:53.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:53.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:53.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:53.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:53.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:53.368+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:53.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:53.635+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:53.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:53.635+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:53.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:53.635+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:53.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:53.636+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:53.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:53.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:53.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:53.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:53.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:53.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:53.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:53.725+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:54.622 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:54.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:54.623 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:54.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:54.623 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:54.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:54.623 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:54.378+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:54.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:54.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:54.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:54.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:54.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:54.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:54.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:54.641+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:54.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:54.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:54.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:54.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:54.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:54.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:54.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:54.735+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:55.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:55.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:55.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:55.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:55.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:55.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:55.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:55.345+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:55.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:55.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:55.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:55.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:55.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:55.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:55.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:55.599+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:55.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:55.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:55.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:55.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:55.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:55.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:55.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:55.760+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:56.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:56.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:56.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:56.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:56.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:56.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:56.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:56.327+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:56.900 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:43:56.794+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 36487 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:43:56.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:56.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:56.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:56.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:56.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:56.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:56.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:56.626+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:56.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:56.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:56.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:56.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:56.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:56.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:56.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:56.795+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:57.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:57.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:57.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:57.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:57.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:57.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:57.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:57.361+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:57.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:57.787+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:57.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:57.787+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:57.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:57.787+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:57.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:57.788+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:57.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:57.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:57.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:57.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:57.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:57.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:57.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:57.665+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:58.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:58.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:58.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:58.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:58.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:58.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:58.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:58.319+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:58.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:58.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:58.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:58.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:58.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:58.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:58.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:58.827+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:58.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:58.700+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:58.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:58.700+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:58.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:58.700+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:58.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:58.701+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:59.644 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:59.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:59.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:59.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:59.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:59.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:43:59.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:43:59.360+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:59.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:59.661+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:59.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:59.661+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:59.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:59.661+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:43:59.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:43:59.662+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:43:59.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:59.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:59.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:59.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:59.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:59.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:43:59.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:43:59.807+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:00.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:00.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:00.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:00.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:00.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:00.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:00.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:00.395+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:00.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:00.678+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:00.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:00.678+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:00.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:00.678+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:00.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:00.679+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:00.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:00.783+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:00.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:00.783+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:00.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:00.783+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:00.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:00.783+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:01.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:01.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:01.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:01.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:01.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:01.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:01.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:01.404+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:02.203 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:44:01.798+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 36576 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:44:02.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:01.727+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:02.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:01.727+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:02.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:01.727+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:02.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:01.728+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:02.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:01.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:02.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:01.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:02.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:01.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:02.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:01.735+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:02.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:02.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:02.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:02.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:02.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:02.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:02.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:02.368+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:03.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:02.724+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:03.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:02.724+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:03.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:02.724+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:03.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:02.725+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:03.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:02.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:03.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:02.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:03.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:02.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:03.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:02.733+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:03.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:03.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:03.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:03.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:03.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:03.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:03.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:03.382+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:03.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:03.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:03.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:03.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:03.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:03.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:03.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:03.739+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:03.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:03.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:03.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:03.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:03.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:03.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:03.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:03.728+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:05.203 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:04.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:05.203 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:04.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:05.203 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:04.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:05.203 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:04.369+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:05.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:04.710+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:05.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:04.710+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:05.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:04.710+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:05.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:04.711+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:05.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:04.684+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:05.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:04.684+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:05.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:04.684+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:05.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:04.684+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:05.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:05.411+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:05.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:05.411+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:05.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:05.411+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:05.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:05.412+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:05.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:05.733+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:05.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:05.733+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:05.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:05.733+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:05.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:05.734+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:05.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:05.701+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:05.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:05.701+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:05.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:05.701+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:05.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:05.702+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:06.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:06.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:06.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:06.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:06.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:06.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:06.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:06.425+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:07.203 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:44:06.803+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 36667 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:44:07.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:06.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:07.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:06.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:07.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:06.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:07.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:06.780+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:07.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:06.732+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:07.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:06.732+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:07.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:06.732+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:07.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:06.733+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:07.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:07.452+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:07.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:07.452+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:07.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:07.452+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:07.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:07.452+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:08.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:07.813+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:08.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:07.813+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:08.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:07.813+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:08.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:07.814+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:08.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:07.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:08.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:07.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:08.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:07.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:08.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:07.755+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:08.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:08.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:08.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:08.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:08.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:08.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:08.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:08.475+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:09.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:08.797+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:09.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:08.797+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:09.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:08.797+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:09.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:08.798+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:09.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:08.723+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:09.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:08.723+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:09.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:08.723+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:09.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:08.723+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:09.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:09.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:09.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:09.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:09.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:09.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:09.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:09.675+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:09.764 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:09.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:09.764 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:09.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:09.765 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:09.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:09.765 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:09.494+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:10.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:09.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:10.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:09.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:10.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:09.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:10.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:09.781+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:10.735 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:10.652+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:10.735 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:10.652+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:10.735 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:10.652+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:10.735 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:10.652+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:10.735 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:10.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:10.735 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:10.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:10.735 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:10.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:10.736 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:10.493+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:11.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:10.751+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:11.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:10.751+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:11.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:10.751+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:11.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:10.752+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:11.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:11.468+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:11.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:11.468+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:11.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:11.468+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:11.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:11.468+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:11.704 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:11.645+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:11.704 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:11.645+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:11.704 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:11.645+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:11.704 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:11.646+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:12.013 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:44:11.807+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 36756 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:44:12.013 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:11.764+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:12.013 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:11.764+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:12.013 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:11.764+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:12.014 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:11.765+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:12.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:12.727+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:12.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:12.727+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:12.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:12.727+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:12.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:12.728+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:12.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:12.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:12.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:12.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:12.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:12.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:12.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:12.660+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:12.954 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:12.512+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:12.954 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:12.512+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:12.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:12.512+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:12.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:12.512+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:13.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:13.557+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:13.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:13.557+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:13.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:13.557+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:13.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:13.557+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:13.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:13.713+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:13.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:13.713+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:13.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:13.713+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:13.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:13.713+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:13.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:13.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:13.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:13.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:13.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:13.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:13.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:13.683+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:14.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:14.733+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:14.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:14.733+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:14.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:14.733+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:14.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:14.733+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:14.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:14.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:14.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:14.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:14.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:14.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:14.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:14.703+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:14.954 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:14.587+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:14.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:14.587+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:14.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:14.587+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:14.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:14.588+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:15.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:15.748+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:15.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:15.748+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:15.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:15.748+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:15.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:15.749+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:15.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:15.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:15.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:15.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:15.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:15.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:15.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:15.693+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:15.954 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:15.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:15.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:15.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:15.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:15.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:15.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:15.580+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:16.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:16.551+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:16.650 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:16.551+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:16.650 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:16.551+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:16.650 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:16.552+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:16.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:44:16.811+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 36848 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:44:16.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:16.738+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:16.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:16.738+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:16.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:16.738+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:16.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:16.739+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:16.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:16.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:16.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:16.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:16.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:16.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:16.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:16.665+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:17.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:17.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:17.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:17.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:17.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:17.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:17.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:17.679+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:17.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:17.763+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:17.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:17.763+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:17.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:17.763+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:17.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:17.763+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:17.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:17.521+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:17.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:17.521+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:17.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:17.521+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:17.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:17.521+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:18.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:18.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:18.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:18.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:18.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:18.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:18.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:18.723+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:18.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:18.715+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:18.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:18.715+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:18.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:18.715+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:18.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:18.716+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:18.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:18.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:18.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:18.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:18.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:18.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:18.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:18.488+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:19.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:19.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:19.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:19.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:19.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:19.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:19.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:19.714+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:19.954 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:19.478+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:19.954 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:19.478+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:19.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:19.478+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:19.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:19.478+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:19.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:19.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:19.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:19.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:19.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:19.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:19.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:19.683+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:20.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:20.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:20.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:20.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:20.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:20.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:20.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:20.683+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:20.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:20.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:20.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:20.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:20.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:20.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:20.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:20.459+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:21.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:20.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:21.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:20.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:21.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:20.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:21.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:20.723+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:21.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:21.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:21.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:21.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:21.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:21.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:21.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:21.446+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:21.927 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:44:21.816+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 36937 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:44:21.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:21.768+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:21.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:21.768+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:21.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:21.768+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:21.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:21.769+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:21.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:21.688+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:21.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:21.688+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:21.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:21.688+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:21.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:21.689+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:22.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:22.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:22.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:22.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:22.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:22.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:22.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:22.433+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:23.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:22.735+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:23.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:22.735+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:23.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:22.735+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:23.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:22.736+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:23.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:22.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:23.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:22.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:23.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:22.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:23.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:22.726+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:23.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:23.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:23.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:23.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:23.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:23.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:23.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:23.427+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:23.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:23.751+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:23.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:23.751+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:23.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:23.751+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:23.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:23.752+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:23.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:23.703+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:23.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:23.703+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:23.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:23.703+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:23.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:23.703+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:24.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:24.437+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:24.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:24.437+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:24.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:24.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:24.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:24.438+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:24.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:24.754+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:24.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:24.754+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:24.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:24.754+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:24.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:24.754+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:24.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:24.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:24.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:24.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:24.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:24.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:24.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:24.683+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:25.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:25.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:25.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:25.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:25.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:25.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:25.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:25.393+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:25.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:25.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:25.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:25.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:25.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:25.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:25.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:25.692+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:25.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:25.789+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:25.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:25.789+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:25.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:25.789+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:25.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:25.789+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:26.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:26.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:26.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:26.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:26.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:26.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:26.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:26.378+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:26.954 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:44:26.820+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 37028 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:44:26.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:26.764+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:26.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:26.764+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:26.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:26.764+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:26.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:26.764+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:26.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:26.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:26.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:26.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:26.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:26.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:26.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:26.707+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:27.680 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:27.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:27.680 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:27.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:27.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:27.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:27.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:27.410+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:27.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:27.750+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:27.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:27.750+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:27.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:27.750+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:27.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:27.750+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:27.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:27.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:27.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:27.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:27.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:27.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:27.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:27.697+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:28.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:28.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:28.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:28.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:28.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:28.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:28.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:28.378+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:28.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:28.793+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:28.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:28.793+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:28.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:28.793+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:28.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:28.794+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:28.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:28.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:28.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:28.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:28.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:28.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:28.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:28.687+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:29.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:29.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:29.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:29.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:29.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:29.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:29.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:29.399+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:30.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:29.808+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:30.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:29.808+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:30.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:29.808+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:30.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:29.808+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:30.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:29.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:30.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:29.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:30.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:29.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:30.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:29.724+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:30.662 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:30.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:30.662 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:30.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:30.662 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:30.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:30.662 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:30.424+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:30.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:30.776+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:30.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:30.776+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:30.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:30.776+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:30.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:30.777+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:30.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:30.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:30.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:30.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:30.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:30.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:30.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:30.678+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:31.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:31.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:31.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:31.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:31.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:31.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:31.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:31.414+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:31.954 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:44:31.826+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 37117 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:44:31.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:31.782+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:31.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:31.782+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:31.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:31.782+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:31.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:31.782+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:31.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:31.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:31.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:31.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:31.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:31.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:31.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:31.712+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:32.704 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:32.675+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:32.705 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:32.675+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:32.705 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:32.675+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:32.705 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:32.676+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:32.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:32.427+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:32.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:32.427+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:32.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:32.427+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:32.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:32.427+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:33.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:32.816+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:33.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:32.816+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:33.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:32.816+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:33.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:32.816+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:33.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:33.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:33.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:33.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:33.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:33.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:33.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:33.425+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:33.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:33.701+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:33.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:33.701+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:33.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:33.701+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:33.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:33.702+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:33.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:33.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:33.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:33.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:33.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:33.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:33.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:33.800+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:34.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:34.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:34.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:34.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:34.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:34.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:34.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:34.386+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:34.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:34.801+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:34.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:34.801+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:34.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:34.801+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:34.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:34.802+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:34.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:34.700+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:34.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:34.700+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:34.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:34.700+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:34.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:34.700+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:35.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:35.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:35.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:35.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:35.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:35.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:35.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:35.344+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:35.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:35.756+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:35.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:35.756+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:35.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:35.756+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:35.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:35.757+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:35.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:35.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:35.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:35.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:35.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:35.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:35.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:35.720+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:36.589 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:36.323+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:36.589 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:36.323+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:36.590 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:36.323+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:36.590 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:36.323+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:36.936 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:44:36.831+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 37209 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:44:36.937 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:36.770+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:36.937 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:36.770+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:36.937 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:36.770+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:36.937 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:36.771+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:36.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:36.714+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:36.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:36.714+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:36.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:36.714+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:36.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:36.715+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:37.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:37.314+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:37.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:37.314+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:37.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:37.314+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:37.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:37.314+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:37.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:37.748+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:37.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:37.748+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:37.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:37.748+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:37.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:37.749+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:37.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:37.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:37.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:37.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:37.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:37.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:37.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:37.712+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:38.531 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:38.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:38.531 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:38.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:38.531 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:38.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:38.531 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:38.285+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:38.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:38.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:38.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:38.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:38.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:38.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:38.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:38.718+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:38.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:38.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:38.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:38.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:38.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:38.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:38.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:38.677+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:39.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:39.290+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:39.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:39.290+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:39.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:39.290+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:39.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:39.290+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:39.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:39.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:39.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:39.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:39.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:39.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:39.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:39.677+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:39.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:39.723+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:39.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:39.723+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:39.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:39.723+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:39.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:39.724+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:40.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:40.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:40.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:40.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:40.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:40.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:40.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:40.335+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:40.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:40.772+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:40.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:40.772+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:40.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:40.772+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:40.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:40.774+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:40.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:40.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:40.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:40.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:40.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:40.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:40.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:40.703+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:41.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:41.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:41.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:41.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:41.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:41.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:41.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:41.331+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:42.204 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:44:41.836+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 37298 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:44:42.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:41.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:42.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:41.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:42.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:41.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:42.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:41.739+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:42.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:41.820+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:42.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:41.820+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:42.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:41.820+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:42.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:41.821+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:42.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:42.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:42.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:42.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:42.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:42.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:42.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:42.360+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:43.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:42.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:43.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:42.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:43.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:42.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:43.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:42.764+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:43.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:42.857+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:43.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:42.857+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:43.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:42.857+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:43.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:42.858+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:43.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:43.316+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:43.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:43.316+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:43.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:43.316+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:43.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:43.316+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:43.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:43.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:43.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:43.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:43.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:43.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:43.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:43.734+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:43.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:43.823+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:43.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:43.823+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:43.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:43.823+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:43.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:43.824+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:44.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:44.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:44.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:44.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:44.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:44.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:44.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:44.277+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:45.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:44.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:45.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:44.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:45.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:44.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:45.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:44.761+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:45.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:44.850+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:45.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:44.850+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:45.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:44.850+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:45.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:44.851+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:45.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:45.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:45.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:45.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:45.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:45.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:45.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:45.228+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:46.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:45.896+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:46.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:45.896+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:46.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:45.896+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:46.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:45.897+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:46.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:45.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:46.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:45.787+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:46.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:45.787+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:46.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:45.787+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:46.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:46.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:46.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:46.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:46.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:46.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:46.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:46.214+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:47.167 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:44:46.840+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 37389 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:44:47.167 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:46.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:47.168 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:46.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:47.168 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:46.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:47.168 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:46.802+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:47.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:46.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:47.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:46.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:47.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:46.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:47.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:46.942+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:47.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:47.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:47.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:47.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:47.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:47.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:47.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:47.183+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:48.134 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:47.958+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:48.134 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:47.958+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:48.134 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:47.958+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:48.134 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:47.959+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:48.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:47.768+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:48.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:47.768+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:48.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:47.768+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:48.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:47.769+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:48.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:48.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:48.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:48.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:48.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:48.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:48.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:48.149+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:48.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:48.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:48.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:48.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:48.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:48.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:48.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:48.753+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:49.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:48.956+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:49.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:48.956+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:49.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:48.956+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:49.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:48.957+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:49.205 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:49.141+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:49.205 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:49.141+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:49.205 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:49.141+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:49.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:49.142+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:50.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:49.940+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:50.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:49.940+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:50.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:49.940+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:50.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:49.941+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:50.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:49.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:50.079 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:49.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:50.079 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:49.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:50.079 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:49.764+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:50.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:50.093+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:50.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:50.093+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:50.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:50.093+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:50.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:50.093+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:50.918+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:50.918+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:50.918+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:51.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:50.918+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:51.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:50.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:51.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:50.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:51.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:50.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:51.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:50.751+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:51.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:51.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:51.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:51.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:51.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:51.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:51.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:51.085+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:52.035 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:44:51.845+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 37478 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:44:52.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:51.960+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:52.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:51.960+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:52.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:51.960+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:52.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:51.961+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:52.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:51.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:52.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:51.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:52.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:51.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:52.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:51.744+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:52.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:52.050+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:52.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:52.050+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:52.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:52.050+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:52.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:52.050+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:52.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:52.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:52.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:52.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:52.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:52.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:52.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:52.728+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:53.455 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:52.985+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:53.455 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:52.985+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:53.455 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:52.985+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:53.455 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:52.986+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:53.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:53.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:53.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:53.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:53.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:53.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:53.456 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:53.097+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:53.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:53.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:53.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:53.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:53.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:53.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:53.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:53.723+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:54.455 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:54.003+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:54.455 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:54.003+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:54.455 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:54.003+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:54.455 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:54.004+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:54.456 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:54.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:54.456 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:54.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:54.456 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:54.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:54.456 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:54.055+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:54.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:54.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:54.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:54.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:54.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:54.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:54.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:54.738+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:55.455 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:54.990+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:55.455 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:54.990+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:55.456 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:54.990+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:55.456 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:54.991+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:55.456 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:55.057+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:55.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:55.057+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:55.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:55.057+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:55.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:55.057+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:56.005 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:55.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:56.005 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:55.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:56.005 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:55.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:56.006 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:55.751+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:56.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:56.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:56.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:56.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:56.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:56.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:56.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:56.028+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:56.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:56.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:56.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:56.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:56.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:56.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:56.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:56.021+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:57.205 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:44:56.850+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 37570 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:44:57.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:56.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:57.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:56.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:57.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:56.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:57.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:56.975+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:57.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:57.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:57.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:57.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:57.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:57.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:57.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:57.022+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:57.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:56.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:57.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:56.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:57.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:56.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:57.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:56.787+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:58.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:57.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:58.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:57.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:58.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:57.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:58.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:57.940+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:58.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:58.016+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:58.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:58.016+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:58.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:58.016+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:58.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:58.016+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:58.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:57.768+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:58.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:57.768+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:58.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:57.768+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:58.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:57.769+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:58.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:58.924+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:58.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:58.924+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:58.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:58.924+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:44:58.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:58.924+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:58.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:58.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:58.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:58.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:58.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:58.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:44:58.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:58.791+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:44:59.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:59.002+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:59.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:59.002+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:59.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:59.002+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:44:59.456 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:59.002+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:00.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:59.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:00.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:59.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:00.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:59.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:00.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:44:59.794+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:00.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:59.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:00.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:59.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:00.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:59.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:00.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:44:59.885+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:00.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:59.976+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:00.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:59.976+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:00.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:59.976+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:00.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:44:59.976+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:01.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:00.835+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:01.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:00.835+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:01.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:00.835+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:01.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:00.835+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:01.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:00.926+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:01.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:00.926+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:01.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:00.926+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:01.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:00.927+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:01.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:00.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:01.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:00.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:01.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:00.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:01.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:00.944+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:02.205 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:45:01.854+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 37659 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:45:02.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:01.896+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:02.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:01.896+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:02.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:01.896+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:02.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:01.897+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:02.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:01.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:02.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:01.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:02.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:01.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:02.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:01.835+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:02.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:01.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:02.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:01.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:02.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:01.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:02.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:01.912+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:03.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:02.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:03.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:02.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:03.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:02.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:03.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:02.794+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:03.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:02.856+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:03.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:02.856+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:03.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:02.856+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:03.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:02.857+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:03.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:02.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:03.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:02.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:03.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:02.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:03.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:02.951+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:03.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:03.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:03.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:03.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:03.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:03.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:03.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:03.828+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:03.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:03.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:03.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:03.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:03.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:03.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:03.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:03.799+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:04.205 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:03.932+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:04.205 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:03.932+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:04.205 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:03.932+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:04.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:03.933+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:05.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:04.842+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:05.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:04.842+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:05.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:04.842+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:05.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:04.843+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:05.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:04.836+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:05.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:04.836+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:05.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:04.836+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:05.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:04.836+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:05.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:04.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:05.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:04.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:05.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:04.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:05.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:04.935+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:06.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:05.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:06.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:05.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:06.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:05.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:06.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:05.813+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:06.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:05.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:06.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:05.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:06.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:05.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:06.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:05.905+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:06.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:05.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:06.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:05.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:06.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:05.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:06.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:05.839+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:07.205 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:45:06.859+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 37750 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:45:07.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:06.797+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:07.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:06.797+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:07.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:06.797+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:07.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:06.798+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:07.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:06.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:07.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:06.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:07.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:06.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:07.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:06.905+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:07.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:06.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:07.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:06.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:07.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:06.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:07.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:06.817+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:08.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:07.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:08.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:07.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:08.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:07.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:08.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:07.832+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:08.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:07.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:08.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:07.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:08.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:07.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:08.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:07.834+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:08.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:07.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:08.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:07.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:08.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:07.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:08.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:07.885+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:09.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:08.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:09.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:08.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:09.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:08.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:09.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:08.806+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:09.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:08.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:09.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:08.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:09.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:08.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:09.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:08.917+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:09.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:08.804+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:09.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:08.804+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:09.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:08.804+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:09.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:08.805+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:10.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:09.821+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:10.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:09.821+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:10.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:09.821+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:10.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:09.821+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:10.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:09.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:10.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:09.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:10.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:09.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:10.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:09.878+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:10.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:09.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:10.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:09.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:10.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:09.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:10.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:09.807+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:11.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:10.820+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:11.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:10.820+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:11.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:10.820+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:11.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:10.820+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:11.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:10.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:11.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:10.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:11.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:10.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:11.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:10.800+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:11.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:10.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:11.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:10.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:11.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:10.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:11.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:10.875+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:12.206 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:45:11.864+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 37839 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:45:12.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:11.823+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:12.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:11.823+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:12.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:11.823+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:12.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:11.824+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:12.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:11.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:12.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:11.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:12.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:11.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:12.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:11.797+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:12.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:11.840+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:12.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:11.840+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:12.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:11.840+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:12.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:11.840+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:13.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:12.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:13.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:12.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:13.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:12.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:13.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:12.828+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:13.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:12.835+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:13.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:12.835+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:13.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:12.835+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:13.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:12.836+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:13.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:12.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:13.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:12.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:13.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:12.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:13.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:12.851+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:14.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:13.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:14.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:13.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:14.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:13.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:14.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:13.898+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:14.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:13.829+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:14.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:13.829+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:14.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:13.829+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:14.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:13.829+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:14.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:13.843+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:14.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:13.843+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:14.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:13.843+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:14.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:13.844+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:15.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:14.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:15.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:14.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:15.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:14.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:15.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:14.874+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:15.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:14.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:15.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:14.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:15.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:14.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:15.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:14.852+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:15.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:14.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:15.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:14.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:15.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:14.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:15.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:14.848+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:16.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:15.867+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:16.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:15.867+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:16.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:15.867+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:16.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:15.868+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:16.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:15.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:16.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:15.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:16.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:15.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:16.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:15.813+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:16.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:15.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:16.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:15.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:16.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:15.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:16.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:15.881+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:17.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:45:16.867+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 37931 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:45:17.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:16.849+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:17.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:16.849+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:17.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:16.849+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:17.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:16.850+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:17.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:16.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:17.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:16.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:17.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:16.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:17.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:16.842+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:17.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:16.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:17.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:16.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:17.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:16.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:17.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:16.793+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:18.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:17.897+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:18.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:17.897+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:18.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:17.898+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:18.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:17.899+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:18.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:17.814+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:18.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:17.814+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:18.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:17.814+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:18.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:17.815+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:18.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:17.860+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:18.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:17.860+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:18.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:17.860+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:18.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:17.861+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:19.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:18.929+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:19.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:18.929+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:19.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:18.929+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:19.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:18.930+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:19.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:18.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:19.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:18.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:19.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:18.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:19.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:18.905+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:19.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:18.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:19.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:18.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:19.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:18.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:19.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:18.853+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:20.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:19.897+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:20.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:19.897+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:20.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:19.897+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:20.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:19.898+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:20.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:19.906+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:20.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:19.906+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:20.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:19.906+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:20.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:19.907+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:20.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:19.902+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:20.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:19.902+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:20.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:19.902+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:20.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:19.903+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:21.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:20.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:21.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:20.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:21.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:20.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:21.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:20.898+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:21.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:20.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:21.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:20.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:21.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:20.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:21.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:20.918+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:21.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:20.866+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:21.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:20.866+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:21.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:20.866+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:21.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:20.866+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:22.206 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:45:21.872+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 38020 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:45:22.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:21.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:22.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:21.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:22.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:21.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:22.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:21.891+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:22.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:21.922+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:22.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:21.922+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:22.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:21.922+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:22.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:21.923+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:22.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:21.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:22.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:21.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:22.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:21.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:22.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:21.824+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:22.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:22.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:22.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:22.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:22.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:22.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:22.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:22.827+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:22.957 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:22.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:22.957 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:22.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:22.957 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:22.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:22.957 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:22.898+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:22.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:22.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:22.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:22.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:22.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:22.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:22.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:22.860+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:23.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:23.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:23.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:23.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:23.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:23.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:23.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:23.865+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:23.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:23.894+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:23.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:23.895+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:23.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:23.895+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:23.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:23.895+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:23.957 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:23.783+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:23.957 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:23.783+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:23.957 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:23.783+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:23.957 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:23.783+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:25.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:24.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:25.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:24.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:25.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:24.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:25.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:24.857+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:25.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:24.791+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:25.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:24.791+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:25.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:24.791+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:25.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:24.791+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:25.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:24.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:25.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:24.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:25.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:24.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:25.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:24.861+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:26.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:25.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:26.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:25.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:26.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:25.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:26.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:25.894+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:26.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:25.839+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:26.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:25.839+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:26.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:25.839+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:26.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:25.839+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:26.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:25.893+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:26.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:25.893+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:26.109 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:25.893+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:26.109 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:25.893+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:27.206 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:45:26.878+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 38111 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:45:27.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:26.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:27.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:26.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:27.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:26.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:27.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:26.883+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:27.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:26.816+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:27.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:26.816+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:27.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:26.816+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:27.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:26.817+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:27.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:26.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:27.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:26.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:27.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:26.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:27.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:26.864+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:28.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:27.788+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:28.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:27.788+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:28.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:27.788+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:28.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:27.789+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:28.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:27.883+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:28.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:27.883+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:28.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:27.883+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:28.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:27.884+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:28.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:27.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:28.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:27.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:28.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:27.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:28.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:27.856+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:29.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:28.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:29.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:28.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:29.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:28.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:29.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:28.828+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:29.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:28.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:29.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:28.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:29.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:28.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:29.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:28.894+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:29.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:28.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:29.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:28.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:29.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:28.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:29.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:28.892+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:30.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:29.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:30.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:29.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:30.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:29.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:30.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:29.875+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:30.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:29.933+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:30.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:29.933+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:30.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:29.933+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:30.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:29.933+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:30.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:29.881+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:30.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:29.881+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:30.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:29.881+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:30.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:29.882+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:31.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:30.867+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:31.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:30.867+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:31.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:30.867+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:31.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:30.868+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:31.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:30.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:31.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:30.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:31.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:30.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:31.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:30.842+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:31.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:30.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:31.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:30.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:31.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:30.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:31.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:30.899+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:32.206 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:45:31.882+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 38200 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:45:32.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:31.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:32.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:31.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:32.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:31.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:32.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:31.837+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:32.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:31.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:32.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:31.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:32.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:31.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:32.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:31.929+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:32.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:31.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:32.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:31.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:32.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:31.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:32.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:31.875+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:33.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:32.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:33.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:32.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:33.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:32.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:33.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:32.865+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:33.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:32.889+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:33.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:32.889+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:33.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:32.889+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:33.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:32.891+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:33.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:32.900+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:33.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:32.900+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:33.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:32.900+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:33.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:32.901+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:33.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:33.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:33.957 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:33.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:33.957 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:33.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:33.957 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:33.819+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:33.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:33.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:33.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:33.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:33.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:33.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:33.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:33.882+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:33.957 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:33.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:33.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:33.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:33.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:33.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:33.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:33.875+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:35.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:34.888+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:35.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:34.888+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:35.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:34.888+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:35.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:34.889+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:35.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:34.918+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:35.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:34.918+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:35.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:34.918+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:35.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:34.919+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:35.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:34.801+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:35.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:34.801+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:35.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:34.801+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:35.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:34.801+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:36.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:35.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:36.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:35.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:36.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:35.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:36.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:35.939+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:36.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:35.878+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:36.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:35.878+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:36.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:35.878+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:36.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:35.878+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:36.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:35.807+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:36.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:35.807+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:36.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:35.807+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:36.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:35.808+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:37.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:45:36.887+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 38292 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:45:37.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:36.808+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:37.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:36.808+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:37.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:36.808+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:37.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:36.809+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:37.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:36.838+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:37.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:36.838+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:37.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:36.838+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:37.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:36.839+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:37.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:36.903+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:37.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:36.903+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:37.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:36.903+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:37.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:36.903+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:38.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:37.934+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:38.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:37.934+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:38.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:37.934+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:38.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:37.934+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:38.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:37.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:38.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:37.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:38.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:37.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:38.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:37.831+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:38.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:37.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:38.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:37.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:38.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:37.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:38.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:37.835+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:39.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:38.837+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:39.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:38.837+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:39.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:38.837+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:39.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:38.837+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:39.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:38.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:39.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:38.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:39.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:38.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:39.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:38.858+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:39.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:38.904+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:39.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:38.904+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:39.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:38.904+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:39.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:38.906+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:39.887 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:39.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:39.887 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:39.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:39.887 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:39.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:39.887 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:39.870+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:39.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:39.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:39.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:39.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:39.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:39.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:39.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:39.868+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:40.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:39.900+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:40.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:39.900+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:40.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:39.900+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:40.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:39.900+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:41.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:40.913+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:41.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:40.913+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:41.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:40.913+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:41.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:40.913+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:41.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:40.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:41.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:40.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:41.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:40.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:41.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:40.937+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:41.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:40.855+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:41.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:40.855+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:41.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:40.855+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:41.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:40.855+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:42.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:45:41.892+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 38381 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:45:42.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:41.871+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:42.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:41.871+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:42.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:41.871+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:42.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:41.872+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:42.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:41.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:42.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:41.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:42.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:41.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:42.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:41.938+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:42.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:41.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:42.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:41.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:42.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:41.896+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:42.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:41.896+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:43.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:42.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:43.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:42.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:43.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:42.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:43.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:42.857+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:43.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:42.926+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:43.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:42.926+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:43.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:42.926+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:43.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:42.927+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:43.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:42.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:43.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:42.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:43.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:42.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:43.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:42.923+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:43.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:43.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:43.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:43.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:43.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:43.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:43.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:43.893+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:43.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:43.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:43.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:43.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:43.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:43.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:43.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:43.842+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:44.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:43.943+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:44.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:43.943+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:44.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:43.943+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:44.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:43.944+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:45.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:44.894+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:45.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:44.894+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:45.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:44.894+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:45.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:44.895+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:45.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:44.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:45.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:44.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:45.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:44.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:45.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:44.851+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:45.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:44.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:45.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:44.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:45.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:44.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:45.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:44.943+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:46.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:45.900+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:46.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:45.900+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:46.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:45.900+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:46.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:45.900+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:46.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:45.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:46.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:45.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:46.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:45.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:46.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:45.839+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:46.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:45.932+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:46.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:45.932+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:46.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:45.932+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:46.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:45.932+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:47.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:45:46.897+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 38472 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:45:47.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:46.923+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:47.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:46.923+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:47.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:46.923+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:47.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:46.924+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:47.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:46.866+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:47.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:46.866+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:47.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:46.866+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:47.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:46.866+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:47.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:46.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:47.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:46.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:47.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:46.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:47.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:46.892+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:48.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:47.889+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:48.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:47.889+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:48.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:47.889+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:48.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:47.890+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:48.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:47.885+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:48.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:47.885+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:48.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:47.885+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:48.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:47.886+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:48.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:47.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:48.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:47.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:48.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:47.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:48.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:47.937+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:49.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:48.865+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:49.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:48.865+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:49.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:48.865+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:49.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:48.866+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:49.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:48.916+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:49.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:48.916+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:49.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:48.916+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:49.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:48.917+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:49.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:48.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:49.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:48.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:49.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:48.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:49.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:48.921+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:50.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:49.855+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:50.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:49.855+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:50.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:49.855+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:50.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:49.856+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:50.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:49.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:50.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:49.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:50.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:49.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:50.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:49.870+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:50.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:49.925+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:50.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:49.925+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:50.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:49.925+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:50.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:49.925+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:51.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:50.884+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:51.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:50.884+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:51.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:50.884+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:51.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:50.884+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:51.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:50.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:51.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:50.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:51.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:50.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:51.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:50.851+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:51.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:50.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:51.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:50.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:51.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:50.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:51.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:50.957+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:52.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:45:51.902+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 38561 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:45:52.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:51.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:52.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:51.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:52.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:51.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:52.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:51.802+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:52.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:51.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:52.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:51.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:52.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:51.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:52.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:51.862+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:52.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:51.990+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:52.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:51.990+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:52.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:51.990+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:52.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:51.990+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:53.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:52.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:53.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:52.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:53.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:52.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:53.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:52.830+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:53.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:52.819+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:53.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:52.819+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:53.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:52.819+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:53.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:52.820+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:53.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:52.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:53.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:52.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:53.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:52.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:53.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:52.991+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:53.941 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:53.809+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:53.941 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:53.810+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:53.942 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:53.810+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:53.942 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:53.811+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:53.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:53.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:53.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:53.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:53.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:53.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:53.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:53.802+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:54.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:53.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:54.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:53.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:54.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:53.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:54.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:53.954+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:55.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:54.828+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:55.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:54.828+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:55.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:54.828+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:55.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:54.829+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:55.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:54.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:55.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:54.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:55.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:54.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:55.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:54.943+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:55.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:54.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:55.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:54.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:55.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:54.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:55.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:54.778+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:56.109 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:55.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:56.109 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:55.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:56.109 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:55.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:56.109 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:55.931+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:56.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:55.848+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:56.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:55.848+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:56.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:55.848+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:56.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:55.850+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:56.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:55.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:56.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:55.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:56.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:55.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:56.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:55.795+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:57.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:45:56.907+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 38653 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:45:57.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:56.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:57.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:56.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:57.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:56.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:57.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:56.816+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:57.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:56.844+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:57.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:56.844+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:57.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:56.844+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:57.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:56.845+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:57.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:56.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:57.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:56.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:57.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:56.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:57.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:56.885+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:58.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:57.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:58.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:57.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:58.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:57.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:58.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:57.891+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:58.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:57.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:58.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:57.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:58.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:57.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:58.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:57.857+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:58.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:57.836+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:58.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:57.836+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:58.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:57.836+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:58.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:57.837+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:59.179 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:58.908+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:59.179 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:58.908+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:59.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:58.908+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:45:59.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:58.908+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:59.180 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:58.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:59.180 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:58.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:59.180 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:58.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:45:59.180 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:58.821+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:45:59.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:58.837+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:59.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:58.837+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:59.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:58.837+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:45:59.181 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:58.837+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:00.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:59.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:00.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:59.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:00.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:59.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:00.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:45:59.799+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:00.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:59.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:00.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:59.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:00.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:59.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:00.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:45:59.794+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:00.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:59.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:00.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:59.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:00.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:59.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:00.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:45:59.949+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:01.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:00.778+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:01.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:00.778+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:01.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:00.778+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:01.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:00.779+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:01.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:00.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:01.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:00.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:01.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:00.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:01.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:00.819+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:01.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:00.904+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:01.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:00.904+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:01.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:00.904+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:01.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:00.904+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:02.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:46:01.911+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 38742 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:46:02.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:01.804+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:02.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:01.804+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:02.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:01.804+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:02.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:01.805+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:02.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:01.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:02.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:01.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:02.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:01.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:02.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:01.821+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:02.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:01.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:02.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:01.940+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:02.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:01.940+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:02.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:01.940+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:03.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:02.916+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:03.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:02.916+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:03.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:02.916+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:03.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:02.917+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:03.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:02.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:03.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:02.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:03.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:02.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:03.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:02.808+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:03.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:02.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:03.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:02.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:03.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:02.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:03.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:02.773+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:03.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:03.838+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:03.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:03.838+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:03.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:03.838+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:03.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:03.839+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:03.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:03.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:03.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:03.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:03.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:03.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:03.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:03.777+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:04.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:03.926+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:04.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:03.926+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:04.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:03.926+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:04.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:03.927+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:05.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:04.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:05.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:04.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:05.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:04.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:05.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:04.802+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:05.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:04.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:05.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:04.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:05.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:04.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:05.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:04.772+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:05.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:04.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:05.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:04.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:05.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:04.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:05.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:04.900+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:05.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:05.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:05.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:05.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:05.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:05.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:05.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:05.835+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:05.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:05.804+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:05.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:05.804+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:05.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:05.804+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:05.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:05.804+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:06.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:05.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:06.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:05.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:06.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:05.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:06.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:05.946+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:07.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:46:06.916+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 38833 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:46:07.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:06.970+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:07.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:06.970+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:07.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:06.970+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:07.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:06.970+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:07.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:06.881+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:07.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:06.881+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:07.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:06.881+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:07.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:06.881+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:07.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:06.837+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:07.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:06.837+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:07.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:06.837+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:07.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:06.837+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:08.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:07.872+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:08.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:07.872+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:08.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:07.872+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:08.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:07.872+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:08.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:07.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:08.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:07.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:08.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:07.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:08.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:07.933+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:08.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:07.901+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:08.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:07.901+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:08.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:07.901+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:08.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:07.902+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:09.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:08.941+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:09.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:08.941+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:09.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:08.941+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:09.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:08.941+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:09.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:08.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:09.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:08.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:09.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:08.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:09.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:08.905+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:09.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:08.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:09.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:08.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:09.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:08.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:09.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:08.965+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:10.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:09.965+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:10.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:09.965+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:10.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:09.965+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:10.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:09.966+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:10.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:09.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:10.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:09.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:10.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:09.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:10.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:09.873+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:10.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:09.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:10.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:09.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:10.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:09.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:10.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:09.926+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:11.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:10.901+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:11.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:10.901+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:11.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:10.901+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:11.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:10.902+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:11.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:10.931+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:11.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:10.931+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:11.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:10.931+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:11.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:10.931+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:11.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:10.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:11.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:10.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:11.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:10.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:11.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:10.915+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:12.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:46:11.921+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 38922 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:46:12.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:11.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:12.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:11.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:12.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:11.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:12.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:11.891+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:12.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:11.881+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:12.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:11.881+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:12.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:11.881+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:12.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:11.882+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:12.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:11.895+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:12.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:11.895+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:12.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:11.895+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:12.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:11.895+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:13.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:12.858+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:13.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:12.858+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:13.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:12.858+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:13.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:12.858+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:13.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:12.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:13.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:12.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:13.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:12.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:13.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:12.845+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:13.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:12.879+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:13.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:12.879+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:13.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:12.879+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:13.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:12.880+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:13.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:13.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:13.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:13.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:13.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:13.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:13.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:13.857+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:13.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:13.887+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:13.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:13.887+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:13.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:13.887+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:13.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:13.887+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:13.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:13.878+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:13.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:13.878+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:13.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:13.878+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:13.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:13.878+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:15.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:14.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:15.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:14.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:15.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:14.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:15.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:14.922+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:15.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:14.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:15.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:14.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:15.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:14.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:15.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:14.881+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:15.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:14.878+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:15.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:14.878+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:15.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:14.878+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:15.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:14.878+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:16.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:15.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:16.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:15.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:16.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:15.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:16.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:15.852+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:16.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:15.958+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:16.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:15.958+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:16.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:15.958+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:16.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:15.958+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:16.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:15.885+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:16.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:15.885+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:16.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:15.885+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:16.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:15.886+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:17.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:46:16.925+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 39014 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:46:17.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:16.875+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:17.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:16.875+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:17.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:16.875+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:17.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:16.876+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:17.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:16.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:17.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:16.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:17.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:16.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:17.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:16.843+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:17.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:16.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:17.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:16.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:17.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:16.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:17.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:16.997+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:18.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:18.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:18.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:18.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:18.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:18.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:18.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:18.009+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:18.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:17.875+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:18.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:17.875+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:18.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:17.875+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:18.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:17.876+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:18.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:17.802+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:18.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:17.802+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:18.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:17.802+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:18.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:17.803+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:19.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:18.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:19.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:18.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:19.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:18.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:19.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:18.854+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:19.042 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:18.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:19.042 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:18.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:19.042 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:18.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:19.042 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:18.772+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:19.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:19.051+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:19.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:19.051+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:19.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:19.051+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:19.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:19.052+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:20.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:19.832+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:20.092 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:19.832+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:20.092 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:19.832+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:20.092 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:19.833+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:20.092 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:19.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:20.092 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:19.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:20.092 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:19.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:20.092 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:19.774+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:20.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:20.102+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:20.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:20.102+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:20.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:20.102+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:20.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:20.103+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:21.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:20.837+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:21.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:20.837+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:21.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:20.837+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:21.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:20.838+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:21.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:20.806+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:21.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:20.806+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:21.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:20.806+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:21.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:20.807+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:21.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:21.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:21.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:21.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:21.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:21.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:21.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:21.061+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:21.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:21.823+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:21.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:21.823+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:21.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:21.823+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:21.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:21.823+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:21.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:21.803+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:21.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:21.803+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:21.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:21.803+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:21.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:21.804+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:22.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:46:21.931+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 39103 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:46:22.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:22.064+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:22.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:22.064+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:22.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:22.064+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:22.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:22.064+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:23.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:22.836+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:23.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:22.836+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:23.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:22.836+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:23.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:22.837+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:23.091 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:22.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:23.092 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:22.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:23.092 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:22.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:23.092 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:22.813+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:23.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:23.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:23.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:23.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:23.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:23.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:23.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:23.102+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:23.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:23.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:23.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:23.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:23.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:23.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:23.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:23.874+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:23.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:23.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:23.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:23.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:23.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:23.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:23.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:23.835+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:24.390 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:24.104+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:24.391 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:24.104+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:24.391 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:24.104+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:24.391 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:24.104+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:25.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:24.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:25.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:24.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:25.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:24.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:25.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:24.911+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:25.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:25.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:25.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:25.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:25.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:25.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:25.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:25.089+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:25.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:24.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:25.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:24.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:25.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:24.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:25.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:24.842+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:26.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:25.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:26.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:25.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:26.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:25.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:26.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:25.817+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:26.097 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:25.890+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:26.098 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:25.890+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:26.098 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:25.890+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:26.098 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:25.890+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:26.355 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:26.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:26.356 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:26.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:26.356 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:26.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:26.356 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:26.110+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:27.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:46:26.936+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 39194 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:46:27.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:26.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:27.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:26.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:27.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:26.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:27.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:26.892+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:27.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:27.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:27.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:27.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:27.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:27.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:27.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:27.065+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:27.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:26.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:27.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:26.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:27.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:26.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:27.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:26.850+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:28.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:27.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:28.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:27.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:28.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:27.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:28.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:27.812+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:28.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:28.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:28.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:28.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:28.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:28.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:28.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:28.053+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:28.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:27.864+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:28.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:27.864+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:28.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:27.864+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:28.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:27.865+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:29.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:29.083+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:29.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:29.083+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:29.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:29.083+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:29.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:29.084+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:29.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:28.894+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:29.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:28.894+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:29.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:28.894+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:29.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:28.895+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:29.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:28.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:29.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:28.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:29.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:28.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:29.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:28.837+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:30.120 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:29.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:30.120 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:29.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:30.120 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:29.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:30.120 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:29.941+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:30.120 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:29.858+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:30.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:29.858+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:30.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:29.858+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:30.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:29.858+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:30.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:30.131+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:30.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:30.131+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:30.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:30.131+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:30.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:30.132+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:31.106 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:30.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:31.106 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:30.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:31.106 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:30.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:31.106 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:30.827+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:31.106 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:30.914+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:31.106 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:30.914+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:31.106 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:30.914+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:31.106 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:30.914+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:31.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:31.117+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:31.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:31.117+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:31.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:31.117+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:31.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:31.118+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:32.056 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:46:31.941+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 39283 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:46:32.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:31.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:32.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:31.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:32.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:31.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:32.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:31.787+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:32.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:31.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:32.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:31.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:32.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:31.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:32.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:31.893+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:32.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:32.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:32.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:32.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:32.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:32.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:32.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:32.146+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:33.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:32.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:33.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:32.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:33.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:32.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:33.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:32.752+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:33.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:32.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:33.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:32.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:33.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:32.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:33.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:32.873+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:33.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:33.139+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:33.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:33.139+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:33.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:33.139+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:33.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:33.140+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:33.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:33.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:33.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:33.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:33.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:33.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:33.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:33.854+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:33.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:33.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:33.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:33.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:33.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:33.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:33.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:33.801+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:34.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:34.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:34.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:34.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:34.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:34.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:34.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:34.149+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:35.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:34.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:35.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:34.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:35.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:34.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:35.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:34.830+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:35.113 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:34.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:35.113 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:34.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:35.113 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:34.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:35.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:34.850+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:35.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:35.124+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:35.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:35.124+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:35.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:35.124+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:35.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:35.125+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:36.086 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:35.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:36.086 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:35.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:36.086 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:35.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:36.086 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:35.832+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:36.086 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:35.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:36.087 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:35.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:36.087 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:35.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:36.087 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:35.845+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:36.355 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:36.097+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:36.355 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:36.097+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:36.355 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:36.097+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:36.355 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:36.098+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:37.059 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:46:36.946+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 39375 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:46:37.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:36.786+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:37.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:36.786+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:37.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:36.786+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:37.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:36.787+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:37.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:36.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:37.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:36.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:37.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:36.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:37.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:36.885+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:37.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:37.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:37.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:37.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:37.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:37.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:37.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:37.120+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:38.120 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:37.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:38.120 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:37.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:38.120 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:37.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:38.121 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:37.908+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:38.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:37.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:38.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:37.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:38.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:37.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:38.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:37.818+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:38.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:38.131+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:38.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:38.131+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:38.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:38.131+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:38.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:38.132+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:39.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:38.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:39.085 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:38.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:39.085 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:38.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:39.085 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:38.950+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:39.085 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:38.803+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:39.085 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:38.803+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:39.085 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:38.803+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:39.085 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:38.804+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:39.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:39.095+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:39.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:39.095+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:39.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:39.095+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:39.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:39.096+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:40.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:40.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:40.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:40.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:40.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:40.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:40.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:40.053+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:40.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:39.994+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:40.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:39.994+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:40.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:39.994+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:40.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:39.995+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:40.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:39.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:40.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:39.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:40.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:39.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:40.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:39.847+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:41.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:41.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:41.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:41.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:41.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:41.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:41.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:41.005+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:41.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:40.980+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:41.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:40.980+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:41.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:40.980+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:41.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:40.980+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:41.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:40.881+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:41.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:40.881+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:41.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:40.881+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:41.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:40.882+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:42.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:46:41.951+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 39465 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:46:42.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:41.995+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:42.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:41.995+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:42.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:41.995+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:42.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:41.996+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:42.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:41.838+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:42.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:41.838+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:42.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:41.838+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:42.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:41.839+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:42.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:41.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:42.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:41.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:42.211 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:41.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:42.211 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:41.980+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:43.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:43.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:43.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:43.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:43.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:43.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:43.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:43.021+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:43.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:42.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:43.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:42.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:43.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:42.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:43.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:42.834+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:43.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:42.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:43.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:42.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:43.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:42.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:43.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:42.953+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:43.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:43.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:43.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:43.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:43.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:43.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:43.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:43.850+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:44.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:43.932+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:44.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:43.932+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:44.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:43.932+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:44.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:43.933+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:44.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:43.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:44.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:43.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:44.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:43.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:44.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:43.974+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:45.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:44.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:45.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:44.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:45.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:44.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:45.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:44.894+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:45.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:44.924+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:45.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:44.924+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:45.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:44.924+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:45.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:44.925+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:45.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:44.955+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:45.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:44.955+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:45.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:44.955+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:45.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:44.956+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:45.910 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:45.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:45.910 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:45.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:45.910 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:45.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:45.910 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:45.861+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:46.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:45.921+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:46.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:45.921+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:46.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:45.921+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:46.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:45.922+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:46.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:45.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:46.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:45.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:46.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:45.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:46.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:45.973+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:46.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:46:46.628+0000 7f770c428700 0 monclient(hunting): authenticate timed out after 300 2024-04-27T02:46:47.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:46:46.957+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 39556 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:46:47.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:46.933+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:47.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:46.933+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:47.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:46.933+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:47.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:46.933+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:47.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:47.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:47.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:47.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:47.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:47.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:47.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:47.020+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:47.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:46.899+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:47.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:46.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:47.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:46.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:47.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:46.900+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:48.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:47.949+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:48.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:47.949+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:48.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:47.949+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:48.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:47.949+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:48.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:47.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:48.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:47.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:48.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:47.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:48.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:47.901+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:48.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:48.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:48.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:48.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:48.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:48.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:48.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:48.014+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:49.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:48.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:49.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:48.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:49.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:48.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:49.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:48.937+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:49.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:48.999+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:49.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:48.999+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:49.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:48.999+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:49.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:49.000+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:49.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:48.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:49.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:48.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:49.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:48.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:49.211 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:48.977+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:50.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:49.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:50.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:49.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:50.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:49.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:50.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:49.999+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:50.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:49.907+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:50.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:49.907+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:50.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:49.907+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:50.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:49.907+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:50.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:50.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:50.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:50.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:50.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:50.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:50.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:50.011+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:51.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:51.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:51.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:51.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:51.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:51.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:51.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:51.012+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:51.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:50.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:51.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:50.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:51.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:50.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:51.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:50.913+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:51.212 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:51.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:51.212 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:51.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:51.212 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:51.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:51.212 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:51.010+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:52.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:46:51.961+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 39645 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:46:52.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:52.043+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:52.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:52.043+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:52.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:52.043+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:52.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:52.044+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:52.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:51.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:52.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:51.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:52.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:51.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:52.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:51.885+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:52.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:52.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:52.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:52.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:52.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:52.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:52.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:52.017+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:53.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:53.017+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:53.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:53.017+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:53.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:53.017+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:53.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:53.018+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:53.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:52.860+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:53.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:52.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:53.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:52.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:53.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:52.861+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:53.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:53.034+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:53.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:53.034+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:53.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:53.034+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:53.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:53.034+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:53.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:53.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:53.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:53.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:53.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:53.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:53.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:53.833+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:54.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:54.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:54.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:54.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:54.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:54.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:54.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:54.082+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:54.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:54.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:54.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:54.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:54.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:54.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:54.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:54.066+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:54.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:54.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:54.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:54.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:54.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:54.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:54.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:54.850+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:55.459 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:55.114+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:55.459 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:55.114+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:55.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:55.114+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:55.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:55.114+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:55.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:55.095+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:55.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:55.095+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:55.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:55.095+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:55.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:55.096+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:56.124 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:55.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:56.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:55.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:56.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:55.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:56.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:55.820+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:56.125 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:56.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:56.125 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:56.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:56.125 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:56.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:56.125 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:56.066+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:56.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:56.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:56.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:56.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:56.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:56.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:56.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:56.136+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:57.040 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:46:56.966+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 39737 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:46:57.040 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:56.797+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:57.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:56.797+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:57.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:56.797+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:57.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:56.797+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:57.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:57.167+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:57.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:57.167+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:57.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:57.167+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:57.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:57.168+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:57.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:57.051+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:57.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:57.051+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:57.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:57.051+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:57.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:57.051+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:58.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:57.808+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:58.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:57.808+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:58.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:57.809+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:58.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:57.809+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:58.459 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:58.063+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:58.459 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:58.063+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:58.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:58.063+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:58.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:58.064+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:58.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:58.122+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:58.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:58.122+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:58.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:58.122+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:58.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:58.123+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:59.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:59.019+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:59.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:59.019+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:59.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:59.019+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:46:59.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:46:59.020+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:59.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:58.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:59.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:58.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:59.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:58.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:59.148 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:58.847+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:59.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:59.157+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:59.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:59.157+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:59.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:59.157+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:46:59.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:46:59.158+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:46:59.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:59.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:59.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:59.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:59.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:59.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:46:59.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:46:59.862+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:00.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:00.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:00.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:00.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:00.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:00.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:00.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:00.110+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:00.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:00.018+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:00.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:00.018+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:00.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:00.018+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:00.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:00.018+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:01.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:00.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:01.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:00.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:01.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:00.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:01.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:00.815+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:01.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:00.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:01.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:00.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:01.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:00.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:01.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:00.980+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:01.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:01.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:01.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:01.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:01.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:01.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:01.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:01.149+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:02.085 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:47:01.972+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 39826 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:47:02.085 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:01.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:02.085 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:01.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:02.085 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:01.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:02.085 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:01.843+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:02.086 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:02.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:02.086 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:02.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:02.086 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:02.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:02.086 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:02.001+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:02.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:02.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:02.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:02.185+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:02.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:02.185+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:02.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:02.186+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:03.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:03.033+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:03.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:03.033+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:03.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:03.033+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:03.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:03.034+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:03.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:02.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:03.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:02.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:03.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:02.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:03.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:02.848+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:03.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:03.230+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:03.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:03.230+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:03.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:03.230+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:03.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:03.232+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:03.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:03.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:03.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:03.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:03.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:03.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:03.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:03.829+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:04.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:04.006+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:04.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:04.006+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:04.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:04.006+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:04.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:04.006+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:04.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:04.246+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:04.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:04.246+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:04.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:04.246+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:04.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:04.248+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:04.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:04.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:04.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:04.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:04.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:04.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:04.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:04.843+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:05.459 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:05.038+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:05.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:05.038+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:05.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:05.038+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:05.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:05.039+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:05.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:05.273+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:05.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:05.273+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:05.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:05.273+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:05.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:05.273+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:06.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:05.803+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:06.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:05.803+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:06.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:05.803+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:06.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:05.803+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:06.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:06.250+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:06.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:06.250+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:06.358 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:06.250+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:06.358 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:06.250+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:06.358 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:06.067+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:06.358 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:06.067+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:06.358 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:06.067+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:06.358 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:06.068+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:07.051 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:47:06.976+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 39917 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:47:07.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:06.766+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:07.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:06.766+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:07.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:06.766+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:07.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:06.767+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:07.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:07.234+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:07.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:07.234+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:07.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:07.234+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:07.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:07.234+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:07.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:07.061+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:07.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:07.061+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:07.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:07.061+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:07.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:07.062+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:08.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:07.725+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:08.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:07.725+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:08.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:07.725+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:08.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:07.725+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:08.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:08.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:08.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:08.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:08.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:08.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:08.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:08.253+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:08.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:08.070+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:08.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:08.070+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:08.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:08.070+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:08.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:08.071+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:09.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:08.745+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:09.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:08.745+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:09.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:08.745+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:09.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:08.745+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:09.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:09.214+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:09.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:09.214+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:09.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:09.214+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:09.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:09.214+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:09.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:09.074+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:09.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:09.074+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:09.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:09.074+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:09.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:09.075+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:10.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:09.761+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:10.106 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:09.761+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:10.106 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:09.761+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:10.106 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:09.762+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:10.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:10.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:10.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:10.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:10.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:10.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:10.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:10.213+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:10.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:10.115+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:10.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:10.115+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:10.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:10.115+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:10.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:10.116+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:11.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:10.721+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:11.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:10.721+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:11.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:10.721+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:11.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:10.722+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:11.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:11.250+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:11.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:11.250+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:11.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:11.250+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:11.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:11.251+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:11.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:11.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:11.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:11.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:11.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:11.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:11.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:11.157+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:11.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:11.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:11.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:11.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:11.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:11.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:11.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:11.724+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:12.257 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:47:11.981+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 40006 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:47:12.258 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:12.203+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:12.258 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:12.203+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:12.258 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:12.203+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:12.258 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:12.203+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:12.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:12.267+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:12.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:12.267+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:12.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:12.267+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:12.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:12.268+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:13.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:12.750+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:13.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:12.750+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:13.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:12.750+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:13.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:12.751+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:13.666 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:13.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:13.666 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:13.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:13.666 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:13.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:13.666 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:13.253+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:13.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:13.243+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:13.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:13.243+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:13.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:13.243+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:13.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:13.245+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:13.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:13.769+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:13.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:13.769+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:13.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:13.769+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:13.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:13.769+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:14.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:14.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:14.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:14.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:14.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:14.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:14.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:14.212+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:14.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:14.259+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:14.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:14.259+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:14.461 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:14.259+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:14.461 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:14.260+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:15.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:14.727+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:15.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:14.727+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:15.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:14.727+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:15.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:14.728+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:15.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:15.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:15.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:15.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:15.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:15.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:15.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:15.262+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:15.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:15.258+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:15.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:15.258+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:15.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:15.258+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:15.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:15.259+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:16.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:15.741+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:16.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:15.741+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:16.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:15.741+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:16.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:15.741+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:16.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:16.271+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:16.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:16.271+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:16.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:16.271+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:16.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:16.272+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:16.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:16.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:16.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:16.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:16.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:16.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:16.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:16.293+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:16.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:16.717+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:16.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:16.717+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:16.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:16.717+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:16.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:16.717+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:17.275 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:47:16.986+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 40098 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:47:17.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:17.284+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:17.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:17.284+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:17.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:17.284+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:17.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:17.285+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:17.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:17.333+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:17.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:17.333+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:17.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:17.333+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:17.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:17.333+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:17.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:17.717+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:17.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:17.717+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:17.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:17.717+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:17.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:17.717+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:18.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:18.317+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:18.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:18.317+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:18.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:18.317+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:18.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:18.318+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:18.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:18.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:18.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:18.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:18.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:18.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:18.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:18.364+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:18.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:18.692+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:18.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:18.692+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:18.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:18.692+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:18.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:18.692+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:19.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:19.380+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:19.686 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:19.380+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:19.686 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:19.380+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:19.686 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:19.381+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:19.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:19.342+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:19.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:19.342+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:19.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:19.342+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:19.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:19.343+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:19.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:19.694+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:19.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:19.694+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:19.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:19.694+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:19.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:19.694+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:20.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:20.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:20.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:20.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:20.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:20.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:20.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:20.403+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:20.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:20.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:20.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:20.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:20.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:20.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:20.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:20.333+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:20.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:20.703+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:20.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:20.703+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:20.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:20.703+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:20.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:20.704+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:21.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:21.343+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:21.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:21.343+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:21.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:21.343+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:21.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:21.343+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:21.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:21.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:21.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:21.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:21.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:21.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:21.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:21.399+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:21.994 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:47:21.991+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 40187 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:47:21.995 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:21.752+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:21.995 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:21.752+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:21.995 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:21.752+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:21.995 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:21.752+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:22.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:22.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:22.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:22.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:22.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:22.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:22.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:22.364+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:22.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:22.366+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:22.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:22.366+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:22.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:22.366+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:22.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:22.368+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:23.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:22.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:23.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:22.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:23.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:22.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:23.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:22.730+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:23.666 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:23.335+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:23.666 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:23.335+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:23.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:23.335+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:23.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:23.336+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:23.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:23.338+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:23.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:23.338+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:23.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:23.338+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:23.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:23.338+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:23.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:23.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:23.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:23.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:23.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:23.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:23.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:23.724+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:24.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:24.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:24.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:24.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:24.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:24.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:24.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:24.385+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:24.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:24.298+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:24.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:24.299+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:24.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:24.299+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:24.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:24.299+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:25.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:24.764+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:25.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:24.764+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:25.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:24.764+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:25.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:24.764+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:25.556 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:25.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:25.556 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:25.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:25.556 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:25.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:25.556 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:25.314+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:25.557 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:25.423+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:25.557 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:25.423+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:25.557 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:25.423+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:25.557 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:25.424+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:25.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:25.763+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:25.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:25.763+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:25.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:25.763+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:25.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:25.763+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:26.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:26.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:26.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:26.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:26.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:26.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:26.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:26.376+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:26.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:26.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:26.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:26.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:26.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:26.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:26.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:26.281+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:26.988 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:26.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:26.988 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:26.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:26.988 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:26.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:26.989 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:26.729+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:27.246 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:47:26.997+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 40278 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:47:27.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:27.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:27.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:27.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:27.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:27.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:27.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:27.256+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:27.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:27.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:27.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:27.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:27.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:27.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:27.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:27.386+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:28.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:27.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:28.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:27.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:28.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:27.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:28.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:27.751+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:28.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:28.396+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:28.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:28.396+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:28.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:28.396+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:28.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:28.398+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:28.703 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:28.301+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:28.703 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:28.301+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:28.703 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:28.301+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:28.703 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:28.302+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:28.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:28.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:28.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:28.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:28.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:28.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:28.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:28.712+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:29.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:29.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:29.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:29.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:29.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:29.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:29.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:29.364+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:29.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:29.277+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:29.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:29.277+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:29.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:29.277+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:29.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:29.278+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:29.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:29.696+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:29.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:29.696+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:29.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:29.696+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:29.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:29.697+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:30.692 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:30.360+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:30.692 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:30.360+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:30.692 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:30.360+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:30.692 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:30.361+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:30.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:30.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:30.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:30.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:30.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:30.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:30.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:30.298+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:30.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:30.701+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:30.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:30.701+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:30.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:30.701+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:30.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:30.702+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:31.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:31.394+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:31.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:31.394+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:31.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:31.394+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:31.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:31.394+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:31.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:31.306+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:31.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:31.306+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:31.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:31.306+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:31.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:31.307+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:31.993 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:31.746+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:31.993 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:31.746+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:31.993 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:31.746+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:31.993 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:31.746+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:32.320 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:47:32.002+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 40367 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:47:32.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:32.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:32.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:32.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:32.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:32.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:32.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:32.330+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:32.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:32.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:32.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:32.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:32.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:32.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:32.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:32.437+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:33.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:32.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:33.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:32.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:33.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:32.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:33.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:32.735+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:33.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:33.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:33.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:33.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:33.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:33.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:33.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:33.480+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:33.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:33.353+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:33.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:33.353+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:33.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:33.353+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:33.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:33.354+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:33.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:33.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:33.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:33.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:33.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:33.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:33.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:33.786+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:34.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:34.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:34.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:34.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:34.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:34.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:34.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:34.480+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:34.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:34.334+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:34.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:34.334+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:34.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:34.334+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:34.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:34.335+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:35.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:34.758+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:35.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:34.758+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:35.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:34.758+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:35.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:34.759+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:35.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:35.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:35.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:35.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:35.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:35.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:35.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:35.483+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:35.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:35.368+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:35.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:35.368+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:35.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:35.368+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:35.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:35.369+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:36.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:35.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:36.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:35.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:36.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:35.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:36.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:35.745+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:36.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:36.361+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:36.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:36.361+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:36.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:36.361+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:36.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:36.361+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:36.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:36.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:36.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:36.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:36.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:36.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:36.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:36.435+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:37.020 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:47:37.007+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 40459 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:47:37.020 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:36.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:37.020 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:36.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:37.020 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:36.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:37.020 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:36.778+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:37.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:37.408+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:37.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:37.408+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:37.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:37.408+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:37.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:37.409+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:37.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:37.416+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:37.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:37.416+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:37.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:37.416+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:37.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:37.417+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:38.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:37.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:38.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:37.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:38.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:37.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:38.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:37.744+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:38.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:38.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:38.692 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:38.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:38.692 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:38.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:38.692 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:38.415+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:38.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:38.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:38.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:38.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:38.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:38.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:38.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:38.393+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:38.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:38.701+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:38.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:38.701+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:38.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:38.701+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:38.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:38.701+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:39.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:39.370+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:39.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:39.370+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:39.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:39.370+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:39.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:39.371+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:39.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:39.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:39.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:39.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:39.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:39.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:39.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:39.419+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:39.991 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:39.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:39.991 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:39.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:39.992 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:39.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:39.992 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:39.742+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:40.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:40.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:40.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:40.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:40.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:40.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:40.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:40.446+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:40.702 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:40.413+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:40.702 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:40.413+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:40.702 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:40.413+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:40.702 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:40.413+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:40.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:40.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:40.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:40.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:40.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:40.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:40.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:40.710+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:41.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:41.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:41.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:41.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:41.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:41.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:41.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:41.421+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:41.665 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:41.424+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:41.665 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:41.424+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:41.666 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:41.424+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:41.666 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:41.424+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:41.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:41.674+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:41.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:41.674+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:41.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:41.674+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:41.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:41.675+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:42.379 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:47:42.012+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 40548 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:47:42.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:42.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:42.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:42.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:42.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:42.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:42.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:42.389+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:42.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:42.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:42.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:42.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:42.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:42.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:42.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:42.436+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:42.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:42.630+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:42.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:42.630+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:42.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:42.630+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:42.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:42.631+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:43.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:43.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:43.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:43.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:43.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:43.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:43.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:43.455+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:43.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:43.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:43.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:43.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:43.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:43.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:43.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:43.435+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:43.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:43.604+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:43.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:43.604+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:43.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:43.604+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:43.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:43.604+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:44.634 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:44.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:44.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:44.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:44.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:44.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:44.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:44.461+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:44.635 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:44.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:44.635 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:44.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:44.635 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:44.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:44.635 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:44.388+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:44.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:44.643+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:44.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:44.643+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:44.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:44.643+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:44.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:44.644+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:45.674 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:45.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:45.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:45.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:45.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:45.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:45.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:45.460+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:45.675 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:45.372+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:45.675 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:45.372+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:45.675 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:45.372+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:45.675 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:45.373+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:45.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:45.683+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:45.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:45.683+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:45.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:45.683+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:45.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:45.684+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:46.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:46.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:46.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:46.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:46.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:46.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:46.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:46.398+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:46.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:46.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:46.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:46.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:46.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:46.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:46.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:46.494+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:47.005 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:46.731+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:47.005 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:46.731+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:47.005 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:46.731+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:47.005 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:46.731+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:47.378 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:47:47.017+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 40639 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:47:47.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:47.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:47.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:47.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:47.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:47.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:47.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:47.506+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:47.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:47.387+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:47.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:47.387+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:47.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:47.387+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:47.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:47.388+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:48.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:47.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:48.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:47.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:48.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:47.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:48.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:47.724+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:48.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:48.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:48.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:48.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:48.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:48.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:48.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:48.530+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:48.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:48.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:48.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:48.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:48.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:48.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:48.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:48.401+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:49.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:48.745+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:49.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:48.745+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:49.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:48.745+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:49.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:48.746+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:49.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:49.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:49.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:49.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:49.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:49.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:49.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:49.513+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:49.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:49.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:49.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:49.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:49.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:49.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:49.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:49.412+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:50.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:49.745+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:50.094 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:49.745+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:50.094 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:49.745+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:50.094 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:49.745+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:50.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:50.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:50.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:50.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:50.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:50.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:50.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:50.473+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:50.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:50.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:50.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:50.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:50.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:50.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:50.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:50.422+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:50.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:50.702+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:50.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:50.702+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:50.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:50.702+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:50.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:50.702+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:51.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:51.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:51.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:51.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:51.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:51.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:51.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:51.491+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:51.660 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:51.413+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:51.660 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:51.413+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:51.660 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:51.413+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:51.660 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:51.413+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:51.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:51.667+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:51.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:51.668+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:51.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:51.668+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:51.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:51.668+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:52.414 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:47:52.023+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 40729 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:47:52.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:52.649+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:52.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:52.649+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:52.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:52.649+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:52.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:52.650+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:52.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:52.515+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:52.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:52.515+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:52.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:52.515+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:52.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:52.516+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:52.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:52.422+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:52.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:52.422+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:52.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:52.422+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:52.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:52.423+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:53.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:53.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:53.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:53.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:53.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:53.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:53.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:53.398+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:53.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:53.695+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:53.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:53.695+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:53.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:53.695+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:53.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:53.695+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:53.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:53.531+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:53.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:53.531+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:53.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:53.531+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:53.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:53.532+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:54.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:54.681+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:54.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:54.681+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:54.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:54.681+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:54.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:54.682+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:54.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:54.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:54.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:54.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:54.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:54.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:54.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:54.575+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:54.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:54.447+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:54.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:54.447+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:54.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:54.447+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:54.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:54.448+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:55.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:55.640+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:55.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:55.640+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:55.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:55.640+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:55.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:55.640+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:55.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:55.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:55.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:55.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:55.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:55.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:55.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:55.610+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:55.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:55.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:55.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:55.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:55.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:55.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:55.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:55.462+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:56.560 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:56.479+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:56.560 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:56.479+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:56.560 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:56.479+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:56.560 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:56.479+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:56.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:56.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:56.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:56.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:56.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:56.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:56.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:56.570+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:56.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:56.673+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:56.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:56.673+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:56.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:56.673+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:56.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:56.674+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:57.461 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:47:57.027+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 40821 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:47:57.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:57.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:57.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:57.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:57.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:57.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:57.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:57.670+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:57.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:57.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:57.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:57.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:57.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:57.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:57.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:57.491+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:57.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:57.577+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:57.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:57.577+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:57.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:57.577+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:57.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:57.579+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:58.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:58.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:58.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:58.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:58.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:58.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:58.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:58.499+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:58.902 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:58.640+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:58.902 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:58.640+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:58.902 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:58.640+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:58.902 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:58.640+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:58.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:58.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:58.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:58.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:58.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:58.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:58.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:58.535+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:59.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:59.609+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:59.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:59.609+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:59.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:59.609+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:47:59.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:47:59.609+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:59.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:59.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:59.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:59.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:59.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:59.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:47:59.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:47:59.538+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:47:59.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:59.504+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:59.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:59.504+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:59.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:59.504+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:47:59.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:47:59.505+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:00.961 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:00.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:00.961 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:00.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:00.961 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:00.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:00.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:00.487+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:00.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:00.576+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:00.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:00.576+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:00.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:00.576+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:00.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:00.577+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:00.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:00.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:00.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:00.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:00.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:00.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:00.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:00.495+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:01.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:01.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:01.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:01.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:01.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:01.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:01.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:01.448+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:01.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:01.484+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:01.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:01.484+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:01.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:01.484+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:01.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:01.485+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:01.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:01.598+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:01.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:01.598+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:01.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:01.598+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:01.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:01.598+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:02.394 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:48:02.033+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 40910 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:48:02.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:02.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:02.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:02.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:02.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:02.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:02.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:02.403+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:02.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:02.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:02.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:02.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:02.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:02.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:02.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:02.461+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:02.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:02.646+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:02.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:02.646+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:02.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:02.646+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:02.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:02.647+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:03.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:03.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:03.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:03.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:03.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:03.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:03.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:03.358+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:03.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:03.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:03.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:03.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:03.672 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:03.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:03.672 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:03.421+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:03.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:03.689+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:03.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:03.689+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:03.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:03.689+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:03.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:03.690+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:04.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:04.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:04.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:04.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:04.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:04.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:04.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:04.322+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:04.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:04.376+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:04.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:04.376+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:04.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:04.376+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:04.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:04.376+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:04.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:04.689+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:04.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:04.689+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:04.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:04.689+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:04.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:04.689+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:05.461 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:05.315+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:05.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:05.315+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:05.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:05.315+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:05.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:05.316+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:05.462 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:05.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:05.462 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:05.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:05.462 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:05.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:05.462 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:05.426+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:05.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:05.713+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:05.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:05.713+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:05.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:05.713+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:05.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:05.714+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:06.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:06.267+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:06.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:06.267+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:06.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:06.267+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:06.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:06.268+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:06.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:06.432+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:06.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:06.432+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:06.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:06.432+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:06.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:06.433+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:07.029 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:06.747+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:07.029 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:06.747+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:07.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:06.747+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:07.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:06.748+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:07.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:48:07.037+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 41001 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:48:07.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:07.224+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:07.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:07.224+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:07.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:07.224+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:07.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:07.225+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:07.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:07.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:07.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:07.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:07.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:07.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:07.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:07.467+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:08.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:07.722+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:08.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:07.722+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:08.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:07.722+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:08.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:07.722+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:08.433 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:08.185+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:08.433 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:08.185+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:08.433 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:08.185+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:08.433 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:08.186+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:08.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:08.689+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:08.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:08.689+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:08.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:08.689+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:08.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:08.690+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:08.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:08.441+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:08.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:08.441+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:08.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:08.441+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:08.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:08.442+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:09.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:09.183+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:09.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:09.183+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:09.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:09.183+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:09.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:09.184+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:09.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:09.648+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:09.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:09.648+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:09.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:09.648+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:09.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:09.649+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:09.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:09.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:09.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:09.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:09.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:09.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:09.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:09.440+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:10.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:10.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:10.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:10.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:10.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:10.204+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:10.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:10.205+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:10.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:10.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:10.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:10.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:10.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:10.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:10.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:10.458+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:10.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:10.680+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:10.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:10.680+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:10.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:10.680+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:10.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:10.680+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:11.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:11.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:11.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:11.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:11.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:11.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:11.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:11.156+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:11.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:11.713+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:11.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:11.713+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:11.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:11.713+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:11.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:11.713+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:11.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:11.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:11.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:11.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:11.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:11.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:11.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:11.501+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:12.461 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:48:12.042+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 41091 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:48:12.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:12.203+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:12.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:12.203+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:12.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:12.203+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:12.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:12.205+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:12.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:12.681+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:12.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:12.681+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:12.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:12.681+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:12.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:12.681+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:12.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:12.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:12.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:12.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:12.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:12.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:12.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:12.538+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:13.543 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:13.225+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:13.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:13.225+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:13.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:13.225+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:13.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:13.225+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:13.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:13.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:13.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:13.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:13.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:13.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:13.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:13.552+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:13.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:13.698+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:13.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:13.698+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:13.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:13.698+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:13.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:13.698+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:14.513 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:14.259+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:14.513 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:14.259+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:14.513 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:14.259+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:14.513 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:14.261+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:14.773 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:14.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:14.773 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:14.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:14.774 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:14.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:14.774 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:14.521+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:14.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:14.746+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:14.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:14.746+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:14.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:14.746+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:14.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:14.746+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:15.535 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:15.241+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:15.535 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:15.241+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:15.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:15.241+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:15.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:15.242+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:15.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:15.542+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:15.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:15.542+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:15.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:15.542+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:15.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:15.543+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:15.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:15.757+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:15.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:15.757+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:15.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:15.757+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:15.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:15.757+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:16.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:16.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:16.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:16.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:16.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:16.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:16.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:16.281+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:16.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:16.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:16.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:16.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:16.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:16.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:16.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:16.591+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:16.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:16.791+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:16.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:16.791+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:16.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:16.791+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:16.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:16.791+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:17.462 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:48:17.047+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 41182 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:48:17.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:17.238+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:17.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:17.238+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:17.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:17.238+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:17.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:17.239+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:17.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:17.584+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:17.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:17.584+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:17.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:17.584+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:17.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:17.585+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:17.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:17.750+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:17.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:17.750+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:17.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:17.750+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:17.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:17.751+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:18.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:18.254+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:18.577 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:18.254+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:18.577 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:18.254+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:18.577 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:18.255+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:18.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:18.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:18.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:18.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:18.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:18.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:18.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:18.584+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:18.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:18.715+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:18.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:18.715+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:18.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:18.715+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:18.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:18.715+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:19.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:19.209+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:19.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:19.209+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:19.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:19.209+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:19.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:19.209+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:19.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:19.552+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:19.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:19.552+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:19.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:19.552+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:19.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:19.553+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:19.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:19.686+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:19.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:19.686+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:19.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:19.686+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:19.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:19.687+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:20.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:20.232+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:20.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:20.232+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:20.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:20.232+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:20.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:20.233+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:20.855 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:20.650+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:20.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:20.650+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:20.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:20.650+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:20.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:20.651+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:20.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:20.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:20.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:20.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:20.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:20.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:20.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:20.590+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:21.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:21.206+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:21.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:21.206+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:21.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:21.206+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:21.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:21.206+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:21.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:21.662+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:21.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:21.662+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:21.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:21.662+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:21.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:21.662+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:21.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:21.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:21.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:21.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:21.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:21.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:21.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:21.545+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:22.462 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:48:22.052+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 41271 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:48:22.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:22.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:22.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:22.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:22.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:22.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:22.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:22.194+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:22.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:22.685+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:22.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:22.685+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:22.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:22.685+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:22.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:22.685+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:22.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:22.509+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:22.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:22.509+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:22.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:22.509+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:22.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:22.509+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:23.521 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:23.245+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:23.522 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:23.245+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:23.522 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:23.245+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:23.522 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:23.245+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:23.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:23.528+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:23.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:23.528+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:23.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:23.528+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:23.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:23.529+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:23.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:23.674+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:23.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:23.674+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:23.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:23.674+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:23.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:23.674+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:24.527 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:24.268+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:24.527 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:24.268+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:24.527 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:24.268+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:24.527 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:24.269+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:24.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:24.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:24.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:24.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:24.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:24.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:24.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:24.535+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:24.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:24.630+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:24.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:24.630+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:24.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:24.630+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:24.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:24.631+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:25.560 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:25.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:25.560 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:25.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:25.560 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:25.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:25.560 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:25.258+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:25.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:25.587+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:25.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:25.587+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:25.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:25.587+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:25.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:25.587+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:25.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:25.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:25.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:25.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:25.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:25.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:25.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:25.568+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:26.537 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:26.269+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:26.537 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:26.269+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:26.537 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:26.269+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:26.537 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:26.270+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:26.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:26.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:26.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:26.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:26.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:26.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:26.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:26.569+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:26.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:26.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:26.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:26.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:26.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:26.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:26.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:26.546+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:27.279 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:48:27.058+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 41362 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:48:27.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:27.287+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:27.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:27.287+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:27.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:27.287+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:27.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:27.288+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:27.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:27.588+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:27.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:27.588+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:27.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:27.588+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:37:58.780327+0000 (oldest deadline 2024-04-27T02:38:18.780327+0000) 2024-04-27T02:48:27.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:27.589+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:27.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:27.605+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:27.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:27.605+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:27.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:27.605+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:27.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:27.605+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:28.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:28.259+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:28.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:28.259+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:28.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:28.259+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:28.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:28.259+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:28.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:28.569+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:28.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:28.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:28.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:28.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:28.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:28.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:28.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:28.575+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:29.530 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:29.234+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:29.530 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:29.234+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:29.530 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:29.234+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:29.530 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:29.235+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:29.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:29.546+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:29.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:29.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:29.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:29.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:29.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:29.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:29.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:29.538+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:30.398 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:30.213+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:30.398 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:30.213+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:30.398 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:30.213+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:30.398 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:30.214+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:30.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:30.551+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:30.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:30.524+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:30.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:30.524+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:30.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:30.524+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:30.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:30.524+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:31.499 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:31.255+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:31.499 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:31.255+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:31.499 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:31.255+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:31.499 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:31.256+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:31.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:31.524+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:31.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:31.506+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:31.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:31.506+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:31.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:31.506+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:31.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:31.507+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:32.462 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:48:32.063+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 41451 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:48:32.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:32.213+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:32.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:32.213+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:32.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:32.213+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:32.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:32.214+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:32.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:32.545+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:32.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:32.547+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:32.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:32.547+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:32.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:32.547+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:32.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:32.547+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:33.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:33.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:33.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:33.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:33.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:33.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:33.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:33.242+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:33.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:33.540+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:33.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:33.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:33.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:33.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:33.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:33.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:02.426125+0000 (oldest deadline 2024-04-27T02:38:22.426125+0000) 2024-04-27T02:48:33.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:33.527+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:34.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:34.213+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:34.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:34.213+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:34.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:34.213+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:34.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:34.214+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:34.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:34.521+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:34.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:34.563+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:35.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:35.178+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:35.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:35.178+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:35.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:35.178+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:35.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:35.179+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:35.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:35.516+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:35.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:35.541+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:36.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:36.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:36.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:36.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:36.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:36.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:36.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:36.162+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:36.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:36.552+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:36.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:36.586+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:37.462 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:48:37.068+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 41542 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:48:37.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:37.195+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:37.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:37.195+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:37.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:37.195+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:37.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:37.196+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:37.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:37.514+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:37.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:37.608+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:38.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:38.166+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:38.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:38.166+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:38.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:38.166+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:38.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:38.166+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:38.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:38.604+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:38.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:38.564+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:39.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:39.168+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:39.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:39.168+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:39.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:39.169+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:38:08.376957+0000 (oldest deadline 2024-04-27T02:38:28.376957+0000) 2024-04-27T02:48:39.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:39.170+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:39.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:39.515+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:39.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:39.588+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:40.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:40.143+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:40.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:40.534+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:40.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:40.586+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:41.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:41.136+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:41.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:41.504+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:41.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:41.540+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:42.359 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:48:42.074+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 41631 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:48:42.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:42.179+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:42.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:42.545+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:42.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:42.586+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:43.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:43.171+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:43.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:43.572+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:43.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:43.567+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:44.598 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:44.221+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:44.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:44.606+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:44.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:44.611+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:45.430 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:45.178+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:45.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:45.599+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:45.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:45.615+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:46.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:46.136+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:46.624 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:46.615+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:46.965 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:46.632+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:47.462 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:48:47.079+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 41722 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:48:47.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:47.146+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:47.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:47.649+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:47.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:47.599+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:48.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:48.190+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:48.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:48.615+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:48.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:48.596+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:49.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:49.181+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:49.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:49.643+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:49.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:49.581+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:50.533 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:50.228+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:50.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:50.634+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:50.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:50.540+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:51.488 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:51.228+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:51.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:51.495+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:51.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:51.611+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:51.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:51.611+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:51.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:51.611+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:51.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:51.612+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:52.463 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:48:52.084+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 41812 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:48:52.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:52.225+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:52.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:52.474+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:52.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:52.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:52.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:52.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:52.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:52.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:52.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:52.607+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:53.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:53.211+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:53.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:53.435+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:53.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:53.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:53.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:53.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:53.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:53.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:53.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:53.619+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:54.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:54.215+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:54.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:54.449+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:54.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:54.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:54.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:54.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:54.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:54.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:54.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:54.635+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:55.595 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:55.231+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:55.595 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:55.458+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:55.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:55.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:55.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:55.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:55.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:55.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:55.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:55.632+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:56.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:56.251+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:56.655 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:56.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:48:56.655 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:56.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:48:56.655 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:56.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:48:56.655 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:56.447+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:56.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:56.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:56.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:56.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:56.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:56.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:56.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:56.662+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:57.394 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:48:57.089+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 41904 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:48:57.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:57.230+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:57.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:57.401+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:48:57.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:57.401+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:48:57.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:57.401+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:48:57.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:57.401+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:57.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:57.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:57.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:57.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:57.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:57.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:57.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:57.625+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:58.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:58.193+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:58.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:58.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:48:58.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:58.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:48:58.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:58.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:48:58.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:58.364+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:58.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:58.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:58.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:58.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:58.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:58.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:58.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:58.581+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:59.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:48:59.177+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:59.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:59.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:48:59.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:59.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:48:59.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:59.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:48:59.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:48:59.355+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:48:59.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:59.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:59.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:59.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:59.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:59.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:48:59.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:48:59.591+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:00.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:00.206+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:00.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:00.206+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:00.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:00.206+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:00.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:00.207+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:00.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:00.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:00.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:00.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:00.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:00.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:00.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:00.344+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:00.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:00.603+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:00.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:00.603+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:00.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:00.603+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:00.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:00.604+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:01.586 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:01.221+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:01.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:01.221+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:01.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:01.221+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:01.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:01.222+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:01.587 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:01.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:01.587 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:01.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:01.587 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:01.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:01.587 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:01.332+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:01.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:01.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:01.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:01.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:01.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:01.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:01.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:01.594+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:02.366 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:49:02.094+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 41991 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:49:02.366 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:02.174+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:02.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:02.174+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:02.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:02.174+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:02.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:02.174+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:02.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:02.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:02.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:02.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:02.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:02.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:02.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:02.592+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:02.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:02.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:02.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:02.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:02.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:02.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:02.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:02.373+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:03.401 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:03.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:03.401 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:03.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:03.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:03.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:03.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:03.136+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:03.676 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:03.585+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:03.676 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:03.585+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:03.676 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:03.585+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:03.676 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:03.587+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:03.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:03.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:03.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:03.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:03.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:03.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:03.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:03.408+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:04.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:04.121+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:04.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:04.121+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:04.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:04.121+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:04.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:04.122+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:04.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:04.626+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:04.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:04.626+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:04.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:04.626+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:04.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:04.627+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:04.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:04.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:04.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:04.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:04.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:04.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:04.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:04.434+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:05.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:05.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:05.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:05.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:05.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:05.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:05.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:05.085+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:05.682 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:05.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:05.682 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:05.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:05.682 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:05.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:05.682 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:05.390+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:05.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:05.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:05.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:05.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:05.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:05.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:05.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:05.597+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:06.360 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:06.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:06.360 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:06.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:06.360 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:06.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:06.361 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:06.061+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:06.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:06.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:06.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:06.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:06.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:06.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:06.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:06.373+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:06.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:06.553+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:06.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:06.553+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:06.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:06.553+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:06.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:06.554+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:07.213 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:49:07.099+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 42082 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:49:07.213 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:07.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:07.213 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:07.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:07.213 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:07.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:07.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:07.062+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:07.517 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:07.401+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:07.517 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:07.401+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:07.517 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:07.401+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:07.517 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:07.402+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:07.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:07.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:07.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:07.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:07.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:07.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:07.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:07.524+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:08.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:08.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:08.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:08.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:08.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:08.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:08.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:08.041+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:08.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:08.570+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:08.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:08.570+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:08.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:08.570+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:08.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:08.571+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:08.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:08.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:08.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:08.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:08.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:08.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:08.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:08.419+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:09.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:09.077+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:09.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:09.077+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:09.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:09.077+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:09.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:09.079+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:09.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:09.429+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:09.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:09.429+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:09.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:09.429+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:09.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:09.430+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:09.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:09.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:09.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:09.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:09.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:09.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:09.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:09.532+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:10.440 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:10.107+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:10.440 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:10.107+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:10.440 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:10.107+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:10.440 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:10.108+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:10.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:10.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:10.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:10.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:10.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:10.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:10.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:10.545+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:10.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:10.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:10.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:10.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:10.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:10.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:10.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:10.447+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:11.413 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:11.080+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:11.413 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:11.080+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:11.413 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:11.080+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:11.413 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:11.080+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:11.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:11.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:11.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:11.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:11.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:11.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:11.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:11.524+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:11.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:11.417+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:11.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:11.417+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:11.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:11.417+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:11.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:11.418+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:12.433 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:49:12.104+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 42171 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:49:12.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:12.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:12.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:12.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:12.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:12.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:12.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:12.040+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:12.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:12.439+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:12.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:12.439+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:12.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:12.439+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:12.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:12.439+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:12.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:12.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:12.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:12.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:12.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:12.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:12.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:12.507+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:13.422 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:13.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:13.422 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:13.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:13.422 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:13.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:13.422 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:13.048+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:13.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:13.553+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:13.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:13.553+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:13.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:13.553+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:13.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:13.554+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:13.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:13.427+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:13.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:13.427+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:13.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:13.427+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:13.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:13.428+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:14.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:14.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:14.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:14.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:14.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:14.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:14.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:14.053+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:14.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:14.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:14.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:14.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:14.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:14.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:14.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:14.383+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:14.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:14.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:14.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:14.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:14.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:14.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:14.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:14.563+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:15.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:15.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:15.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:15.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:15.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:15.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:15.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:15.040+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:15.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:15.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:15.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:15.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:15.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:15.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:15.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:15.523+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:15.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:15.431+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:15.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:15.431+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:15.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:15.431+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:15.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:15.431+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:16.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:16.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:16.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:16.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:16.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:16.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:16.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:16.010+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:16.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:16.517+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:16.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:16.517+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:16.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:16.517+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:16.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:16.518+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:16.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:16.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:16.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:16.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:16.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:16.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:16.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:16.426+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:17.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:16.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:17.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:16.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:17.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:16.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:17.105 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:16.967+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:17.396 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:49:17.109+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 42262 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:49:17.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:17.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:17.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:17.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:17.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:17.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:17.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:17.564+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:17.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:17.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:17.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:17.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:17.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:17.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:17.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:17.402+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:18.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:17.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:18.358 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:17.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:18.358 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:17.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:18.358 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:17.972+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:18.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:18.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:18.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:18.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:18.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:18.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:18.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:18.528+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:18.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:18.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:18.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:18.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:18.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:18.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:18.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:18.363+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:19.339 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:19.019+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:19.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:19.019+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:19.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:19.019+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:19.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:19.020+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:19.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:19.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:19.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:19.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:19.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:19.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:19.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:19.502+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:19.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:19.345+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:19.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:19.345+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:19.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:19.345+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:19.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:19.345+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:20.331 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:20.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:20.332 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:20.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:20.332 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:20.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:20.332 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:20.053+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:20.332 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:20.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:20.335 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:20.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:20.335 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:20.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:20.335 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:20.337+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:20.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:20.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:20.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:20.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:20.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:20.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:20.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:20.458+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:21.313 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:21.068+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:21.314 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:21.068+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:21.314 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:21.068+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:21.314 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:21.068+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:21.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:21.481+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:21.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:21.481+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:21.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:21.481+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:21.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:21.482+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:21.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:21.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:21.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:21.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:21.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:21.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:21.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:21.319+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:22.347 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:49:22.114+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 42351 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:49:22.348 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:22.084+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:22.348 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:22.084+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:22.348 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:22.084+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:22.348 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:22.085+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:22.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:22.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:22.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:22.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:22.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:22.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:22.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:22.354+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:22.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:22.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:22.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:22.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:22.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:22.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:22.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:22.526+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:23.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:23.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:23.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:23.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:23.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:23.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:23.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:23.130+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:23.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:23.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:23.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:23.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:23.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:23.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:23.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:23.319+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:23.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:23.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:23.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:23.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:23.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:23.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:23.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:23.489+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:24.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:24.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:24.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:24.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:24.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:24.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:24.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:24.114+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:24.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:24.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:24.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:24.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:24.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:24.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:24.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:24.352+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:24.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:24.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:24.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:24.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:24.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:24.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:24.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:24.475+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:25.361 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:25.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:25.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:25.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:25.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:25.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:25.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:25.091+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:25.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:25.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:25.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:25.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:25.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:25.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:25.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:25.456+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:25.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:25.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:25.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:25.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:25.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:25.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:25.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:25.368+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:26.213 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:26.051+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:26.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:26.051+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:26.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:26.051+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:26.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:26.052+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:26.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:26.414+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:26.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:26.414+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:26.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:26.414+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:26.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:26.414+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:26.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:26.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:26.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:26.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:26.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:26.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:26.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:26.414+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:27.380 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:49:27.121+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 42442 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:49:27.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:27.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:27.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:27.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:27.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:27.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:27.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:27.073+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:27.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:27.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:27.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:27.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:27.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:27.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:27.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:27.415+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:27.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:27.386+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:27.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:27.386+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:27.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:27.386+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:27.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:27.387+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:28.371 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:28.106+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:28.371 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:28.106+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:28.372 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:28.106+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:28.372 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:28.107+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:28.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:28.377+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:28.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:28.377+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:28.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:28.377+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:28.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:28.378+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:28.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:28.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:28.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:28.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:28.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:28.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:28.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:28.435+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:29.420 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:29.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:29.420 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:29.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:29.420 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:29.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:29.420 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:29.120+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:29.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:29.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:29.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:29.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:29.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:29.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:29.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:29.427+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:29.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:29.447+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:29.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:29.447+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:29.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:29.447+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:29.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:29.448+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:30.453 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:30.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:30.453 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:30.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:30.453 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:30.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:30.453 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:30.137+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:30.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:30.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:30.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:30.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:30.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:30.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:30.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:30.473+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:30.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:30.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:30.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:30.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:30.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:30.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:30.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:30.459+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:31.213 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:31.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:31.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:31.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:31.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:31.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:31.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:31.111+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:31.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:31.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:31.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:31.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:31.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:31.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:31.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:31.443+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:31.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:31.491+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:31.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:31.491+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:31.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:31.491+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:31.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:31.491+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:32.406 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:49:32.126+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 42532 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:49:32.406 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:32.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:32.406 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:32.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:32.406 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:32.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:32.406 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:32.091+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:32.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:32.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:32.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:32.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:32.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:32.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:32.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:32.443+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:32.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:32.412+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:32.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:32.412+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:32.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:32.412+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:32.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:32.413+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:33.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:33.099+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:33.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:33.099+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:33.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:33.099+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:33.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:33.101+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:33.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:33.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:33.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:33.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:33.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:33.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:33.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:33.481+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:33.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:33.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:33.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:33.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:33.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:33.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:33.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:33.373+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:34.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:34.102+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:34.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:34.102+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:34.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:34.102+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:34.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:34.103+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:34.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:34.495+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:34.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:34.495+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:34.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:34.495+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:34.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:34.495+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:34.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:34.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:34.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:34.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:34.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:34.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:34.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:34.399+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:35.420 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:35.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:35.420 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:35.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:35.420 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:35.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:35.420 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:35.066+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:35.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:35.472+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:35.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:35.472+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:35.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:35.472+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:35.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:35.473+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:35.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:35.426+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:35.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:35.426+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:35.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:35.426+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:35.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:35.427+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:36.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:36.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:36.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:36.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:36.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:36.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:36.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:36.061+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:36.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:36.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:36.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:36.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:36.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:36.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:36.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:36.455+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:36.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:36.439+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:36.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:36.439+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:36.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:36.439+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:36.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:36.439+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:37.407 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:49:37.131+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 42619 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:49:37.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:37.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:37.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:37.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:37.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:37.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:37.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:37.017+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:37.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:37.422+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:37.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:37.422+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:37.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:37.422+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:37.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:37.423+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:37.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:37.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:37.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:37.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:37.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:37.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:37.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:37.414+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:38.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:37.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:38.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:37.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:38.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:37.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:38.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:37.968+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:38.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:38.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:38.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:38.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:38.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:38.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:38.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:38.388+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:38.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:38.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:38.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:38.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:38.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:38.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:38.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:38.413+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:39.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:38.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:39.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:38.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:39.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:38.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:39.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:38.938+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:39.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:39.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:39.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:39.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:39.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:39.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:39.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:39.396+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:39.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:39.453+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:39.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:39.453+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:39.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:39.453+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:39.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:39.453+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:40.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:39.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:40.215 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:39.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:40.215 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:39.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:40.215 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:39.946+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:40.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:40.404+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:40.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:40.404+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:40.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:40.404+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:40.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:40.405+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:40.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:40.485+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:40.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:40.485+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:40.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:40.485+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:40.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:40.485+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:41.000 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:40.965+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:41.000 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:40.965+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:41.001 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:40.965+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:41.001 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:40.966+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:41.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:41.427+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:41.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:41.427+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:41.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:41.427+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:41.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:41.429+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:41.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:41.525+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:41.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:41.525+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:41.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:41.525+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:41.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:41.526+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:42.214 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:49:42.137+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 42708 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:49:42.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:41.965+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:42.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:41.965+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:42.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:41.965+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:42.215 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:41.966+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:42.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:42.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:42.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:42.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:42.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:42.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:42.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:42.402+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:42.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:42.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:42.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:42.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:42.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:42.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:42.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:42.477+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:43.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:42.979+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:43.360 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:42.979+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:43.360 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:42.979+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:43.360 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:42.980+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:43.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:43.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:43.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:43.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:43.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:43.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:43.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:43.494+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:43.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:43.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:43.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:43.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:43.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:43.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:43.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:43.366+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:44.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:44.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:44.339 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:44.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:44.339 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:44.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:44.339 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:44.011+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:44.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:44.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:44.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:44.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:44.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:44.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:44.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:44.518+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:44.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:44.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:44.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:44.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:44.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:44.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:44.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:44.345+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:45.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:45.032+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:45.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:45.032+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:45.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:45.032+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:45.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:45.033+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:45.561 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:45.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:45.561 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:45.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:45.561 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:45.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:45.562 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:45.308+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:45.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:45.567+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:45.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:45.567+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:45.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:45.567+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:45.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:45.567+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:46.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:46.057+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:46.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:46.057+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:46.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:46.057+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:46.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:46.058+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:46.362 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:46.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:46.362 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:46.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:46.362 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:46.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:46.362 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:46.264+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:46.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:46.603+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:46.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:46.603+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:46.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:46.603+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:46.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:46.604+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:47.464 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:49:47.143+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 42799 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:49:47.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:47.303+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:47.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:47.303+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:47.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:47.303+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:47.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:47.304+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:47.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:47.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:47.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:47.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:47.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:47.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:47.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:47.083+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:47.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:47.615+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:47.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:47.615+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:47.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:47.615+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:47.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:47.616+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:48.464 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:48.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:48.464 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:48.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:48.464 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:48.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:48.464 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:48.292+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:48.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:48.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:48.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:48.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:48.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:48.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:48.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:48.091+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:48.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:48.569+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:48.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:48.569+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:48.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:48.569+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:48.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:48.570+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:49.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:49.087+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:49.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:49.087+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:49.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:49.087+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:49.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:49.088+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:49.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:49.328+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:49.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:49.328+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:49.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:49.328+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:49.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:49.329+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:49.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:49.538+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:49.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:49.538+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:49.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:49.538+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:49.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:49.538+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:50.337 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:50.071+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:50.337 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:50.071+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:50.337 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:50.071+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:50.337 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:50.072+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:50.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:50.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:50.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:50.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:50.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:50.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:50.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:50.537+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:50.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:50.342+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:50.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:50.342+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:50.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:50.342+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:50.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:50.343+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:51.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:51.122+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:51.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:51.122+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:51.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:51.122+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:51.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:51.123+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:51.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:51.563+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:51.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:51.563+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:51.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:51.563+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:51.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:51.564+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:51.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:51.357+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:51.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:51.357+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:51.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:51.357+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:51.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:51.359+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:52.394 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:49:52.149+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 42889 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:49:52.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:52.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:52.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:52.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:52.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:52.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:52.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:52.166+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:52.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:52.399+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:52.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:52.399+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:52.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:52.399+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:52.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:52.400+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:52.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:52.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:52.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:52.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:52.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:52.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:52.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:52.540+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:53.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:53.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:53.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:53.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:53.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:53.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:53.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:53.209+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:53.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:53.209+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:53.389 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:53.209+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:53.389 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:53.210+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:53.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:53.394+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:53.680 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:53.572+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:53.680 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:53.572+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:53.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:53.572+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:53.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:53.572+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:54.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:54.217+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:54.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:54.217+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:54.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:54.217+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:54.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:54.218+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:54.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:54.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:54.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:54.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:54.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:54.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:54.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:54.352+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:54.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:54.585+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:54.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:54.585+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:54.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:54.585+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:54.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:54.585+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:55.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:55.257+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:55.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:55.257+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:55.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:55.257+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:55.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:55.258+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:55.617 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:55.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:55.617 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:55.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:55.617 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:55.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:55.617 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:55.310+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:55.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:55.621+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:55.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:55.621+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:55.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:55.621+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:55.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:55.622+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:56.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:56.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:56.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:56.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:56.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:56.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:56.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:56.210+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:56.629 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:56.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:56.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:56.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:56.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:56.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:56.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:56.348+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:56.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:56.635+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:56.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:56.635+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:56.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:56.635+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:56.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:56.635+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:57.464 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:49:57.154+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 42980 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:49:57.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:57.377+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:57.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:57.377+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:57.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:57.377+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:57.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:57.377+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:57.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:57.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:57.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:57.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:57.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:57.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:57.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:57.213+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:57.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:57.675+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:57.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:57.675+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:57.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:57.675+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:57.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:57.676+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:58.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:58.199+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:58.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:58.199+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:58.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:58.199+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:58.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:58.200+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:58.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:58.399+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:58.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:58.399+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:58.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:58.399+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:58.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:58.399+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:58.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:58.675+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:58.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:58.675+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:58.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:58.675+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:58.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:58.675+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:59.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:59.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:59.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:59.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:59.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:59.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:49:59.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:49:59.213+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:59.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:59.377+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:59.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:59.377+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:59.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:59.377+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:49:59.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:49:59.378+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:49:59.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:59.718+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:59.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:59.718+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:59.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:59.718+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:49:59.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:49:59.718+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:00.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:00.230+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:00.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:00.230+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:00.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:00.230+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:00.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:00.231+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:00.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:00.384+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:00.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:00.384+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:00.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:00.384+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:00.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:00.384+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:00.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:00.691+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:00.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:00.691+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:00.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:00.691+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:00.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:00.692+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:01.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:01.271+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:01.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:01.272+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:01.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:01.272+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:01.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:01.272+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:01.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:01.374+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:01.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:01.374+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:01.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:01.374+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:01.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:01.375+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:01.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:01.714+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:01.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:01.714+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:01.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:01.714+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:01.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:01.714+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:02.464 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:50:02.160+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 43069 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:50:02.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:02.385+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:02.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:02.385+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:02.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:02.385+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:02.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:02.386+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:02.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:02.310+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:02.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:02.310+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:02.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:02.310+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:02.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:02.311+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:02.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:02.698+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:02.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:02.698+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:02.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:02.698+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:02.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:02.698+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:03.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:03.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:03.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:03.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:03.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:03.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:03.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:03.302+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:03.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:03.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:03.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:03.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:03.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:03.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:03.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:03.369+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:03.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:03.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:03.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:03.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:03.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:03.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:03.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:03.730+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:04.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:04.281+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:04.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:04.281+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:04.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:04.281+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:04.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:04.282+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:04.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:04.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:04.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:04.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:04.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:04.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:04.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:04.352+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:05.214 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:04.726+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:05.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:04.726+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:05.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:04.726+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:05.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:04.726+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:05.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:05.236+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:05.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:05.236+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:05.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:05.236+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:05.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:05.237+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:05.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:05.402+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:05.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:05.402+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:05.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:05.402+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:05.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:05.402+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:05.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:05.690+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:05.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:05.690+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:05.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:05.690+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:05.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:05.690+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:06.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:06.258+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:06.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:06.258+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:06.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:06.258+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:06.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:06.260+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:06.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:06.400+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:06.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:06.400+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:06.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:06.400+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:06.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:06.400+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:06.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:06.695+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:06.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:06.696+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:06.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:06.696+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:06.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:06.696+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:07.464 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:50:07.166+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 43160 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:50:07.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:07.284+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:07.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:07.284+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:07.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:07.284+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:07.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:07.285+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:07.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:07.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:07.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:07.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:07.466 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:07.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:07.466 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:07.380+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:07.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:07.692+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:07.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:07.692+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:07.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:07.692+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:07.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:07.693+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:08.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:08.318+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:08.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:08.318+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:08.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:08.318+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:08.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:08.319+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:08.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:08.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:08.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:08.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:08.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:08.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:08.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:08.410+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:08.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:08.714+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:08.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:08.714+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:08.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:08.714+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:08.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:08.714+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:09.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:09.335+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:09.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:09.335+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:09.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:09.335+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:09.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:09.336+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:09.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:09.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:09.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:09.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:09.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:09.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:09.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:09.434+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:10.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:09.758+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:10.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:09.758+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:10.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:09.758+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:10.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:09.759+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:10.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:10.351+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:10.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:10.351+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:10.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:10.351+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:10.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:10.352+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:10.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:10.387+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:10.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:10.387+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:10.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:10.387+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:10.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:10.388+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:11.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:10.778+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:11.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:10.778+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:11.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:10.778+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:11.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:10.778+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:11.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:11.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:11.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:11.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:11.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:11.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:11.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:11.339+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:11.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:11.390+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:11.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:11.390+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:11.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:11.390+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:11.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:11.391+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:12.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:11.737+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:12.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:11.737+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:12.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:11.737+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:12.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:11.737+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:12.424 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:50:12.171+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 43249 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:50:12.424 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:12.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:12.424 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:12.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:12.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:12.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:12.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:12.322+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:12.686 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:12.428+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:12.686 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:12.428+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:12.686 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:12.428+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:12.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:12.429+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:12.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:12.690+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:12.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:12.690+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:12.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:12.690+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:12.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:12.691+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:13.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:13.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:13.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:13.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:13.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:13.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:13.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:13.340+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:13.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:13.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:13.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:13.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:13.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:13.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:13.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:13.437+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:13.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:13.704+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:13.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:13.704+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:13.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:13.704+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:13.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:13.704+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:14.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:14.299+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:14.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:14.299+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:14.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:14.299+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:14.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:14.300+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:14.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:14.443+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:14.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:14.443+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:14.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:14.443+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:14.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:14.444+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:15.042 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:14.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:15.042 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:14.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:15.042 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:14.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:15.042 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:14.745+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:15.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:15.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:15.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:15.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:15.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:15.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:15.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:15.313+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:15.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:15.489+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:15.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:15.489+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:15.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:15.489+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:15.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:15.490+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:16.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:15.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:16.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:15.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:16.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:15.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:16.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:15.795+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:16.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:16.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:16.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:16.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:16.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:16.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:16.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:16.324+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:16.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:16.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:16.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:16.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:16.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:16.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:16.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:16.458+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:17.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:16.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:17.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:16.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:17.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:16.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:17.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:16.827+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:17.465 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:50:17.176+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 43340 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:50:17.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:17.352+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:17.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:17.352+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:17.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:17.352+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:17.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:17.353+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:17.825 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:17.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:17.826 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:17.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:17.826 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:17.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:17.826 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:17.500+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:18.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:17.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:18.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:17.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:18.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:17.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:18.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:17.830+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:18.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:18.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:18.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:18.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:18.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:18.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:18.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:18.377+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:18.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:18.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:18.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:18.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:18.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:18.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:18.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:18.466+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:19.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:18.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:19.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:18.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:19.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:18.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:19.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:18.857+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:19.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:19.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:19.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:19.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:19.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:19.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:19.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:19.437+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:19.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:19.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:19.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:19.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:19.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:19.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:19.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:19.421+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:20.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:19.840+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:20.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:19.840+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:20.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:19.840+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:20.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:19.840+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:20.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:20.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:20.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:20.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:20.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:20.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:20.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:20.457+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:20.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:20.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:20.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:20.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:20.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:20.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:20.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:20.449+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:21.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:20.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:21.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:20.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:21.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:20.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:21.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:20.798+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:21.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:21.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:21.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:21.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:21.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:21.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:21.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:21.496+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:21.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:21.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:21.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:21.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:21.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:21.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:21.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:21.483+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:22.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:21.781+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:22.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:21.781+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:22.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:21.781+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:22.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:21.781+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:22.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:50:22.183+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 43429 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:50:22.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:22.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:22.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:22.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:22.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:22.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:22.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:22.517+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:22.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:22.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:22.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:22.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:22.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:22.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:22.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:22.465+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:23.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:22.768+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:23.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:22.768+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:23.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:22.768+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:23.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:22.769+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:23.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:23.469+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:23.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:23.469+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:23.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:23.469+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:23.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:23.471+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:23.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:23.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:23.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:23.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:23.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:23.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:23.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:23.499+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:24.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:23.807+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:24.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:23.807+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:24.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:23.807+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:24.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:23.807+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:24.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:24.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:24.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:24.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:24.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:24.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:24.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:24.451+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:24.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:24.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:24.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:24.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:24.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:24.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:24.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:24.474+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:25.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:24.766+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:25.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:24.766+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:25.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:24.766+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:25.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:24.766+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:25.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:25.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:25.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:25.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:25.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:25.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:25.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:25.474+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:25.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:25.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:25.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:25.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:25.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:25.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:25.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:25.433+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:26.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:25.731+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:26.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:25.731+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:26.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:25.731+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:26.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:25.732+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:26.498 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:26.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:26.498 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:26.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:26.498 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:26.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:26.498 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:26.404+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:26.809 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:26.703+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:26.809 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:26.703+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:26.809 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:26.703+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:26.809 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:26.704+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:26.809 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:26.502+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:26.809 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:26.502+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:26.810 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:26.502+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:26.810 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:26.504+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:27.465 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:50:27.188+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 43520 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:50:27.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:27.396+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:27.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:27.396+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:27.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:27.396+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:27.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:27.397+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:27.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:27.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:27.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:27.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:27.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:27.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:27.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:27.664+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:27.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:27.492+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:27.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:27.492+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:27.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:27.492+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:27.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:27.494+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:28.640 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:28.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:28.641 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:28.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:28.641 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:28.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:28.641 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:28.446+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:28.641 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:28.394+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:28.641 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:28.394+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:28.641 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:28.394+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:28.641 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:28.394+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:28.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:28.645+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:28.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:28.645+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:28.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:28.645+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:28.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:28.646+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:29.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:29.419+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:29.665 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:29.419+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:29.665 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:29.419+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:29.665 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:29.420+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:29.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:29.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:29.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:29.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:29.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:29.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:29.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:29.420+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:29.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:29.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:29.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:29.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:29.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:29.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:29.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:29.670+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:30.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:30.620+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:30.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:30.620+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:30.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:30.620+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:30.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:30.620+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:30.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:30.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:30.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:30.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:30.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:30.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:30.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:30.451+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:30.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:30.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:30.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:30.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:30.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:30.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:30.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:30.460+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:31.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:31.479+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:31.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:31.479+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:31.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:31.479+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:31.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:31.480+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:31.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:31.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:31.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:31.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:31.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:31.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:31.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:31.429+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:31.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:31.648+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:31.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:31.648+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:31.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:31.648+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:31.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:31.649+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:32.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:50:32.193+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 43606 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:50:32.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:32.673+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:32.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:32.673+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:32.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:32.673+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:32.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:32.673+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:32.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:32.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:32.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:32.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:32.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:32.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:32.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:32.456+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:32.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:32.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:32.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:32.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:32.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:32.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:32.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:32.474+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:33.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:33.687+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:33.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:33.687+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:33.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:33.687+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:33.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:33.687+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:33.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:33.496+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:33.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:33.496+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:33.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:33.496+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:33.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:33.496+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:33.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:33.469+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:33.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:33.469+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:33.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:33.469+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:33.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:33.471+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:34.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:34.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:34.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:34.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:34.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:34.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:34.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:34.453+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:34.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:34.700+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:34.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:34.700+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:34.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:34.700+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:34.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:34.701+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:34.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:34.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:34.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:34.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:34.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:34.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:34.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:34.532+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:35.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:35.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:35.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:35.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:35.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:35.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:35.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:35.436+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:35.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:35.504+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:35.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:35.504+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:35.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:35.504+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:35.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:35.505+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:35.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:35.718+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:35.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:35.718+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:35.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:35.718+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:35.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:35.718+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:36.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:36.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:36.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:36.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:36.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:36.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:36.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:36.453+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:36.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:36.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:36.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:36.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:36.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:36.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:36.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:36.534+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:37.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:36.767+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:37.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:36.767+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:37.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:36.767+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:37.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:36.767+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:37.465 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:50:37.199+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 43691 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:50:37.779 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:37.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:37.779 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:37.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:37.779 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:37.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:37.779 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:37.492+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:37.780 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:37.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:37.780 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:37.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:37.780 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:37.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:37.780 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:37.545+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:38.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:37.783+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:38.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:37.783+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:38.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:37.783+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:38.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:37.784+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:38.828 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:38.492+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:38.828 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:38.492+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:38.828 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:38.492+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:38.828 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:38.493+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:38.828 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:38.581+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:38.828 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:38.581+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:38.828 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:38.581+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:38.828 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:38.582+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:39.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:38.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:39.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:38.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:39.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:38.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:39.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:38.833+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:39.786 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:39.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:39.787 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:39.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:39.787 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:39.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:39.787 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:39.472+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:39.787 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:39.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:39.787 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:39.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:39.787 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:39.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:39.787 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:39.623+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:40.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:39.791+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:40.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:39.791+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:40.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:39.791+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:40.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:39.791+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:40.771 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:40.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:40.772 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:40.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:40.772 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:40.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:40.772 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:40.515+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:40.772 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:40.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:40.772 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:40.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:40.772 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:40.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:40.772 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:40.592+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:41.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:40.776+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:41.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:40.776+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:41.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:40.776+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:41.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:40.776+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:41.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:41.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:41.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:41.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:41.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:41.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:41.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:41.519+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:41.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:41.753+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:41.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:41.753+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:41.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:41.753+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:41.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:41.754+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:41.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:41.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:41.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:41.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:41.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:41.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:41.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:41.598+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:42.465 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:50:42.205+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 43771 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:50:42.756 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:42.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:42.756 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:42.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:42.756 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:42.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:42.756 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:42.495+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:42.756 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:42.557+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:42.756 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:42.557+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:42.756 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:42.557+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:42.756 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:42.558+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:43.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:42.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:43.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:42.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:43.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:42.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:43.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:42.760+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:43.965 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:43.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:43.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:43.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:43.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:43.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:43.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:43.543+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:43.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:43.769+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:43.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:43.769+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:43.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:43.769+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:43.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:43.769+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:43.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:43.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:43.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:43.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:43.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:43.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:43.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:43.554+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:44.813 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:44.538+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:44.813 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:44.538+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:44.813 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:44.538+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:44.813 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:44.539+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:44.813 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:44.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:44.813 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:44.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:44.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:44.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:44.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:44.496+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:45.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:44.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:45.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:44.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:45.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:44.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:45.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:44.818+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:45.516 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:45.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:45.516 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:45.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:45.516 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:45.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:45.516 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:45.452+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:45.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:45.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:45.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:45.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:45.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:45.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:45.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:45.521+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:46.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:45.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:46.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:45.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:46.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:45.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:46.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:45.848+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:46.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:46.498+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:46.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:46.498+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:46.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:46.498+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:46.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:46.498+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:46.635 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:46.478+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:46.635 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:46.478+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:46.635 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:46.478+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:46.636 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:46.479+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:46.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:46.802+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:46.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:46.802+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:46.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:46.802+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:46.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:46.803+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:47.465 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:50:47.210+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 43854 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:50:47.810 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:47.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:47.811 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:47.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:47.811 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:47.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:47.811 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:47.514+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:47.811 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:47.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:47.811 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:47.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:47.811 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:47.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:47.811 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:47.523+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:48.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:47.815+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:48.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:47.815+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:48.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:47.815+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:48.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:47.815+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:48.773 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:48.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:48.773 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:48.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:48.773 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:48.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:48.774 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:48.537+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:48.774 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:48.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:48.774 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:48.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:48.774 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:48.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:48.774 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:48.498+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:49.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:48.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:49.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:48.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:49.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:48.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:49.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:48.778+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:49.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:49.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:49.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:49.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:49.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:49.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:49.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:49.491+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:49.914 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:49.502+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:49.914 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:49.502+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:49.914 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:49.502+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:49.914 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:49.502+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:49.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:49.738+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:49.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:49.738+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:49.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:49.738+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:49.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:49.739+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:50.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:50.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:50.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:50.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:50.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:50.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:50.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:50.521+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:50.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:50.716+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:50.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:50.716+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:50.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:50.716+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:50.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:50.716+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:50.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:50.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:50.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:50.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:50.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:50.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:50.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:50.495+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:51.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:51.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:51.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:51.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:51.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:51.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:51.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:51.509+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:51.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:51.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:51.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:51.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:51.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:51.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:51.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:51.514+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:51.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:51.687+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:51.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:51.687+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:51.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:51.687+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:51.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:51.687+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:52.466 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:50:52.215+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 43938 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:50:52.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:52.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:52.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:52.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:52.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:52.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:52.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:52.548+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:52.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:52.717+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:52.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:52.717+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:52.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:52.717+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:52.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:52.717+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:52.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:52.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:52.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:52.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:52.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:52.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:52.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:52.486+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:53.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:53.503+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:53.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:53.503+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:53.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:53.503+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:53.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:53.503+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:53.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:53.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:53.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:53.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:53.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:53.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:53.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:53.455+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:54.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:53.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:54.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:53.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:54.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:53.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:54.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:53.752+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:54.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:54.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:54.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:54.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:54.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:54.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:54.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:54.439+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:54.705 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:54.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:54.705 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:54.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:54.706 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:54.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:54.706 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:54.507+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:54.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:54.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:54.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:54.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:54.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:54.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:54.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:54.709+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:55.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:55.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:55.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:55.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:55.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:55.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:55.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:55.497+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:55.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:55.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:55.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:55.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:55.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:55.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:55.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:55.431+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:55.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:55.714+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:55.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:55.714+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:55.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:55.714+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:55.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:55.714+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:56.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:56.484+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:56.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:56.484+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:56.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:56.484+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:56.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:56.485+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:56.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:56.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:56.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:56.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:56.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:56.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:56.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:56.404+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:57.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:56.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:57.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:56.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:57.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:56.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:57.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:56.761+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:57.486 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:50:57.221+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 44026 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:50:57.486 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:57.361+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:57.486 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:57.361+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:57.487 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:57.361+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:57.487 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:57.362+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:57.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:57.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:57.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:57.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:57.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:57.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:57.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:57.491+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:57.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:57.723+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:57.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:57.723+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:57.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:57.723+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:57.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:57.724+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:58.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:58.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:58.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:58.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:58.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:58.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:58.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:58.531+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:58.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:58.342+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:58.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:58.342+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:58.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:58.342+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:58.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:58.343+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:59.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:58.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:59.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:58.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:59.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:58.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:50:59.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:58.730+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:59.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:59.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:59.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:59.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:59.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:59.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:50:59.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:50:59.375+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:50:59.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:59.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:59.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:59.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:59.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:59.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:50:59.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:50:59.495+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:00.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:59.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:00.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:59.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:00.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:59.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:00.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:50:59.742+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:00.456 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:00.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:00.456 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:00.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:00.456 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:00.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:00.456 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:00.351+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:00.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:00.696+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:00.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:00.696+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:00.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:00.696+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:00.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:00.696+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:00.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:00.460+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:00.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:00.460+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:00.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:00.460+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:00.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:00.461+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:01.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:01.443+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:01.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:01.443+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:01.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:01.443+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:01.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:01.444+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:01.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:01.387+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:01.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:01.387+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:01.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:01.387+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:01.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:01.387+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:02.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:01.723+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:02.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:01.723+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:02.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:01.723+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:02.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:01.724+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:02.716 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:51:02.226+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 44115 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:51:02.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:02.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:02.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:02.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:02.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:02.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:02.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:02.339+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:02.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:02.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:02.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:02.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:02.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:02.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:02.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:02.445+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:03.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:02.723+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:03.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:02.723+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:03.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:02.723+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:03.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:02.724+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:03.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:03.329+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:03.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:03.330+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:03.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:03.330+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:03.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:03.331+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:03.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:03.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:03.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:03.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:03.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:03.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:03.689 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:03.476+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:03.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:03.755+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:03.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:03.755+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:03.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:03.755+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:03.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:03.755+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:04.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:04.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:04.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:04.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:04.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:04.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:04.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:04.449+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:04.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:04.292+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:04.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:04.292+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:04.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:04.292+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:04.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:04.293+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:05.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:04.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:05.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:04.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:05.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:04.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:05.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:04.793+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:05.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:05.292+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:05.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:05.292+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:05.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:05.292+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:05.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:05.292+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:05.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:05.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:05.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:05.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:05.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:05.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:05.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:05.495+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:06.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:05.776+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:06.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:05.776+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:06.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:05.776+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:06.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:05.776+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:06.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:06.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:06.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:06.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:06.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:06.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:06.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:06.503+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:06.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:06.301+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:06.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:06.301+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:06.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:06.301+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:06.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:06.301+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:07.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:06.806+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:07.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:06.806+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:07.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:06.806+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:07.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:06.807+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:07.716 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:51:07.232+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 44206 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:51:07.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:07.347+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:07.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:07.347+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:07.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:07.347+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:07.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:07.348+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:07.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:07.476+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:07.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:07.476+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:07.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:07.476+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:07.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:07.477+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:08.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:07.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:08.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:07.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:08.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:07.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:08.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:07.822+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:08.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:08.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:08.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:08.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:08.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:08.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:08.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:08.359+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:08.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:08.503+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:08.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:08.503+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:08.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:08.503+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:08.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:08.505+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:09.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:08.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:09.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:08.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:09.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:08.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:09.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:08.778+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:09.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:09.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:09.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:09.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:09.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:09.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:09.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:09.389+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:09.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:09.529+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:09.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:09.529+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:09.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:09.529+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:09.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:09.531+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:10.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:09.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:10.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:09.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:10.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:09.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:10.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:09.735+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:10.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:10.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:10.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:10.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:10.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:10.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:10.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:10.427+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:10.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:10.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:10.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:10.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:10.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:10.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:10.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:10.500+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:10.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:10.703+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:10.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:10.703+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:10.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:10.703+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:10.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:10.704+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:11.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:11.683+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:11.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:11.683+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:11.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:11.683+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:11.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:11.683+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:11.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:11.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:11.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:11.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:11.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:11.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:11.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:11.462+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:11.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:11.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:11.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:11.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:11.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:11.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:11.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:11.521+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:12.496 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:51:12.237+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 44296 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:51:12.496 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:12.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:12.496 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:12.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:12.497 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:12.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:12.497 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:12.483+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:12.497 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:12.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:12.497 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:12.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:12.497 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:12.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:12.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:12.500+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:12.889 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:12.634+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:12.890 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:12.634+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:12.890 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:12.634+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:12.890 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:12.635+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:13.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:13.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:13.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:13.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:13.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:13.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:13.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:13.497+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:13.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:13.510+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:13.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:13.510+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:13.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:13.510+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:13.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:13.511+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:13.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:13.617+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:13.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:13.617+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:13.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:13.617+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:13.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:13.618+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:14.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:14.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:14.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:14.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:14.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:14.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:14.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:14.472+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:14.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:14.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:14.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:14.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:14.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:14.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:14.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:14.490+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:14.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:14.610+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:14.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:14.610+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:14.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:14.610+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:14.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:14.611+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:15.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:15.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:15.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:15.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:15.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:15.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:15.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:15.464+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:15.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:15.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:15.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:15.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:15.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:15.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:15.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:15.530+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:15.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:15.594+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:15.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:15.594+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:15.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:15.594+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:15.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:15.595+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:16.623 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:16.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:16.623 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:16.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:16.623 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:16.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:16.623 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:16.465+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:16.624 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:16.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:16.624 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:16.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:16.624 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:16.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:16.624 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:16.536+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:16.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:16.626+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:16.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:16.626+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:16.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:16.626+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:16.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:16.626+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:17.546 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:51:17.243+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 44386 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:51:17.546 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:17.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:17.547 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:17.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:17.547 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:17.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:17.547 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:17.475+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:17.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:17.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:17.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:17.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:17.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:17.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:17.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:17.550+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:17.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:17.661+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:17.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:17.661+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:17.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:17.661+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:17.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:17.662+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:18.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:18.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:18.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:18.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:18.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:18.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:18.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:18.711+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:18.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:18.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:18.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:18.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:18.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:18.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:18.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:18.599+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:18.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:18.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:18.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:18.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:18.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:18.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:18.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:18.495+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:19.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:19.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:19.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:19.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:19.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:19.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:19.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:19.639+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:19.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:19.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:19.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:19.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:19.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:19.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:19.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:19.534+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:19.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:19.671+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:20.003 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:19.671+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:20.003 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:19.671+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:20.003 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:19.671+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:20.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:20.628+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:20.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:20.628+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:20.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:20.628+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:20.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:20.629+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:20.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:20.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:20.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:20.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:20.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:20.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:20.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:20.583+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:20.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:20.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:20.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:20.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:20.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:20.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:20.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:20.662+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:21.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:21.644+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:21.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:21.644+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:21.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:21.644+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:21.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:21.645+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:21.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:21.584+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:21.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:21.584+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:21.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:21.584+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:21.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:21.584+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:21.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:21.617+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:21.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:21.617+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:21.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:21.617+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:21.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:21.618+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:22.581 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:51:22.249+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 44475 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:51:22.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:22.584+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:22.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:22.584+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:22.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:22.584+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:22.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:22.585+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:22.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:22.631+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:22.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:22.631+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:22.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:22.631+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:22.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:22.632+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:22.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:22.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:22.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:22.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:22.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:22.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:22.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:22.619+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:23.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:23.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:23.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:23.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:23.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:23.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:23.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:23.580+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:23.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:23.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:23.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:23.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:23.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:23.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:23.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:23.632+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:23.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:23.613+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:23.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:23.613+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:23.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:23.613+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:23.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:23.614+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:24.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:24.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:24.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:24.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:24.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:24.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:24.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:24.639+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:24.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:24.556+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:24.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:24.556+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:24.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:24.556+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:24.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:24.557+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:24.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:24.605+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:24.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:24.605+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:24.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:24.605+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:24.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:24.606+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:25.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:25.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:25.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:25.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:25.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:25.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:25.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:25.565+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:25.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:25.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:25.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:25.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:25.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:25.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:25.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:25.590+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:25.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:25.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:25.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:25.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:25.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:25.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:25.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:25.575+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:26.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:26.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:26.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:26.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:26.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:26.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:26.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:26.526+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:26.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:26.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:26.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:26.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:26.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:26.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:26.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:26.569+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:26.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:26.614+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:26.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:26.614+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:26.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:26.614+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:26.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:26.614+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:27.274 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:51:27.255+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 44548 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:51:27.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:27.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:27.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:27.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:27.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:27.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:27.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:27.523+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:27.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:27.572+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:27.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:27.572+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:27.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:27.572+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:27.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:27.573+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:27.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:27.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:27.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:27.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:27.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:27.593+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:27.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:27.594+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:28.501 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:28.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:28.501 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:28.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:28.501 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:28.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:28.501 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:28.504+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:28.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:28.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:28.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:28.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:28.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:28.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:28.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:28.587+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:28.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:28.545+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:28.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:28.545+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:28.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:28.545+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:28.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:28.545+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:29.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:29.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:29.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:29.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:29.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:29.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:29.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:29.508+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:29.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:29.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:29.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:29.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:29.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:29.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:29.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:29.549+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:29.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:29.601+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:29.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:29.601+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:29.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:29.601+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:29.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:29.602+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:30.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:30.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:30.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:30.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:30.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:30.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:30.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:30.477+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:30.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:30.513+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:30.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:30.513+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:30.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:30.513+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:30.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:30.513+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:30.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:30.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:30.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:30.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:30.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:30.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:30.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:30.573+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:31.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:31.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:31.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:31.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:31.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:31.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:31.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:31.440+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:31.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:31.522+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:31.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:31.522+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:31.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:31.522+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:31.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:31.523+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:31.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:31.566+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:31.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:31.566+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:31.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:31.566+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:31.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:31.567+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:32.381 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:51:32.261+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 44593 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:51:32.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:32.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:32.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:32.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:32.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:32.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:32.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:32.455+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:32.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:32.557+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:32.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:32.557+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:32.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:32.557+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:32.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:32.558+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:32.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:32.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:32.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:32.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:32.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:32.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:32.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:32.523+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:33.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:33.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:33.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:33.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:33.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:33.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:33.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:33.492+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:33.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:33.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:33.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:33.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:33.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:33.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:33.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:33.495+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:33.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:33.540+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:33.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:33.540+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:33.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:33.540+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:33.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:33.540+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:34.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:34.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:34.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:34.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:34.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:34.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:34.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:34.448+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:34.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:34.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:34.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:34.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:34.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:34.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:34.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:34.493+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:34.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:34.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:34.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:34.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:34.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:34.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:34.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:34.504+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:35.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:35.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:35.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:35.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:35.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:35.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:35.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:35.488+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:35.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:35.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:35.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:35.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:35.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:35.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:35.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:35.445+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:35.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:35.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:35.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:35.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:35.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:35.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:35.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:35.494+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:36.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:36.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:36.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:36.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:36.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:36.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:36.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:36.529+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:36.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:36.419+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:36.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:36.419+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:36.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:36.419+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:36.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:36.420+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:36.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:36.458+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:36.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:36.458+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:36.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:36.458+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:36.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:36.458+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:37.272 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:51:37.267+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 44639 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:51:37.553 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:37.382+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:37.553 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:37.382+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:37.553 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:37.382+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:37.553 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:37.383+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:37.554 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:37.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:37.554 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:37.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:37.554 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:37.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:37.554 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:37.488+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:37.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:37.556+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:37.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:37.556+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:37.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:37.556+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:37.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:37.557+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:38.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:38.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:38.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:38.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:38.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:38.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:38.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:38.570+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:38.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:38.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:38.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:38.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:38.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:38.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:38.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:38.460+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:38.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:38.359+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:38.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:38.359+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:38.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:38.359+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:38.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:38.360+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:39.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:39.361+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:39.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:39.361+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:39.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:39.361+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:39.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:39.362+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:39.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:39.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:39.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:39.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:39.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:39.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:39.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:39.604+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:39.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:39.434+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:39.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:39.434+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:39.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:39.434+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:39.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:39.435+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:40.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:40.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:40.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:40.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:40.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:40.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:40.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:40.584+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:40.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:40.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:40.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:40.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:40.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:40.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:40.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:40.433+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:40.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:40.406+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:40.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:40.406+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:40.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:40.406+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:40.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:40.407+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:41.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:41.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:41.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:41.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:41.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:41.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:41.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:41.423+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:41.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:41.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:41.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:41.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:41.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:41.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:41.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:41.426+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:41.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:41.594+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:41.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:41.594+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:41.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:41.594+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:41.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:41.595+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:42.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:51:42.272+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 44683 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:51:42.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:42.421+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:42.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:42.421+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:42.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:42.421+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:42.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:42.422+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:42.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:42.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:42.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:42.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:42.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:42.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:42.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:42.569+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:42.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:42.438+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:42.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:42.438+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:42.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:42.438+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:42.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:42.439+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:43.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:43.453+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:43.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:43.453+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:43.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:43.453+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:43.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:43.454+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:43.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:43.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:43.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:43.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:43.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:43.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:43.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:43.458+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:43.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:43.610+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:43.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:43.610+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:43.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:43.610+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:43.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:43.611+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:44.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:44.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:44.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:44.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:44.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:44.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:44.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:44.581+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:44.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:44.435+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:44.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:44.435+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:44.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:44.435+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:44.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:44.435+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:44.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:44.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:44.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:44.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:44.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:44.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:44.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:44.425+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:45.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:45.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:45.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:45.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:45.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:45.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:45.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:45.537+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:45.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:45.392+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:45.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:45.392+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:45.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:45.392+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:45.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:45.393+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:45.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:45.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:45.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:45.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:45.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:45.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:45.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:45.446+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:46.626 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:46.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:46.626 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:46.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:46.626 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:46.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:46.626 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:46.434+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:46.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:51:46.629+0000 7f770c428700 0 monclient(hunting): authenticate timed out after 300 2024-04-27T02:51:46.627 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:46.531+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:46.627 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:46.531+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:46.627 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:46.531+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:46.627 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:46.532+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:46.627 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:46.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:46.627 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:46.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:46.628 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:46.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:46.628 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:46.471+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:47.467 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:51:47.278+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 44729 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:51:47.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:47.406+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:47.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:47.406+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:47.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:47.406+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:47.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:47.407+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:47.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:47.560+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:47.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:47.560+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:47.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:47.560+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:47.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:47.561+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:47.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:47.514+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:47.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:47.514+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:47.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:47.514+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:47.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:47.514+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:48.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:48.416+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:48.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:48.416+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:48.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:48.416+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:48.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:48.417+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:48.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:48.546+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:48.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:48.546+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:48.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:48.546+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:48.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:48.547+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:48.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:48.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:48.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:48.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:48.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:48.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:48.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:48.530+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:49.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:49.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:49.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:49.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:49.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:49.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:49.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:49.434+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:49.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:49.540+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:49.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:49.540+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:49.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:49.540+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:49.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:49.540+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:49.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:49.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:49.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:49.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:49.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:49.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:49.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:49.484+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:50.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:50.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:50.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:50.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:50.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:50.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:50.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:50.396+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:50.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:50.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:50.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:50.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:50.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:50.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:50.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:50.521+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:50.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:50.505+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:50.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:50.505+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:50.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:50.505+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:50.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:50.506+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:51.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:51.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:51.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:51.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:51.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:51.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:51.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:51.491+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:51.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:51.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:51.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:51.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:51.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:51.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:51.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:51.523+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:51.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:51.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:51.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:51.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:51.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:51.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:51.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:51.412+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:52.530 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:51:52.284+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 44774 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:51:52.531 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:52.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:52.531 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:52.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:52.531 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:52.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:52.531 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:52.451+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:52.531 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:52.496+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:52.531 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:52.496+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:52.531 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:52.496+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:52.531 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:52.497+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:52.827 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:52.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:52.827 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:52.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:52.827 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:52.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:52.827 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:52.534+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:53.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:53.577+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:53.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:53.577+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:53.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:53.577+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:53.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:53.578+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:53.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:53.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:53.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:53.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:53.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:53.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:53.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:53.456+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:53.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:53.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:53.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:53.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:53.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:53.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:53.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:53.529+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:54.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:54.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:54.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:54.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:54.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:54.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:54.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:54.551+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:54.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:54.432+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:54.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:54.432+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:54.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:54.432+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:54.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:54.432+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:54.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:54.566+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:54.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:54.566+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:54.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:54.566+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:54.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:54.567+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:55.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:55.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:55.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:55.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:55.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:55.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:55.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:55.476+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:55.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:55.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:55.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:55.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:55.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:55.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:55.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:55.550+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:55.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:55.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:55.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:55.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:55.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:55.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:55.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:55.534+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:56.554 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:56.443+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:56.555 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:56.443+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:56.555 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:56.443+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:56.555 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:56.444+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:56.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:56.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:56.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:56.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:56.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:56.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:56.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:56.558+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:56.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:56.599+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:56.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:56.599+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:56.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:56.599+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:56.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:56.600+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:57.549 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:51:57.289+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 44820 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:51:57.549 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:57.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:57.550 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:57.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:57.550 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:57.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:57.550 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:57.448+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:57.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:57.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:57.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:57.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:57.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:57.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:57.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:57.553+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:57.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:57.556+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:57.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:57.556+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:57.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:57.556+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:57.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:57.557+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:58.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:58.536+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:58.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:58.536+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:58.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:58.536+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:58.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:58.536+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:58.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:58.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:58.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:58.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:58.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:58.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:58.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:58.462+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:58.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:58.532+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:58.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:58.532+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:58.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:58.532+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:58.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:58.533+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:59.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:59.552+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:59.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:59.552+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:59.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:59.552+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:51:59.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:51:59.553+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:59.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:59.573+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:59.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:59.573+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:59.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:59.573+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:51:59.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:51:59.575+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:51:59.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:59.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:59.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:59.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:59.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:59.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:51:59.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:51:59.486+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:00.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:00.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:00.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:00.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:00.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:00.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:00.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:00.514+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:00.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:00.532+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:00.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:00.532+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:00.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:00.532+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:00.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:00.532+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:00.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:00.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:00.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:00.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:00.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:00.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:00.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:00.583+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:01.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:01.497+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:01.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:01.497+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:01.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:01.497+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:01.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:01.497+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:01.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:01.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:01.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:01.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:01.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:01.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:01.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:01.612+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:01.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:01.536+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:01.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:01.536+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:01.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:01.536+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:01.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:01.537+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:02.572 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:52:02.295+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 44864 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:52:02.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:02.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:02.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:02.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:02.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:02.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:02.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:02.449+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:02.573 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:02.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:02.573 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:02.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:02.573 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:02.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:02.573 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:02.503+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:02.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:02.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:02.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:02.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:02.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:02.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:02.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:02.575+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:03.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:03.551+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:03.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:03.551+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:03.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:03.551+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:03.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:03.552+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:03.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:03.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:03.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:03.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:03.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:03.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:03.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:03.475+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:03.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:03.440+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:03.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:03.440+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:03.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:03.440+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:03.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:03.440+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:04.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:04.478+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:04.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:04.478+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:04.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:04.478+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:04.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:04.479+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:04.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:04.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:04.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:04.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:04.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:04.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:04.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:04.528+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:04.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:04.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:04.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:04.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:04.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:04.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:04.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:04.428+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:05.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:05.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:05.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:05.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:05.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:05.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:05.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:05.468+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:05.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:05.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:05.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:05.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:05.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:05.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:05.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:05.430+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:05.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:05.515+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:05.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:05.515+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:05.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:05.515+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:05.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:05.515+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:06.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:06.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:06.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:06.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:06.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:06.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:06.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:06.524+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:06.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:06.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:06.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:06.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:06.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:06.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:06.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:06.424+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:06.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:06.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:06.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:06.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:06.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:06.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:06.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:06.392+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:07.565 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:52:07.300+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 44910 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:52:07.565 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:07.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:07.565 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:07.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:07.565 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:07.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:07.565 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:07.432+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:07.566 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:07.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:07.566 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:07.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:07.566 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:07.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:07.566 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:07.463+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:07.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:07.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:07.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:07.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:07.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:07.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:07.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:07.569+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:08.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:08.450+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:08.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:08.450+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:08.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:08.450+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:08.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:08.451+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:08.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:08.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:08.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:08.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:08.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:08.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:08.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:08.492+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:08.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:08.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:08.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:08.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:08.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:08.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:08.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:08.596+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:09.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:09.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:09.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:09.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:09.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:09.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:09.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:09.406+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:09.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:09.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:09.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:09.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:09.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:09.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:09.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:09.626+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:09.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:09.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:09.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:09.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:09.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:09.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:09.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:09.512+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:10.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:10.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:10.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:10.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:10.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:10.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:10.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:10.388+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:10.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:10.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:10.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:10.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:10.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:10.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:10.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:10.464+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:10.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:10.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:10.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:10.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:10.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:10.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:10.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:10.582+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:11.580 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:11.417+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:11.581 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:11.417+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:11.581 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:11.417+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:11.581 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:11.418+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:11.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:11.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:11.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:11.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:11.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:11.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:11.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:11.352+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:11.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:11.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:11.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:11.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:11.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:11.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:11.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:11.583+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:12.610 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:52:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:52:12.305+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 44955 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:52:12.610 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:12.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:12.611 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:12.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:12.611 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:12.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:12.611 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:12.379+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:12.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:12.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:12.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:12.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:12.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:12.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:12.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:12.312+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:12.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:12.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:12.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:12.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:12.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:12.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:12.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:12.613+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:13.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:13.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:13.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:13.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:13.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:13.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:13.609 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:13.289+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:13.609 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:13.413+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:13.609 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:13.413+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:13.609 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:13.413+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:13.609 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:13.414+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:13.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:13.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:13.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:13.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:13.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:13.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:13.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:13.611+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:14.631 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:14.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:14.631 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:14.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:14.631 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:14.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:14.631 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:14.319+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:14.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:14.412+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:14.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:14.412+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:14.632 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:14.412+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:14.632 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:14.413+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:14.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:14.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:14.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:14.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:14.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:14.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:14.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:14.634+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:15.666 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:15.367+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:15.666 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:15.367+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:15.666 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:15.367+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:15.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:15.367+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:15.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:15.346+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:15.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:15.346+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:15.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:15.346+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:15.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:15.347+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:15.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:15.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:15.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:15.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:15.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:15.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:15.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:15.669+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:16.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:16.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:16.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:16.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:16.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:16.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:16.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:16.375+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:16.647 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:16.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:16.647 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:16.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:16.647 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:16.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:16.647 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:16.384+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:16.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:16.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:16.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:16.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:16.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:16.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:16.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:16.648+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:17.625 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:52:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:52:17.311+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 44997 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:52:17.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:17.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:17.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:17.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:17.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:17.340+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:17.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:17.340+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:17.626 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:17.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:17.626 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:17.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:17.626 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:17.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:17.626 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:17.408+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:17.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:17.626+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:17.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:17.626+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:17.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:17.626+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:17.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:17.628+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:18.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:18.585+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:18.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:18.585+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:18.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:18.585+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:18.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:18.586+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:18.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:18.367+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:18.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:18.367+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:18.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:18.367+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:18.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:18.368+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:18.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:18.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:18.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:18.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:18.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:18.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:18.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:18.378+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:19.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:19.600+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:19.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:19.600+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:19.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:19.600+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:19.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:19.600+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:19.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:19.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:19.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:19.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:19.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:19.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:19.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:19.384+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:19.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:19.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:19.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:19.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:19.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:19.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:19.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:19.383+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:20.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:20.337+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:20.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:20.337+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:20.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:20.337+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:20.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:20.338+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:20.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:20.608+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:20.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:20.608+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:20.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:20.608+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:20.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:20.609+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:20.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:20.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:20.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:20.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:20.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:20.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:20.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:20.406+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:21.632 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:21.324+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:21.632 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:21.324+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:21.632 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:21.324+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:21.632 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:21.324+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:21.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:21.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:21.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:21.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:21.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:21.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:21.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:21.361+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:21.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:21.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:21.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:21.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:21.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:21.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:21.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:21.635+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:22.624 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:52:22.317+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 45041 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:52:22.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:22.337+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:22.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:22.337+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:22.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:22.337+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:22.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:22.338+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:22.625 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:22.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:22.625 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:22.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:22.625 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:22.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:22.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:22.329+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:22.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:22.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:22.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:22.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:22.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:22.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:22.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:22.628+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:23.467 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:23.304+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:23.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:23.304+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:23.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:23.304+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:23.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:23.304+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:23.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:23.340+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:23.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:23.340+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:23.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:23.340+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:23.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:23.341+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:23.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:23.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:23.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:23.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:23.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:23.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:23.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:23.599+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:24.589 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:24.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:24.589 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:24.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:24.589 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:24.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:24.589 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:24.314+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:24.589 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:24.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:24.589 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:24.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:24.589 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:24.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:24.589 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:24.340+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:24.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:24.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:24.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:24.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:24.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:24.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:24.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:24.592+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:25.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:25.299+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:25.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:25.299+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:25.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:25.299+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:25.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:25.299+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:25.608 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:25.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:25.608 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:25.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:25.608 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:25.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:25.608 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:25.314+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:25.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:25.610+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:25.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:25.610+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:25.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:25.610+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:25.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:25.611+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:26.619 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:26.300+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:26.619 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:26.300+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:26.619 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:26.300+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:26.619 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:26.301+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:26.620 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:26.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:26.620 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:26.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:26.620 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:26.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:26.620 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:26.280+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:26.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:26.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:26.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:26.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:26.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:26.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:26.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:26.628+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:27.590 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:52:27.323+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 45087 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:52:27.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:27.329+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:27.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:27.329+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:27.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:27.329+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:27.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:27.329+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:27.591 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:27.336+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:27.591 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:27.336+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:27.591 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:27.336+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:27.592 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:27.337+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:27.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:27.626+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:27.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:27.626+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:27.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:27.626+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:27.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:27.627+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:28.610 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:28.349+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:28.610 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:28.349+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:28.610 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:28.349+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:28.610 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:28.350+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:28.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:28.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:28.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:28.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:28.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:28.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:28.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:28.368+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:28.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:28.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:28.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:28.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:28.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:28.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:28.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:28.613+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:29.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:29.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:29.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:29.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:29.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:29.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:29.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:29.408+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:29.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:29.337+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:29.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:29.337+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:29.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:29.337+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:29.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:29.339+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:29.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:29.615+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:29.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:29.615+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:29.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:29.615+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:29.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:29.616+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:30.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:30.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:30.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:30.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:30.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:30.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:30.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:30.364+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:30.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:30.315+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:30.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:30.315+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:30.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:30.315+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:30.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:30.317+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:30.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:30.594+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:30.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:30.594+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:30.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:30.594+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:30.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:30.595+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:31.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:31.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:31.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:31.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:31.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:31.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:31.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:31.383+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:31.618 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:31.362+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:31.618 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:31.362+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:31.618 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:31.362+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:31.618 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:31.363+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:31.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:31.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:31.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:31.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:31.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:31.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:31.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:31.620+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:32.643 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:52:32.328+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 45132 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:52:32.644 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:32.360+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:32.644 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:32.360+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:32.644 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:32.360+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:32.644 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:32.361+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:32.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:32.405+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:32.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:32.405+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:32.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:32.405+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:32.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:32.406+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:32.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:32.645+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:32.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:32.645+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:32.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:32.645+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:32.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:32.647+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:33.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:33.422+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:33.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:33.422+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:33.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:33.422+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:33.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:33.422+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:33.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:33.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:33.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:33.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:33.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:33.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:33.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:33.344+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:33.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:33.685+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:33.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:33.685+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:33.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:33.685+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:33.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:33.686+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:34.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:34.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:34.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:34.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:34.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:34.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:34.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:34.307+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:34.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:34.427+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:34.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:34.427+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:34.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:34.427+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:34.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:34.428+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:34.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:34.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:34.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:34.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:34.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:34.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:34.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:34.713+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:35.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:35.323+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:35.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:35.323+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:35.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:35.323+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:35.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:35.324+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:35.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:35.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:35.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:35.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:35.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:35.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:35.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:35.403+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:35.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:35.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:35.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:35.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:35.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:35.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:35.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:35.674+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:36.630 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:36.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:36.630 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:36.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:36.630 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:36.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:36.630 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:36.283+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:36.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:36.371+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:36.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:36.371+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:36.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:36.371+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:36.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:36.372+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:36.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:36.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:36.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:36.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:36.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:36.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:36.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:36.633+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:37.656 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:52:37.334+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 45178 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:52:37.657 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:37.415+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:37.657 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:37.415+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:37.657 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:37.415+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:37.657 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:37.416+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:37.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:37.244+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:37.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:37.244+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:37.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:37.244+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:37.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:37.245+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:37.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:37.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:37.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:37.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:37.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:37.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:37.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:37.660+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:38.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:38.224+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:38.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:38.224+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:38.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:38.224+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:38.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:38.224+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:38.650 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:38.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:38.650 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:38.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:38.650 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:38.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:38.650 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:38.374+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:38.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:38.651+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:38.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:38.651+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:38.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:38.651+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:38.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:38.652+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:39.361 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:39.177+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:39.361 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:39.177+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:39.361 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:39.177+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:39.361 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:39.178+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:39.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:39.605+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:39.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:39.605+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:39.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:39.605+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:39.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:39.606+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:39.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:39.362+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:39.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:39.362+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:39.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:39.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:39.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:39.364+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:40.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:40.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:40.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:40.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:40.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:40.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:40.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:40.374+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:40.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:40.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:40.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:40.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:40.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:40.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:40.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:40.221+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:40.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:40.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:40.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:40.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:40.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:40.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:40.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:40.647+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:41.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:41.194+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:41.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:41.194+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:41.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:41.194+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:41.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:41.194+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:41.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:41.387+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:41.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:41.387+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:41.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:41.387+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:41.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:41.387+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:41.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:41.669+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:41.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:41.669+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:41.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:41.669+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:41.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:41.669+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:42.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:42.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:42.338 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:42.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:42.338 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:42.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:42.338 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:42.210+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:42.668 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:52:42.339+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 45222 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:52:42.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:42.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:42.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:42.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:42.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:42.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:42.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:42.419+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:42.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:42.670+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:42.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:42.670+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:42.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:42.670+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:42.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:42.671+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:43.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:43.222+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:43.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:43.222+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:43.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:43.222+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:43.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:43.223+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:43.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:43.449+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:43.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:43.449+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:43.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:43.449+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:43.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:43.450+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:43.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:43.691+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:43.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:43.691+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:43.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:43.691+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:43.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:43.692+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:44.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:44.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:44.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:44.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:44.655 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:44.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:44.655 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:44.253+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:44.655 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:44.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:44.655 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:44.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:44.655 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:44.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:44.655 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:44.451+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:44.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:44.656+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:44.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:44.656+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:44.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:44.656+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:44.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:44.657+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:45.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:45.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:45.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:45.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:45.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:45.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:45.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:45.232+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:45.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:45.650+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:45.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:45.650+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:45.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:45.650+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:45.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:45.651+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:45.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:45.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:45.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:45.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:45.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:45.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:45.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:45.484+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:46.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:46.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:46.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:46.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:46.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:46.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:46.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:46.262+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:46.675 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:46.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:46.675 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:46.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:46.676 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:46.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:46.676 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:46.501+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:46.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:46.677+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:46.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:46.677+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:46.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:46.677+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:46.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:46.677+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:47.549 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:52:47.344+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 45268 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:52:47.550 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:47.266+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:47.550 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:47.266+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:47.550 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:47.266+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:47.550 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:47.267+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:47.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:47.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:47.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:47.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:47.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:47.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:47.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:47.643+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:47.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:47.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:47.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:47.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:47.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:47.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:47.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:47.553+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:48.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:48.258+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:48.598 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:48.258+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:48.598 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:48.258+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:48.598 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:48.259+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:48.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:48.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:48.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:48.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:48.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:48.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:48.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:48.681+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:48.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:48.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:48.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:48.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:48.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:48.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:48.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:48.600+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:49.622 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:49.226+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:49.622 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:49.226+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:49.622 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:49.226+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:49.622 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:49.227+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:49.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:49.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:49.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:49.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:49.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:49.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:49.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:49.625+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:49.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:49.693+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:49.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:49.693+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:49.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:49.693+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:49.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:49.694+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:50.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:50.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:50.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:50.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:50.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:50.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:50.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:50.249+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:50.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:50.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:50.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:50.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:50.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:50.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:50.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:50.639+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:50.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:50.691+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:50.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:50.691+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:50.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:50.691+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:50.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:50.691+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:51.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:51.207+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:51.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:51.207+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:51.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:51.207+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:51.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:51.207+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:51.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:51.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:51.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:51.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:51.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:51.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:51.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:51.646+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:51.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:51.720+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:51.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:51.720+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:51.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:51.720+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:51.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:51.720+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:52.468 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:52:52.350+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 45313 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:52:52.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:52.204+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:52.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:52.204+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:52.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:52.204+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:52.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:52.205+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:52.826 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:52.613+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:52.826 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:52.613+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:52.826 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:52.613+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:52.826 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:52.614+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:52.826 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:52.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:52.826 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:52.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:52.826 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:52.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:52.827 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:52.683+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:53.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:53.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:53.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:53.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:53.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:53.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:53.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:53.216+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:53.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:53.695+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:53.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:53.695+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:53.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:53.695+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:53.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:53.696+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:53.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:53.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:53.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:53.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:53.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:53.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:53.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:53.632+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:54.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:54.201+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:54.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:54.201+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:54.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:54.201+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:54.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:54.201+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:54.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:54.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:54.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:54.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:54.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:54.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:54.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:54.668+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:54.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:54.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:54.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:54.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:54.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:54.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:54.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:54.679+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:55.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:55.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:55.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:55.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:55.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:55.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:55.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:55.215+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:55.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:55.690+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:55.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:55.690+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:55.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:55.690+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:55.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:55.691+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:55.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:55.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:55.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:55.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:55.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:55.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:55.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:55.685+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:56.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:56.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:56.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:56.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:56.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:56.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:56.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:56.199+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:56.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:56.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:56.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:56.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:56.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:56.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:56.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:56.699+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:56.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:56.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:56.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:56.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:56.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:56.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:56.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:56.695+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:57.468 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:52:57.356+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 45359 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:52:57.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:57.166+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:57.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:57.166+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:57.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:57.166+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:57.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:57.167+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:57.741 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:57.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:57.741 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:57.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:57.741 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:57.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:57.741 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:57.707+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:58.133 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:57.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:58.133 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:57.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:58.133 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:57.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:58.134 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:57.743+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:58.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:58.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:58.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:58.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:58.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:58.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:58.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:58.135+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:58.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:58.771+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:58.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:58.771+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:58.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:58.771+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:58.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:58.772+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:58.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:58.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:58.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:58.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:58.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:58.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:58.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:58.666+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:59.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:59.114+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:59.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:59.114+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:59.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:59.114+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:52:59.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:52:59.115+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:59.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:59.792+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:59.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:59.792+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:59.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:59.792+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:52:59.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:52:59.793+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:52:59.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:59.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:59.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:59.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:59.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:59.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:52:59.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:52:59.634+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:00.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:00.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:00.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:00.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:00.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:00.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:00.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:00.113+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:00.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:00.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:00.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:00.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:00.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:00.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:00.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:00.659+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:01.146 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:00.755+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:01.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:00.755+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:01.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:00.755+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:01.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:00.756+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:01.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:01.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:01.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:01.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:01.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:01.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:01.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:01.148+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:01.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:01.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:01.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:01.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:01.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:01.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:01.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:01.690+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:01.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:01.735+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:01.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:01.735+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:01.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:01.735+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:01.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:01.736+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:02.468 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:53:02.362+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 45403 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:53:02.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:02.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:02.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:02.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:02.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:02.156+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:02.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:02.156+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:02.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:02.710+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:02.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:02.710+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:02.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:02.710+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:02.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:02.710+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:02.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:02.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:02.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:02.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:02.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:02.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:02.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:02.649+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:03.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:03.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:03.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:03.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:03.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:03.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:03.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:03.174+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:03.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:03.704+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:03.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:03.704+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:03.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:03.704+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:03.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:03.705+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:03.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:03.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:03.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:03.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:03.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:03.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:03.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:03.630+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:04.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:04.211+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:04.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:04.211+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:04.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:04.211+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:04.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:04.211+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:04.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:04.716+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:04.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:04.716+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:04.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:04.716+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:04.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:04.717+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:04.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:04.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:04.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:04.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:04.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:04.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:04.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:04.654+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:05.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:05.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:05.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:05.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:05.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:05.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:05.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:05.233+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:05.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:05.716+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:05.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:05.716+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:05.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:05.716+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:05.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:05.716+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:05.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:05.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:05.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:05.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:05.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:05.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:05.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:05.677+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:06.555 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:06.268+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:06.555 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:06.268+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:06.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:06.268+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:06.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:06.268+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:06.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:06.724+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:06.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:06.724+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:06.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:06.724+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:06.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:06.725+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:06.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:06.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:06.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:06.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:06.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:06.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:06.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:06.721+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:07.698 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:53:07.367+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 45449 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:53:07.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:07.242+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:07.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:07.242+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:07.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:07.242+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:07.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:07.242+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:07.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:07.699+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:07.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:07.699+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:07.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:07.699+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:07.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:07.700+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:07.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:07.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:07.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:07.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:07.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:07.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:07.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:07.757+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:08.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:08.275+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:08.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:08.275+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:08.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:08.275+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:08.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:08.275+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:09.218 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:08.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:09.218 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:08.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:09.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:08.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:09.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:08.782+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:09.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:08.727+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:09.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:08.727+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:09.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:08.727+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:09.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:08.728+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:09.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:09.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:09.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:09.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:09.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:09.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:09.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:09.303+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:10.153 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:09.741+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:10.153 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:09.741+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:10.153 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:09.741+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:10.153 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:09.742+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:10.153 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:09.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:10.154 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:09.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:10.154 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:09.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:10.154 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:09.738+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:10.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:10.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:10.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:10.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:10.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:10.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:10.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:10.313+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:11.218 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:10.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:11.218 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:10.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:11.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:10.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:11.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:10.758+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:11.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:10.737+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:11.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:10.737+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:11.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:10.737+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:11.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:10.737+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:11.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:11.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:11.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:11.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:11.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:11.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:11.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:11.313+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:12.218 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:11.762+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:12.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:11.762+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:12.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:11.762+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:12.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:11.763+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:12.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:11.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:12.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:11.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:12.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:11.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:12.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:11.771+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:12.699 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:53:12.373+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 45494 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:53:12.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:12.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:12.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:12.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:12.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:12.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:12.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:12.315+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:12.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:12.713+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:12.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:12.713+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:12.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:12.713+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:12.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:12.714+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:12.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:12.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:12.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:12.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:12.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:12.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:12.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:12.753+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:13.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:13.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:13.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:13.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:13.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:13.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:13.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:13.311+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:13.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:13.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:13.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:13.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:13.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:13.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:13.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:13.687+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:13.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:13.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:13.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:13.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:13.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:13.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:13.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:13.748+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:14.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:14.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:14.662 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:14.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:14.662 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:14.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:14.662 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:14.279+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:14.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:14.774+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:14.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:14.774+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:14.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:14.774+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:14.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:14.775+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:14.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:14.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:14.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:14.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:14.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:14.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:14.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:14.663+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:15.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:15.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:15.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:15.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:15.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:15.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:15.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:15.311+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:15.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:15.668+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:15.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:15.668+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:15.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:15.668+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:15.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:15.670+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:15.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:15.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:15.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:15.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:15.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:15.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:15.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:15.737+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:16.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:16.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:16.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:16.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:16.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:16.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:16.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:16.278+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:16.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:16.713+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:16.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:16.713+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:16.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:16.713+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:16.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:16.714+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:16.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:16.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:16.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:16.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:16.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:16.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:16.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:16.788+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:17.718 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:53:17.379+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 45540 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:53:17.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:17.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:17.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:17.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:17.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:17.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:17.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:17.261+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:18.062 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:17.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:18.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:17.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:18.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:17.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:18.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:17.821+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:18.063 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:17.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:18.063 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:17.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:18.063 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:17.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:18.063 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:17.723+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:18.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:18.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:18.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:18.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:18.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:18.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:18.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:18.215+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:18.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:18.689+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:18.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:18.689+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:18.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:18.689+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:18.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:18.689+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:18.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:18.778+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:18.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:18.778+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:18.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:18.778+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:18.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:18.778+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:19.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:19.185+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:19.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:19.185+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:19.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:19.185+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:19.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:19.185+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:20.148 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:19.729+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:20.148 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:19.729+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:20.148 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:19.729+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:20.148 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:19.730+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:20.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:19.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:20.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:19.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:20.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:19.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:20.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:19.791+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:20.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:20.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:20.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:20.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:20.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:20.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:20.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:20.150+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:21.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:20.770+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:21.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:20.770+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:21.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:20.770+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:21.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:20.770+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:21.178 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:20.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:21.178 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:20.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:21.178 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:20.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:21.178 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:20.744+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:21.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:21.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:21.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:21.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:21.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:21.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:21.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:21.179+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:22.146 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:21.817+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:22.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:21.817+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:22.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:21.817+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:22.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:21.818+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:22.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:21.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:22.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:21.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:22.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:21.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:22.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:21.749+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:22.406 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:53:22.386+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 45584 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:53:22.406 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:22.147+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:22.406 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:22.147+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:22.406 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:22.147+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:22.406 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:22.148+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:23.109 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:22.732+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:23.109 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:22.732+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:23.109 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:22.732+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:23.109 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:22.733+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:23.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:22.825+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:23.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:22.825+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:23.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:22.825+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:23.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:22.826+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:23.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:23.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:23.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:23.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:23.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:23.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:23.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:23.144+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:23.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:23.796+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:23.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:23.796+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:23.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:23.796+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:23.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:23.798+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:23.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:23.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:23.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:23.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:23.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:23.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:23.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:23.758+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:24.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:24.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:24.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:24.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:24.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:24.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:24.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:24.109+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:25.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:24.820+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:25.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:24.820+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:25.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:24.820+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:25.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:24.820+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:25.068 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:24.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:25.068 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:24.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:25.068 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:24.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:25.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:24.806+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:25.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:25.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:25.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:25.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:25.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:25.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:25.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:25.069+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:26.218 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:25.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:26.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:25.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:26.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:25.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:26.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:25.827+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:26.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:25.847+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:26.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:25.847+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:26.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:25.847+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:26.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:25.848+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:26.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:26.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:26.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:26.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:26.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:26.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:26.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:26.031+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:27.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:27.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:27.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:27.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:27.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:27.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:27.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:27.002+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:27.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:26.875+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:27.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:26.875+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:27.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:26.875+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:27.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:26.876+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:27.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:26.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:27.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:26.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:27.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:26.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:27.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:26.800+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:27.718 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:53:27.392+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 45630 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:53:28.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:27.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:28.061 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:27.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:28.061 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:27.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:28.061 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:27.777+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:28.061 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:27.853+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:28.061 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:27.853+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:28.061 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:27.853+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:28.061 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:27.853+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:28.062 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:28.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:28.062 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:28.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:28.062 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:28.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:28.062 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:28.024+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:29.053 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:28.848+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:29.053 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:28.848+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:29.053 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:28.848+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:29.053 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:28.849+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:29.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:28.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:29.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:28.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:29.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:28.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:29.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:28.782+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:29.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:29.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:29.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:29.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:29.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:29.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:29.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:29.054+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:30.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:29.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:30.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:29.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:30.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:29.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:30.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:29.838+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:30.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:29.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:30.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:29.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:30.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:29.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:30.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:29.738+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:30.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:30.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:30.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:30.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:30.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:30.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:30.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:30.070+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:31.101 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:30.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:31.101 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:30.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:31.101 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:30.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:31.101 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:30.839+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:31.101 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:30.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:31.102 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:30.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:31.102 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:30.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:31.102 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:30.753+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:31.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:31.102+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:31.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:31.102+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:31.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:31.102+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:31.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:31.102+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:31.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:31.832+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:31.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:31.832+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:31.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:31.832+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:31.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:31.833+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:31.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:31.708+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:31.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:31.708+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:31.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:31.708+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:31.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:31.709+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:32.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:32.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:32.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:32.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:32.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:32.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:32.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:32.058+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:32.712 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:53:32.396+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 45675 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:53:32.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:32.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:32.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:32.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:32.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:32.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:32.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:32.714+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:32.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:32.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:32.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:32.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:32.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:32.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:32.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:32.861+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:33.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:33.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:33.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:33.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:33.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:33.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:33.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:33.053+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:33.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:33.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:33.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:33.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:33.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:33.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:33.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:33.891+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:33.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:33.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:33.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:33.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:33.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:33.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:33.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:33.765+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:34.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:34.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:34.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:34.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:34.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:34.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:34.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:34.084+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:35.083 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:34.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:35.083 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:34.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:35.083 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:34.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:35.083 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:34.801+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:35.083 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:34.854+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:35.084 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:34.854+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:35.084 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:34.854+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:35.084 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:34.855+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:35.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:35.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:35.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:35.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:35.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:35.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:35.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:35.084+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:36.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:35.854+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:36.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:35.854+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:36.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:35.854+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:36.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:35.855+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:36.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:35.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:36.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:35.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:36.090 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:35.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:36.090 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:35.794+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:36.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:36.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:36.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:36.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:36.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:36.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:36.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:36.090+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:37.052 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:36.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:37.052 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:36.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:37.053 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:36.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:37.053 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:36.837+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:37.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:36.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:37.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:36.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:37.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:36.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:37.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:36.762+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:37.401 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:37.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:37.401 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:37.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:37.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:37.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:37.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:37.053+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:37.714 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:53:37.402+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 45720 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:53:37.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:37.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:37.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:37.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:37.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:37.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:37.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:37.716+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:37.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:37.855+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:37.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:37.855+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:37.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:37.855+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:37.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:37.856+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:38.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:38.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:38.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:38.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:38.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:38.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:38.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:38.049+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:38.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:38.675+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:38.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:38.675+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:38.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:38.675+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:38.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:38.676+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:38.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:38.886+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:38.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:38.886+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:38.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:38.886+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:38.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:38.887+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:39.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:39.089+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:39.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:39.089+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:39.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:39.089+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:39.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:39.089+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:39.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:39.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:39.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:39.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:39.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:39.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:39.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:39.894+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:39.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:39.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:39.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:39.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:39.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:39.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:39.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:39.682+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:40.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:40.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:40.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:40.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:40.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:40.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:40.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:40.114+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:40.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:40.914+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:40.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:40.914+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:40.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:40.914+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:40.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:40.916+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:40.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:40.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:40.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:40.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:40.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:40.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:40.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:40.713+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:41.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:41.109+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:41.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:41.109+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:41.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:41.109+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:41.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:41.109+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:42.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:41.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:42.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:41.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:42.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:41.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:42.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:41.955+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:42.150 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:41.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:42.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:41.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:42.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:41.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:42.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:41.732+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:42.406 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:42.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:42.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:42.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:42.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:42.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:42.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:42.151+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:42.722 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:53:42.407+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 45757 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:53:43.101 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:42.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:43.101 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:42.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:43.101 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:42.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:43.101 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:42.921+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:43.101 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:42.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:43.101 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:42.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:43.101 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:42.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:43.101 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:42.729+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:43.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:43.101+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:43.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:43.101+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:43.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:43.101+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:43.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:43.102+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:43.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:43.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:43.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:43.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:43.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:43.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:43.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:43.726+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:43.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:43.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:43.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:43.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:43.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:43.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:43.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:43.901+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:44.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:44.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:44.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:44.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:44.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:44.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:44.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:44.114+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:44.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:44.889+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:44.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:44.889+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:44.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:44.889+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:44.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:44.890+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:44.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:44.681+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:44.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:44.681+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:44.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:44.681+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:44.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:44.682+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:45.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:45.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:45.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:45.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:45.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:45.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:45.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:45.069+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:46.050 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:45.876+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:46.050 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:45.876+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:46.050 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:45.876+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:46.050 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:45.876+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:46.051 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:45.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:46.051 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:45.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:46.051 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:45.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:46.051 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:45.721+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:46.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:46.050+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:46.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:46.050+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:46.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:46.050+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:46.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:46.050+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:46.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:46.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:46.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:46.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:46.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:46.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:46.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:46.831+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:46.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:46.714+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:46.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:46.714+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:46.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:46.714+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:46.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:46.715+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:47.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:47.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:47.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:47.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:47.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:47.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:47.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:47.022+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:47.698 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:53:47.413+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 45803 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:53:47.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:47.857+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:47.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:47.857+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:47.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:47.857+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:47.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:47.857+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:47.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:47.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:47.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:47.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:47.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:47.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:47.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:47.699+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:48.338 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:47.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:48.339 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:47.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:48.339 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:47.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:48.339 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:47.994+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:49.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:48.831+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:49.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:48.831+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:49.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:48.831+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:49.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:48.832+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:49.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:48.959+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:49.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:48.959+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:49.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:48.959+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:49.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:48.960+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:49.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:48.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:49.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:48.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:49.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:48.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:49.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:48.743+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:49.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:49.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:49.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:49.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:49.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:49.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:49.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:49.876+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:49.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:49.697+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:49.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:49.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:49.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:49.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:49.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:49.699+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:50.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:49.984+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:50.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:49.984+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:50.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:49.984+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:50.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:49.984+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:51.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:50.859+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:51.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:50.859+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:51.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:50.859+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:51.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:50.860+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:51.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:50.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:51.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:50.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:51.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:50.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:51.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:50.739+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:51.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:50.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:51.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:50.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:51.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:50.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:51.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:50.936+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:52.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:51.855+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:52.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:51.855+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:52.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:51.855+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:52.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:51.857+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:52.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:51.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:52.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:51.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:52.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:51.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:52.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:51.746+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:52.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:51.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:52.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:51.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:52.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:51.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:52.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:51.940+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:52.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:53:52.418+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 45848 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:53:53.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:52.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:53.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:52.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:53.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:52.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:53.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:52.763+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:53.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:52.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:53.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:52.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:53.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:52.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:53.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:52.948+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:53.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:52.883+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:53.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:52.883+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:53.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:52.883+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:53.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:52.884+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:53.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:53.727+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:53.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:53.727+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:53.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:53.727+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:53.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:53.728+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:53.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:53.901+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:53.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:53.901+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:53.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:53.901+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:53.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:53.902+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:54.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:53.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:54.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:53.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:54.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:53.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:54.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:53.993+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:54.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:54.896+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:54.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:54.896+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:54.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:54.896+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:54.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:54.897+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:54.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:54.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:54.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:54.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:54.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:54.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:54.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:54.705+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:55.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:54.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:55.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:54.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:55.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:54.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:55.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:54.969+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:55.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:55.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:55.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:55.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:55.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:55.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:55.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:55.694+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:55.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:55.889+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:55.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:55.889+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:55.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:55.889+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:55.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:55.890+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:56.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:55.986+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:56.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:55.986+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:56.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:55.986+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:56.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:55.987+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:56.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:56.918+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:56.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:56.918+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:56.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:56.918+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:56.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:56.919+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:56.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:56.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:56.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:56.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:56.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:56.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:56.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:56.695+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:56.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:56.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:56.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:56.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:56.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:56.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:56.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:56.939+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:57.694 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:53:57.424+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 45895 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:53:57.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:57.695+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:57.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:57.695+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:57.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:57.695+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:57.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:57.696+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:57.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:57.882+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:57.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:57.882+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:57.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:57.882+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:57.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:57.883+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:58.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:57.989+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:58.429 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:57.989+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:58.429 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:57.989+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:58.429 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:57.989+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:59.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:58.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:59.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:58.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:59.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:58.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:53:59.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:58.969+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:59.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:58.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:59.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:58.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:59.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:58.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:53:59.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:58.862+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:53:59.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:58.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:59.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:58.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:59.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:58.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:53:59.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:58.746+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:00.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:59.832+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:00.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:59.832+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:00.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:59.832+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:00.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:53:59.832+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:00.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:59.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:00.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:59.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:00.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:59.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:00.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:53:59.790+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:00.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:59.989+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:00.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:59.989+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:00.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:59.989+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:00.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:53:59.989+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:01.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:00.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:01.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:00.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:01.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:00.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:01.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:00.744+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:01.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:00.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:01.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:00.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:01.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:00.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:01.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:00.967+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:01.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:00.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:01.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:00.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:01.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:00.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:01.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:00.825+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:02.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:01.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:02.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:01.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:02.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:01.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:02.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:01.925+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:02.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:01.846+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:02.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:01.846+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:02.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:01.846+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:02.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:01.848+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:02.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:01.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:02.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:01.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:02.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:01.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:02.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:01.774+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:02.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:54:02.430+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 45939 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:54:03.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:02.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:03.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:02.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:03.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:02.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:03.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:02.772+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:03.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:02.829+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:03.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:02.829+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:03.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:02.829+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:03.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:02.829+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:03.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:02.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:03.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:02.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:03.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:02.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:03.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:02.971+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:04.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:03.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:04.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:03.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:04.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:03.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:04.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:03.929+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:04.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:03.793+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:04.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:03.793+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:04.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:03.793+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:04.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:03.793+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:04.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:03.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:04.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:03.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:04.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:03.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:04.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:03.758+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:05.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:04.775+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:05.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:04.775+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:05.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:04.775+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:05.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:04.775+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:05.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:04.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:05.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:04.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:05.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:04.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:05.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:04.783+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:05.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:04.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:05.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:04.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:05.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:04.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:05.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:04.927+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:06.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:05.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:06.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:05.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:06.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:05.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:06.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:05.901+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:06.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:05.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:06.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:05.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:06.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:05.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:06.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:05.759+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:06.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:05.749+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:06.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:05.749+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:06.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:05.749+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:06.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:05.750+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:06.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:06.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:06.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:06.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:06.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:06.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:06.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:06.703+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:06.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:06.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:06.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:06.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:06.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:06.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:06.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:06.739+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:06.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:06.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:06.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:06.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:06.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:06.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:06.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:06.931+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:07.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:54:07.434+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 45985 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:54:07.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:07.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:07.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:07.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:07.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:07.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:07.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:07.659+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:08.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:07.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:08.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:07.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:08.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:07.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:08.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:07.940+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:08.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:07.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:08.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:07.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:08.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:07.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:08.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:07.726+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:08.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:08.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:08.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:08.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:08.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:08.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:08.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:08.676+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:08.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:08.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:08.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:08.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:08.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:08.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:08.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:08.762+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:09.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:08.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:09.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:08.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:09.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:08.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:09.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:08.966+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:09.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:09.783+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:09.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:09.783+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:09.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:09.783+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:09.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:09.784+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:09.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:09.687+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:09.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:09.687+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:09.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:09.687+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:09.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:09.688+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:10.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:09.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:10.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:09.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:10.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:09.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:10.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:09.973+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:10.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:10.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:10.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:10.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:10.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:10.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:10.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:10.677+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:10.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:10.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:10.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:10.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:10.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:10.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:10.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:10.776+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:11.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:10.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:11.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:10.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:11.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:10.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:11.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:10.945+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:11.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:11.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:11.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:11.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:11.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:11.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:11.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:11.759+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:11.942 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:11.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:11.942 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:11.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:11.942 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:11.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:11.942 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:11.708+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:12.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:11.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:12.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:11.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:12.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:11.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:12.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:11.940+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:12.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:54:12.440+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 46030 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:54:12.984 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:12.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:12.984 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:12.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:12.984 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:12.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:12.984 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:12.726+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:12.984 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:12.757+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:12.985 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:12.757+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:12.985 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:12.757+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:12.985 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:12.758+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:13.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:12.983+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:13.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:12.983+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:13.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:12.983+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:13.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:12.984+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:13.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:13.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:13.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:13.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:13.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:13.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:13.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:13.707+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:13.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:13.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:13.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:13.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:13.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:13.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:13.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:13.723+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:14.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:13.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:14.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:13.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:14.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:13.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:14.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:13.966+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:15.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:14.737+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:15.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:14.737+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:15.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:14.737+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:15.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:14.738+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:15.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:14.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:15.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:14.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:15.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:14.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:15.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:14.741+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:15.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:14.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:15.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:14.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:15.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:14.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:15.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:14.936+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:16.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:15.785+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:16.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:15.785+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:16.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:15.785+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:16.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:15.786+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:16.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:15.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:16.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:15.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:16.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:15.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:16.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:15.772+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:16.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:15.903+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:16.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:15.903+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:16.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:15.903+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:16.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:15.904+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:17.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:16.740+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:17.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:16.740+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:17.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:16.740+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:17.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:16.742+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:17.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:16.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:17.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:16.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:17.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:16.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:17.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:16.901+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:17.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:16.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:17.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:16.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:17.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:16.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:17.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:16.756+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:17.696 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:54:17.446+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 46076 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:54:17.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:17.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:17.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:17.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:17.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:17.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:17.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:17.759+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:17.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:17.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:17.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:17.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:17.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:17.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:17.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:17.914+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:17.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:17.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:17.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:17.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:17.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:17.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:17.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:17.697+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:18.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:18.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:18.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:18.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:18.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:18.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:18.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:18.744+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:18.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:18.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:18.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:18.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:18.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:18.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:18.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:18.771+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:19.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:18.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:19.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:18.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:19.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:18.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:19.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:18.900+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:20.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:19.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:20.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:19.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:20.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:19.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:20.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:19.732+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:20.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:19.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:20.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:19.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:20.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:19.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:20.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:19.816+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:20.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:19.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:20.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:19.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:20.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:19.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:20.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:19.935+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:21.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:20.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:21.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:20.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:21.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:20.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:21.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:20.732+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:21.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:20.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:21.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:20.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:21.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:20.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:21.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:20.854+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:21.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:20.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:21.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:20.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:21.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:20.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:21.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:20.924+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:21.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:21.700+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:21.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:21.700+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:21.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:21.700+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:21.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:21.701+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:21.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:21.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:21.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:21.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:21.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:21.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:21.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:21.862+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:22.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:21.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:22.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:21.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:22.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:21.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:22.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:21.969+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:22.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:54:22.452+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 46120 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:54:23.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:22.738+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:23.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:22.738+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:23.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:22.738+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:23.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:22.739+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:23.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:22.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:23.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:22.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:23.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:22.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:23.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:22.840+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:23.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:22.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:23.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:22.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:23.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:22.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:23.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:22.987+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:23.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:23.730+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:23.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:23.730+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:23.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:23.730+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:23.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:23.731+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:23.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:23.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:23.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:23.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:23.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:23.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:23.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:23.887+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:24.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:23.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:24.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:23.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:24.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:23.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:24.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:23.966+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:25.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:24.771+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:25.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:24.771+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:25.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:24.771+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:25.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:24.772+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:25.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:24.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:25.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:24.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:25.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:24.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:25.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:24.929+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:25.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:24.926+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:25.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:24.926+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:25.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:24.926+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:25.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:24.926+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:26.121 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:25.820+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:26.121 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:25.820+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:26.121 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:25.820+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:26.121 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:25.821+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:26.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:25.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:26.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:25.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:26.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:25.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:26.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:25.955+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:26.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:25.945+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:26.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:25.945+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:26.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:25.945+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:26.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:25.946+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:27.005 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:26.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:27.006 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:26.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:27.006 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:26.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:27.006 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:26.828+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:27.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:26.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:27.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:26.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:27.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:26.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:27.007 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:26.936+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:27.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:26.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:27.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:26.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:27.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:26.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:27.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:26.968+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:27.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:54:27.459+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 46166 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:54:28.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:27.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:28.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:27.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:28.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:27.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:28.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:27.874+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:28.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:27.904+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:28.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:27.904+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:28.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:27.904+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:28.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:27.904+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:28.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:27.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:28.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:27.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:28.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:27.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:28.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:27.990+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:28.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:28.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:28.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:28.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:28.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:28.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:28.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:28.909+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:28.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:28.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:28.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:28.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:28.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:28.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:28.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:28.851+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:29.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:29.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:29.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:29.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:29.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:29.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:29.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:29.010+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:30.160 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:29.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:30.160 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:29.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:30.160 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:29.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:30.160 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:29.894+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:30.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:29.879+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:30.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:29.879+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:30.161 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:29.879+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:30.161 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:29.879+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:30.161 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:29.970+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:30.161 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:29.970+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:30.161 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:29.970+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:30.161 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:29.971+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:31.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:30.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:31.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:30.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:31.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:30.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:31.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:30.921+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:31.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:30.923+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:31.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:30.923+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:31.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:30.923+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:31.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:30.924+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:31.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:30.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:31.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:30.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:31.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:30.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:31.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:30.875+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:32.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:31.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:32.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:31.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:32.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:31.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:32.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:31.961+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:32.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:31.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:32.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:31.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:32.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:31.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:32.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:31.876+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:32.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:31.876+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:32.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:31.876+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:32.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:31.876+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:32.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:31.876+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:32.720 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:54:32.464+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 46212 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:54:33.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:32.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:33.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:32.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:33.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:32.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:33.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:32.846+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:33.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:32.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:33.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:32.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:33.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:32.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:33.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:32.850+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:33.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:32.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:33.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:32.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:33.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:32.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:33.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:32.932+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:33.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:33.864+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:33.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:33.864+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:33.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:33.864+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:33.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:33.865+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:33.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:33.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:33.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:33.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:33.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:33.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:33.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:33.830+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:33.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:33.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:33.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:33.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:33.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:33.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:33.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:33.891+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:35.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:34.847+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:35.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:34.847+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:35.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:34.847+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:35.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:34.847+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:35.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:34.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:35.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:34.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:35.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:34.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:35.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:34.861+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:35.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:34.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:35.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:34.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:35.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:34.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:35.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:34.825+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:35.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:35.823+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:35.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:35.823+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:35.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:35.823+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:35.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:35.824+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:35.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:35.837+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:35.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:35.837+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:35.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:35.837+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:35.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:35.837+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:35.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:35.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:35.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:35.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:35.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:35.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:35.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:35.805+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:37.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:36.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:37.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:36.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:37.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:36.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:37.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:36.825+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:37.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:36.785+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:37.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:36.785+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:37.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:36.785+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:37.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:36.786+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:37.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:36.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:37.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:36.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:37.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:36.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:37.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:36.853+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:37.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:54:37.470+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 46257 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:54:38.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:37.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:38.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:37.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:38.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:37.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:38.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:37.780+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:38.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:37.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:38.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:37.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:38.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:37.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:38.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:37.815+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:38.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:37.830+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:38.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:37.830+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:38.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:37.830+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:38.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:37.830+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:38.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:38.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:38.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:38.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:38.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:38.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:38.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:38.781+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:38.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:38.774+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:38.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:38.774+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:38.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:38.774+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:38.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:38.775+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:38.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:38.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:38.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:38.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:38.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:38.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:38.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:38.795+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:40.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:39.737+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:40.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:39.737+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:40.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:39.737+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:40.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:39.738+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:40.161 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:39.784+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:40.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:39.784+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:40.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:39.784+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:40.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:39.785+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:40.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:39.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:40.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:39.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:40.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:39.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:40.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:39.827+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:40.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:40.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:40.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:40.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:40.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:40.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:40.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:40.739+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:40.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:40.690+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:40.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:40.690+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:40.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:40.690+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:40.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:40.690+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:40.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:40.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:40.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:40.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:40.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:40.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:40.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:40.847+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:41.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:41.705+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:41.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:41.705+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:41.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:41.705+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:41.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:41.705+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:41.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:41.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:41.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:41.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:41.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:41.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:41.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:41.848+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:41.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:41.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:41.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:41.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:41.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:41.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:41.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:41.713+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:42.740 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:54:42.476+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 46302 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:54:42.741 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:42.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:42.741 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:42.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:42.741 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:42.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:42.741 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:42.674+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:43.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:42.810+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:43.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:42.810+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:43.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:42.810+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:43.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:42.811+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:43.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:42.740+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:43.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:42.740+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:43.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:42.740+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:43.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:42.741+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:43.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:43.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:43.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:43.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:43.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:43.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:43.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:43.836+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:43.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:43.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:43.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:43.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:43.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:43.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:43.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:43.669+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:43.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:43.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:43.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:43.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:43.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:43.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:43.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:43.770+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:44.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:44.808+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:44.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:44.808+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:44.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:44.808+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:44.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:44.808+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:44.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:44.730+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:44.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:44.730+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:44.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:44.730+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:44.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:44.731+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:44.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:44.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:44.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:44.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:44.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:44.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:44.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:44.681+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:45.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:45.749+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:45.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:45.749+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:45.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:45.749+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:45.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:45.750+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:45.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:45.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:45.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:45.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:45.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:45.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:45.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:45.679+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:45.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:45.790+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:45.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:45.790+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:45.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:45.790+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:45.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:45.790+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:46.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:46.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:46.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:46.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:46.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:46.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:46.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:46.713+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:46.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:46.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:46.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:46.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:46.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:46.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:46.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:46.693+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:46.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:46.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:46.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:46.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:46.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:46.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:46.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:46.819+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:47.754 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:54:47.482+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 46347 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:54:47.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:47.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:47.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:47.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:47.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:47.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:47.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:47.717+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:48.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:47.754+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:48.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:47.754+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:48.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:47.754+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:48.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:47.755+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:48.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:47.805+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:48.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:47.805+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:48.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:47.805+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:48.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:47.806+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:49.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:48.759+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:49.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:48.759+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:49.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:48.759+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:49.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:48.760+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:49.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:48.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:49.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:48.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:49.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:48.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:49.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:48.852+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:49.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:48.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:49.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:48.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:49.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:48.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:49.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:48.746+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:50.162 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:49.732+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:50.162 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:49.732+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:50.162 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:49.732+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:50.162 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:49.734+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:50.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:49.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:50.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:49.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:50.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:49.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:50.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:49.875+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:50.163 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:49.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:50.163 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:49.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:50.163 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:49.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:50.164 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:49.792+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:51.005 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:50.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:51.005 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:50.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:51.005 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:50.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:51.005 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:50.923+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:51.006 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:50.748+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:51.006 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:50.748+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:51.006 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:50.748+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:51.006 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:50.748+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:51.006 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:50.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:51.006 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:50.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:51.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:50.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:51.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:50.830+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:51.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:51.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:51.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:51.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:51.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:51.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:51.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:51.931+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:51.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:51.700+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:51.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:51.700+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:51.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:51.700+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:51.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:51.701+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:51.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:51.864+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:51.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:51.864+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:51.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:51.864+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:51.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:51.865+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:52.852 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:54:52.487+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 46393 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:54:52.853 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:52.705+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:52.853 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:52.705+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:52.853 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:52.705+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:52.853 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:52.706+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:53.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:52.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:53.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:52.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:53.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:52.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:53.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:52.912+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:53.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:52.852+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:53.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:52.852+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:53.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:52.852+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:53.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:52.853+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:53.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:53.893+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:53.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:53.893+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:53.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:53.893+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:53.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:53.894+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:53.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:53.736+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:53.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:53.736+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:53.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:53.736+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:53.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:53.737+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:53.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:53.922+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:53.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:53.922+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:53.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:53.922+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:53.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:53.923+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:55.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:54.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:55.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:54.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:55.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:54.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:55.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:54.874+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:55.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:54.759+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:55.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:54.759+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:55.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:54.759+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:55.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:54.760+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:55.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:54.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:55.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:54.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:55.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:54.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:55.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:54.887+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:56.122 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:55.727+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:56.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:55.727+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:56.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:55.727+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:56.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:55.728+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:56.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:55.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:56.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:55.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:56.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:55.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:56.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:55.900+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:56.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:55.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:56.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:55.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:56.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:55.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:56.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:55.867+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:57.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:56.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:57.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:56.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:57.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:56.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:57.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:56.852+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:57.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:56.765+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:57.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:56.765+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:57.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:56.765+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:57.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:56.766+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:57.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:56.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:57.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:56.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:57.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:56.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:57.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:56.868+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:57.762 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:54:57.494+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 46438 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:54:58.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:57.868+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:58.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:57.868+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:58.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:57.868+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:58.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:57.869+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:58.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:57.761+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:58.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:57.761+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:58.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:57.761+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:58.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:57.763+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:58.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:57.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:58.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:57.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:58.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:57.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:58.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:57.852+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:59.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:58.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:59.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:58.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:59.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:58.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:54:59.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:58.831+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:59.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:58.815+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:59.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:58.815+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:59.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:58.815+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:54:59.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:58.815+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:54:59.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:58.778+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:59.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:58.778+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:59.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:58.778+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:54:59.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:58.779+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:00.163 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:59.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:00.163 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:59.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:00.163 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:59.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:00.163 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:54:59.732+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:00.164 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:59.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:00.164 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:59.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:00.164 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:59.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:00.164 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:54:59.875+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:00.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:59.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:00.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:59.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:00.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:59.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:00.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:54:59.825+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:01.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:00.758+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:01.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:00.758+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:01.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:00.758+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:01.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:00.759+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:01.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:00.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:01.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:00.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:01.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:00.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:01.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:00.871+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:01.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:00.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:01.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:00.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:01.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:00.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:01.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:00.871+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:01.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:01.893+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:01.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:01.893+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:01.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:01.893+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:01.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:01.895+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:01.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:01.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:01.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:01.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:01.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:01.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:01.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:01.849+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:01.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:01.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:01.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:01.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:01.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:01.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:01.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:01.712+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:02.821 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:55:02.500+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 46483 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:55:02.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:02.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:02.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:02.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:02.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:02.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:02.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:02.677+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:03.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:02.933+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:03.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:02.933+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:03.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:02.933+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:03.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:02.934+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:03.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:02.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:03.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:02.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:03.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:02.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:03.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:02.821+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:03.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:03.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:03.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:03.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:03.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:03.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:03.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:03.858+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:03.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:03.703+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:03.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:03.703+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:03.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:03.703+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:03.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:03.704+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:04.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:03.975+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:04.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:03.975+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:04.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:03.975+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:04.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:03.976+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:04.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:04.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:04.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:04.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:04.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:04.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:04.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:04.658+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:04.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:04.888+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:04.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:04.888+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:04.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:04.888+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:04.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:04.888+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:05.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:04.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:05.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:04.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:05.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:04.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:05.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:04.952+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:05.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:05.706+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:05.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:05.706+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:05.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:05.706+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:05.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:05.706+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:05.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:05.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:05.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:05.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:05.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:05.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:05.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:05.936+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:06.365 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:05.972+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:06.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:05.972+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:06.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:05.972+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:06.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:05.972+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:06.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:06.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:06.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:06.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:06.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:06.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:06.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:06.683+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:07.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:06.954+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:07.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:06.954+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:07.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:06.954+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:07.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:06.955+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:07.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:06.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:07.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:06.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:07.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:06.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:07.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:06.955+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:07.970 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:55:07.505+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 46528 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:55:07.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:07.715+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:07.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:07.715+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:07.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:07.715+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:07.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:07.716+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:08.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:07.988+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:08.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:07.988+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:08.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:07.988+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:08.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:07.989+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:08.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:07.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:08.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:07.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:08.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:07.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:08.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:07.988+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:09.073 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:08.996+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:09.073 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:08.996+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:09.073 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:08.996+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:09.073 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:08.996+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:09.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:08.765+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:09.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:08.765+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:09.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:08.765+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:09.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:08.765+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:09.074 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:08.946+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:09.074 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:08.946+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:09.074 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:08.946+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:09.074 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:08.947+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:10.164 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:09.727+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:10.164 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:09.727+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:10.165 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:09.727+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:10.165 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:09.728+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:10.165 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:09.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:10.165 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:09.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:10.165 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:09.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:10.165 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:09.927+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:10.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:09.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:10.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:09.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:10.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:09.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:10.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:09.975+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:10.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:10.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:10.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:10.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:10.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:10.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:10.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:10.930+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:10.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:10.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:10.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:10.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:10.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:10.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:10.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:10.712+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:10.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:10.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:10.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:10.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:10.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:10.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:10.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:10.935+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:12.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:11.730+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:12.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:11.730+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:12.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:11.730+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:12.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:11.731+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:12.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:11.916+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:12.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:11.916+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:12.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:11.916+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:12.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:11.917+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:12.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:11.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:12.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:11.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:12.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:11.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:12.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:11.966+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:12.945 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:55:12.511+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 46574 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:55:12.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:12.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:12.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:12.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:12.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:12.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:12.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:12.699+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:13.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:12.943+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:13.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:12.943+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:13.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:12.943+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:13.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:12.944+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:13.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:12.945+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:13.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:12.945+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:13.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:12.945+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:13.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:12.945+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:13.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:13.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:13.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:13.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:13.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:13.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:13.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:13.900+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:13.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:13.656+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:13.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:13.656+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:13.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:13.656+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:13.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:13.657+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:14.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:13.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:14.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:13.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:14.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:13.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:14.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:13.982+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:14.873 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:14.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:14.873 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:14.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:14.873 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:14.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:14.873 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:14.620+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:15.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:14.871+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:15.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:14.871+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:15.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:14.871+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:15.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:14.872+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:15.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:14.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:15.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:14.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:15.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:14.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:15.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:14.975+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:15.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:15.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:15.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:15.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:15.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:15.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:15.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:15.667+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:16.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:16.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:16.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:16.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:16.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:16.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:16.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:16.015+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:16.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:15.877+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:16.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:15.877+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:16.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:15.877+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:16.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:15.877+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:16.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:16.665+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:16.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:16.665+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:16.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:16.665+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:16.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:16.665+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:16.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:16.912+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:16.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:16.912+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:16.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:16.912+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:16.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:16.912+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:17.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:17.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:17.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:17.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:17.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:17.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:17.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:17.023+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:17.896 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:55:17.516+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 46619 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:55:17.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:17.687+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:17.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:17.687+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:17.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:17.687+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:17.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:17.688+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:18.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:17.896+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:18.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:17.896+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:18.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:17.896+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:18.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:17.897+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:18.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:18.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:18.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:18.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:18.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:18.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:18.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:18.023+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:18.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:18.706+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:18.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:18.706+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:18.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:18.706+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:18.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:18.707+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:18.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:18.900+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:18.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:18.900+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:18.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:18.900+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:18.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:18.901+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:19.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:18.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:19.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:18.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:19.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:18.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:19.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:18.979+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:19.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:19.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:19.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:19.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:19.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:19.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:19.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:19.684+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:19.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:19.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:19.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:19.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:19.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:19.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:19.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:19.893+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:20.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:19.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:20.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:19.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:20.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:19.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:20.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:19.954+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:20.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:20.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:20.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:20.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:20.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:20.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:20.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:20.699+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:20.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:20.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:20.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:20.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:20.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:20.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:20.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:20.918+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:20.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:20.879+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:20.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:20.879+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:20.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:20.879+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:20.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:20.880+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:21.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:21.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:21.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:21.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:21.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:21.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:21.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:21.677+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:21.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:21.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:21.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:21.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:21.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:21.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:21.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:21.883+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:21.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:21.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:21.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:21.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:21.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:21.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:21.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:21.867+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:22.822 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:55:22.523+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 46664 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:55:22.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:22.651+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:22.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:22.651+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:22.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:22.651+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:22.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:22.652+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:23.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:22.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:23.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:22.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:23.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:22.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:23.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:22.823+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:23.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:22.919+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:23.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:22.919+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:23.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:22.919+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:23.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:22.919+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:23.870 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:23.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:23.870 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:23.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:23.871 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:23.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:23.871 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:23.610+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:24.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:23.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:24.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:23.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:24.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:23.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:24.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:23.870+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:24.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:23.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:24.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:23.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:24.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:23.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:24.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:23.906+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:24.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:24.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:24.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:24.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:24.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:24.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:24.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:24.592+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:24.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:24.838+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:24.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:24.838+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:24.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:24.838+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:24.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:24.838+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:25.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:24.925+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:25.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:24.925+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:25.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:24.925+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:25.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:24.926+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:25.873 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:25.614+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:25.873 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:25.614+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:25.873 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:25.614+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:25.873 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:25.615+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:26.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:25.872+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:26.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:25.872+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:26.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:25.872+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:26.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:25.873+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:26.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:25.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:26.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:25.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:26.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:25.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:26.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:25.912+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:26.868 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:26.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:26.868 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:26.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:26.868 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:26.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:26.868 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:26.592+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:27.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:26.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:27.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:26.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:27.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:26.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:27.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:26.959+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:27.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:26.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:27.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:26.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:27.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:26.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:27.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:26.868+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:27.832 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:55:27.528+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 46709 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:55:27.833 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:27.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:27.833 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:27.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:27.833 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:27.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:27.833 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:27.593+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:28.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:27.832+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:28.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:27.832+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:28.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:27.832+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:28.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:27.832+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:28.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:27.909+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:28.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:27.909+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:28.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:27.909+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:28.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:27.910+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:28.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:28.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:28.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:28.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:28.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:28.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:28.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:28.801+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:28.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:28.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:28.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:28.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:28.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:28.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:28.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:28.608+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:29.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:28.945+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:29.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:28.945+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:29.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:28.945+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:29.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:28.946+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:29.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:29.599+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:29.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:29.599+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:29.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:29.599+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:29.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:29.600+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:29.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:29.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:29.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:29.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:29.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:29.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:29.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:29.796+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:30.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:29.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:30.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:29.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:30.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:29.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:30.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:29.953+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:30.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:30.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:30.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:30.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:30.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:30.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:30.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:30.751+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:30.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:30.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:30.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:30.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:30.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:30.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:30.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:30.565+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:31.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:31.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:31.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:31.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:31.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:31.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:31.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:31.002+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:31.793 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:31.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:31.793 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:31.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:31.793 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:31.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:31.793 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:31.529+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:32.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:31.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:32.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:31.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:32.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:31.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:32.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:31.792+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:32.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:31.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:32.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:31.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:32.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:31.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:32.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:31.952+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:32.822 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:55:32.534+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 46755 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:55:32.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:32.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:32.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:32.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:32.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:32.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:32.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:32.550+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:33.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:32.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:33.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:32.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:33.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:32.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:33.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:32.822+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:33.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:32.984+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:33.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:32.984+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:33.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:32.984+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:33.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:32.985+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:33.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:33.587+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:33.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:33.587+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:33.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:33.587+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:33.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:33.588+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:33.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:33.816+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:33.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:33.816+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:33.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:33.816+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:33.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:33.817+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:34.293 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:33.996+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:34.293 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:33.996+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:34.293 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:33.996+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:34.293 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:33.997+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:34.860 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:34.594+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:34.860 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:34.594+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:34.860 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:34.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:34.861 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:34.596+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:35.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:34.859+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:35.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:34.859+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:35.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:34.859+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:35.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:34.860+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:35.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:34.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:35.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:34.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:35.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:34.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:35.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:34.999+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:35.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:35.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:35.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:35.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:35.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:35.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:35.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:35.605+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:36.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:35.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:36.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:35.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:36.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:35.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:36.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:35.883+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:36.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:35.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:36.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:35.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:36.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:35.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:36.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:35.966+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:36.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:36.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:36.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:36.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:36.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:36.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:36.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:36.582+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:37.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:36.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:37.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:36.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:37.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:36.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:37.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:36.852+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:37.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:36.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:37.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:36.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:37.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:36.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:37.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:36.958+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:37.874 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:55:37.540+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 46800 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:55:37.874 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:37.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:37.874 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:37.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:37.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:37.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:37.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:37.534+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:38.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:37.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:38.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:37.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:38.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:37.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:38.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:37.874+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:38.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:37.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:38.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:37.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:38.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:37.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:38.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:37.947+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:38.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:38.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:38.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:38.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:38.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:38.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:38.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:38.546+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:39.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:38.904+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:39.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:38.904+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:39.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:38.904+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:39.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:38.904+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:39.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:38.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:39.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:38.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:39.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:38.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:39.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:38.875+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:39.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:39.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:39.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:39.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:39.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:39.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:39.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:39.551+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:40.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:39.920+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:40.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:39.920+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:40.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:39.920+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:40.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:39.920+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:40.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:39.902+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:40.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:39.902+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:40.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:39.902+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:40.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:39.903+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:40.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:40.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:40.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:40.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:40.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:40.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:40.891 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:40.572+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:41.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:40.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:41.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:40.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:41.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:40.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:41.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:40.890+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:41.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:40.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:41.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:40.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:41.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:40.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:41.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:40.949+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:41.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:41.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:41.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:41.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:41.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:41.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:41.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:41.548+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:42.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:41.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:42.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:41.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:42.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:41.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:42.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:41.893+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:42.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:41.964+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:42.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:41.964+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:42.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:41.964+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:42.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:41.965+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:42.912 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:55:42.546+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 46845 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:55:42.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:42.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:42.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:42.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:42.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:42.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:42.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:42.558+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:43.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:43.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:43.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:43.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:43.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:43.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:43.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:43.004+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:43.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:42.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:43.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:42.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:43.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:42.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:43.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:42.912+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:43.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:43.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:43.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:43.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:43.876 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:43.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:43.876 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:43.594+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:44.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:43.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:44.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:43.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:44.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:43.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:44.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:43.875+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:44.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:43.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:44.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:43.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:44.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:43.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:44.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:43.993+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:44.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:44.596+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:44.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:44.596+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:44.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:44.596+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:44.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:44.597+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:45.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:44.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:45.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:44.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:45.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:44.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:45.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:44.892+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:45.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:44.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:45.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:44.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:45.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:44.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:45.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:44.987+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:45.848 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:45.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:45.849 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:45.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:45.849 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:45.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:45.849 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:45.559+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:46.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:45.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:46.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:45.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:46.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:45.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:46.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:45.975+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:46.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:45.847+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:46.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:45.847+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:46.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:45.847+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:46.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:45.848+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:46.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:46.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:46.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:46.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:46.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:46.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:46.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:46.527+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:47.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:46.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:47.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:46.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:47.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:46.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:47.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:46.821+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:47.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:46.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:47.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:46.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:47.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:46.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:47.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:46.957+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:47.843 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:55:47.552+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 46890 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:55:47.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:47.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:47.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:47.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:47.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:47.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:47.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:47.492+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:48.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:47.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:48.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:47.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:48.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:47.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:48.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:47.999+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:48.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:47.842+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:48.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:47.842+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:48.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:47.842+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:48.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:47.843+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:48.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:48.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:48.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:48.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:48.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:48.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:48.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:48.443+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:49.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:49.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:49.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:49.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:49.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:49.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:49.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:49.048+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:49.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:48.808+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:49.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:48.808+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:49.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:48.808+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:49.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:48.809+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:49.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:49.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:49.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:49.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:49.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:49.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:49.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:49.426+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:50.034 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:49.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:50.034 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:49.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:50.034 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:49.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:50.034 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:49.765+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:50.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:50.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:50.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:50.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:50.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:50.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:50.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:50.033+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:50.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:50.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:50.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:50.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:50.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:50.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:50.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:50.448+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:51.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:50.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:51.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:50.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:51.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:50.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:51.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:50.760+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:51.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:51.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:51.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:51.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:51.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:51.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:51.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:51.030+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:51.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:51.411+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:51.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:51.411+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:51.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:51.411+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:51.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:51.412+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:52.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:51.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:52.079 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:51.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:52.079 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:51.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:52.079 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:51.761+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:52.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:52.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:52.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:52.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:52.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:52.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:52.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:52.078+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:52.680 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:55:52.559+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 46936 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:55:52.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:52.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:52.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:52.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:52.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:52.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:52.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:52.407+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:52.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:52.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:52.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:52.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:52.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:52.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:52.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:52.748+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:53.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:53.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:53.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:53.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:53.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:53.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:53.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:53.091+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:53.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:53.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:53.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:53.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:53.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:53.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:53.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:53.403+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:53.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:53.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:53.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:53.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:53.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:53.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:53.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:53.774+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:54.386 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:54.125+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:54.386 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:54.125+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:54.386 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:54.125+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:54.386 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:54.126+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:54.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:54.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:54.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:54.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:54.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:54.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:54.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:54.385+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:55.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:54.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:55.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:54.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:55.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:54.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:55.092 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:54.762+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:55.431 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:55.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:55.431 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:55.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:55.431 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:55.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:55.431 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:55.090+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:55.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:55.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:55.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:55.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:55.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:55.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:55.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:55.431+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:56.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:55.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:56.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:55.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:56.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:55.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:56.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:55.776+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:56.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:56.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:56.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:56.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:56.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:56.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:56.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:56.106+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:56.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:56.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:56.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:56.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:56.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:56.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:56.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:56.403+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:57.106 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:56.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:57.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:56.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:57.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:56.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:57.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:56.797+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:57.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:57.105+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:57.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:57.105+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:57.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:57.105+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:57.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:57.106+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:57.687 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:55:57.564+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 46981 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:55:57.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:57.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:57.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:57.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:57.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:57.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:57.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:57.416+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:57.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:57.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:57.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:57.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:57.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:57.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:57.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:57.812+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:58.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:58.119+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:58.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:58.119+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:58.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:58.119+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:58.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:58.119+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:58.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:58.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:58.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:58.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:58.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:58.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:58.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:58.436+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:59.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:58.845+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:59.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:58.845+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:59.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:58.845+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:55:59.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:58.847+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:59.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:59.147+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:59.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:59.147+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:59.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:59.147+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:55:59.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:55:59.148+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:55:59.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:59.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:59.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:59.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:59.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:59.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:55:59.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:55:59.434+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:00.169 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:59.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:00.169 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:59.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:00.169 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:59.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:00.169 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:55:59.800+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:00.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:00.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:00.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:00.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:00.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:00.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:00.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:00.183+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:00.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:00.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:00.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:00.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:00.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:00.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:00.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:00.462+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:01.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:00.787+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:01.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:00.787+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:01.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:00.787+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:01.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:00.789+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:01.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:01.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:01.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:01.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:01.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:01.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:01.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:01.203+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:01.782 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:01.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:01.782 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:01.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:01.782 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:01.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:01.783 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:01.471+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:02.183 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:01.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:02.183 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:01.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:02.183 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:01.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:02.183 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:01.781+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:02.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:02.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:02.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:02.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:02.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:02.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:02.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:02.182+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:02.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:56:02.570+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 47026 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:56:02.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:02.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:02.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:02.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:02.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:02.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:02.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:02.462+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:03.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:02.780+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:03.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:02.780+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:03.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:02.780+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:03.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:02.782+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:03.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:03.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:03.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:03.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:03.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:03.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:03.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:03.175+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:03.760 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:03.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:03.761 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:03.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:03.761 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:03.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:03.761 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:03.476+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:04.174 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:03.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:04.174 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:03.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:04.175 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:03.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:04.175 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:03.760+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:04.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:04.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:04.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:04.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:04.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:04.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:04.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:04.173+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:04.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:04.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:04.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:04.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:04.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:04.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:04.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:04.444+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:05.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:04.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:05.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:04.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:05.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:04.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:05.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:04.747+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:05.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:05.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:05.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:05.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:05.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:05.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:05.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:05.134+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:05.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:05.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:05.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:05.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:05.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:05.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:05.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:05.460+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:06.150 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:05.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:06.150 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:05.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:06.150 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:05.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:06.150 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:05.746+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:06.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:06.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:06.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:06.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:06.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:06.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:06.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:06.148+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:06.764 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:06.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:06.765 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:06.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:06.765 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:06.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:06.765 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:06.484+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:07.126 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:06.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:07.126 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:06.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:07.126 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:06.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:07.126 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:06.763+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:07.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:07.123+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:07.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:07.123+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:07.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:07.123+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:07.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:07.123+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:07.971 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:56:07.574+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 47071 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:56:07.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:07.532+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:07.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:07.532+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:07.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:07.532+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:07.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:07.532+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:07.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:07.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:07.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:07.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:07.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:07.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:07.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:07.753+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:08.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:08.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:08.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:08.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:08.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:08.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:08.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:08.096+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:08.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:08.765+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:08.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:08.765+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:08.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:08.765+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:08.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:08.766+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:08.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:08.522+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:08.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:08.522+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:08.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:08.522+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:08.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:08.522+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:09.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:09.116+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:09.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:09.116+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:09.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:09.116+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:09.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:09.117+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:09.805 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:09.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:09.805 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:09.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:09.805 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:09.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:09.805 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:09.477+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:10.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:09.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:10.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:09.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:10.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:09.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:10.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:09.803+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:10.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:10.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:10.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:10.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:10.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:10.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:10.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:10.105+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:10.772 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:10.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:10.772 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:10.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:10.772 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:10.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:10.772 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:10.512+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:11.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:10.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:11.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:10.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:11.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:10.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:11.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:10.771+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:11.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:11.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:11.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:11.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:11.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:11.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:11.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:11.112+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:11.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:11.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:11.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:11.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:11.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:11.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:11.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:11.501+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:11.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:11.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:11.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:11.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:11.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:11.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:11.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:11.750+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:12.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:12.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:12.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:12.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:12.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:12.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:12.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:12.112+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:12.971 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:56:12.580+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 47117 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:56:12.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:12.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:12.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:12.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:12.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:12.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:12.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:12.759+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:12.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:12.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:12.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:12.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:12.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:12.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:12.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:12.530+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:13.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:13.105+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:13.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:13.105+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:13.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:13.105+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:13.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:13.106+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:13.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:13.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:13.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:13.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:13.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:13.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:13.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:13.485+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:14.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:13.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:14.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:13.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:14.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:13.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:14.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:13.796+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:14.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:14.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:14.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:14.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:14.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:14.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:14.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:14.125+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:14.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:14.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:14.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:14.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:14.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:14.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:14.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:14.437+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:15.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:14.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:15.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:14.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:15.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:14.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:15.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:14.764+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:15.401 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:15.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:15.401 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:15.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:15.401 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:15.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:15.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:15.110+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:15.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:15.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:15.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:15.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:15.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:15.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:15.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:15.399+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:16.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:15.727+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:16.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:15.727+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:16.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:15.727+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:16.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:15.728+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:16.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:16.114+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:16.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:16.114+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:16.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:16.114+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:16.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:16.114+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:16.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:16.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:16.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:16.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:16.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:16.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:16.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:16.395+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:17.023 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:16.749+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:17.023 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:16.749+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:17.023 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:16.749+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:17.023 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:16.749+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:17.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:17.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:17.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:17.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:17.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:17.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:17.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:17.135+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:17.713 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:56:17.587+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 47162 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:56:17.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:17.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:17.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:17.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:17.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:17.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:17.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:17.443+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:17.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:17.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:17.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:17.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:17.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:17.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:17.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:17.747+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:18.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:18.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:18.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:18.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:18.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:18.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:18.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:18.160+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:18.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:18.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:18.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:18.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:18.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:18.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:18.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:18.433+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:19.190 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:18.767+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:19.190 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:18.767+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:19.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:18.767+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:19.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:18.769+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:19.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:19.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:19.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:19.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:19.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:19.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:19.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:19.189+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:19.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:19.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:19.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:19.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:19.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:19.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:19.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:19.480+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:20.153 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:19.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:20.153 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:19.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:20.153 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:19.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:20.153 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:19.819+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:20.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:20.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:20.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:20.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:20.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:20.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:20.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:20.151+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:20.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:20.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:20.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:20.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:20.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:20.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:20.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:20.469+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:21.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:20.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:21.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:20.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:21.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:20.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:21.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:20.793+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:21.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:21.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:21.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:21.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:21.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:21.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:21.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:21.159+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:21.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:21.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:21.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:21.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:21.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:21.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:21.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:21.431+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:22.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:21.831+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:22.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:21.831+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:22.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:21.831+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:22.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:21.832+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:22.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:22.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:22.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:22.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:22.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:22.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:22.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:22.112+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:22.716 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:56:22.593+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 47207 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:56:22.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:22.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:22.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:22.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:22.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:22.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:22.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:22.387+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:22.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:22.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:22.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:22.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:22.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:22.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:22.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:22.862+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:23.391 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:23.071+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:23.391 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:23.071+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:23.391 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:23.071+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:23.391 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:23.071+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:23.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:23.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:23.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:23.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:23.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:23.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:23.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:23.391+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:23.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:23.859+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:23.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:23.859+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:23.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:23.859+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:23.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:23.860+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:24.381 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:24.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:24.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:24.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:24.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:24.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:24.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:24.046+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:24.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:24.380+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:24.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:24.380+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:24.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:24.380+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:24.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:24.381+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:25.067 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:24.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:25.067 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:24.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:25.067 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:24.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:25.068 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:24.857+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:25.364 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:25.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:25.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:25.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:25.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:25.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:25.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:25.066+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:25.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:25.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:25.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:25.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:25.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:25.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:25.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:25.364+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:26.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:25.875+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:26.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:25.875+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:26.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:25.875+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:26.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:25.876+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:26.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:26.087+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:26.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:26.087+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:26.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:26.087+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:26.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:26.087+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:26.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:26.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:26.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:26.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:26.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:26.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:26.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:26.357+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:27.128 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:26.833+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:27.128 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:26.833+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:27.128 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:26.833+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:27.128 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:26.834+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:27.399 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:27.126+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:27.399 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:27.126+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:27.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:27.126+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:27.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:27.127+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:27.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:56:27.599+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 47252 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:56:27.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:27.398+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:27.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:27.398+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:27.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:27.398+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:27.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:27.398+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:28.175 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:27.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:28.175 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:27.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:28.175 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:27.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:28.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:27.801+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:28.435 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:28.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:28.435 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:28.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:28.435 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:28.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:28.435 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:28.174+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:28.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:28.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:28.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:28.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:28.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:28.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:28.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:28.434+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:29.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:28.849+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:29.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:28.849+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:29.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:28.849+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:29.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:28.850+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:29.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:29.142+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:29.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:29.142+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:29.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:29.142+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:29.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:29.143+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:29.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:29.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:29.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:29.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:29.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:29.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:29.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:29.442+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:30.164 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:29.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:30.164 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:29.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:30.165 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:29.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:30.165 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:29.863+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:30.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:30.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:30.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:30.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:30.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:30.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:30.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:30.163+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:30.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:30.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:30.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:30.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:30.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:30.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:30.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:30.456+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:31.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:30.876+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:31.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:30.876+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:31.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:30.876+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:31.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:30.877+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:31.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:31.157+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:31.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:31.157+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:31.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:31.157+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:31.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:31.158+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:31.840 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:31.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:31.840 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:31.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:31.840 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:31.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:31.840 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:31.475+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:32.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:31.838+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:32.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:31.838+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:32.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:31.838+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:32.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:31.839+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:32.383 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:32.117+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:32.383 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:32.117+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:32.383 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:32.117+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:32.383 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:32.118+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:32.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:56:32.605+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 47298 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:56:32.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:32.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:32.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:32.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:32.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:32.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:32.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:32.512+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:33.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:32.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:33.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:32.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:33.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:32.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:33.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:32.857+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:33.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:33.117+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:33.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:33.117+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:33.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:33.117+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:33.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:33.118+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:33.865 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:33.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:33.865 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:33.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:33.865 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:33.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:33.865 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:33.539+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:34.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:33.863+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:34.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:33.863+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:34.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:33.863+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:34.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:33.864+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:34.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:34.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:34.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:34.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:34.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:34.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:34.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:34.145+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:34.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:34.555+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:34.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:34.555+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:34.845 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:34.555+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:34.845 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:34.556+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:35.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:34.842+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:35.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:34.842+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:35.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:34.842+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:35.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:34.843+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:35.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:35.103+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:35.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:35.103+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:35.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:35.103+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:35.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:35.104+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:35.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:35.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:35.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:35.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:35.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:35.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:35.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:35.579+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:36.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:35.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:36.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:35.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:36.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:35.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:36.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:35.868+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:36.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:36.102+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:36.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:36.102+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:36.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:36.102+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:36.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:36.103+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:36.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:36.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:36.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:36.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:36.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:36.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:36.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:36.539+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:37.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:36.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:37.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:36.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:37.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:36.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:37.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:36.825+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:37.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:37.066+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:37.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:37.066+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:37.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:37.066+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:37.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:37.066+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:37.839 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:56:37.611+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 47343 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:56:37.839 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:37.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:37.839 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:37.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:37.839 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:37.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:37.840 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:37.519+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:38.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:37.837+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:38.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:37.837+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:38.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:37.837+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:38.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:37.838+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:38.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:38.114+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:38.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:38.114+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:38.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:38.114+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:38.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:38.114+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:38.872 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:38.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:38.872 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:38.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:38.872 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:38.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:38.872 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:38.525+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:39.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:38.870+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:39.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:38.870+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:39.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:38.870+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:39.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:38.871+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:39.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:39.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:39.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:39.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:39.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:39.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:39.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:39.119+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:39.851 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:39.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:39.851 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:39.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:39.851 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:39.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:39.851 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:39.530+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:40.161 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:39.849+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:40.161 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:39.849+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:40.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:39.849+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:40.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:39.849+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:40.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:40.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:40.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:40.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:40.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:40.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:40.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:40.160+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:40.806 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:40.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:40.806 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:40.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:40.807 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:40.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:40.807 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:40.550+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:41.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:40.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:41.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:40.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:41.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:40.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:41.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:40.805+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:41.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:41.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:41.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:41.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:41.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:41.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:41.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:41.122+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:41.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:41.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:41.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:41.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:41.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:41.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:41.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:41.529+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:41.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:41.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:41.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:41.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:41.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:41.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:41.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:41.773+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:42.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:42.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:42.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:42.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:42.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:42.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:42.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:42.152+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:42.972 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:56:42.617+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 47388 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:56:42.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:42.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:42.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:42.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:42.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:42.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:42.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:42.771+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:42.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:42.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:42.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:42.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:42.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:42.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:42.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:42.538+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:43.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:43.147+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:43.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:43.147+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:43.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:43.147+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:43.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:43.147+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:43.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:43.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:43.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:43.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:43.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:43.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:43.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:43.794+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:43.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:43.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:43.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:43.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:43.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:43.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:43.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:43.587+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:44.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:44.164+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:44.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:44.164+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:44.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:44.164+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:44.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:44.164+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:44.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:44.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:44.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:44.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:44.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:44.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:44.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:44.587+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:44.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:44.831+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:44.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:44.831+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:44.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:44.831+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:44.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:44.832+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:45.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:45.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:45.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:45.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:45.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:45.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:45.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:45.147+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:45.867 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:45.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:45.867 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:45.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:45.867 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:45.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:45.867 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:45.553+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:46.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:45.864+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:46.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:45.864+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:46.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:45.864+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:46.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:45.866+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:46.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:46.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:46.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:46.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:46.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:46.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:46.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:46.133+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:46.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T02:56:46.629+0000 7f770c428700 0 monclient(hunting): authenticate timed out after 300 2024-04-27T02:56:46.632 INFO:teuthology.orchestra.run.smithi029.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-04-27T02:56:46.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:46.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:46.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:46.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:46.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:46.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:46.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:46.519+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:47.088 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:46.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:47.088 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:46.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:47.088 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:46.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:47.088 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:46.826+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:47.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:47.086+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:47.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:47.086+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:47.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:47.086+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:47.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:47.086+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:47.488 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph orch ps' 2024-04-27T02:56:47.722 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:56:47.623+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 47433 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:56:47.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:47.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:47.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:47.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:47.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:47.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:47.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:47.510+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:47.729 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/mon.smithi029/config 2024-04-27T02:56:48.001 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:47.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:48.001 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:47.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:48.002 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:47.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:48.002 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:47.800+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:48.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:48.098+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:48.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:48.098+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:48.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:48.098+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:48.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:48.098+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:48.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:48.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:48.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:48.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:48.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:48.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:48.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:48.469+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:49.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:48.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:49.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:48.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:49.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:48.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:49.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:48.756+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:49.314 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:49.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:49.314 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:49.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:49.314 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:49.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:49.314 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:49.061+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:49.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:49.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:49.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:49.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:49.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:49.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:49.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:49.456+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:49.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:49.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:49.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:49.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:49.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:49.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:49.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:49.794+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:50.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:50.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:50.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:50.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:50.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:50.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:50.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:50.033+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:50.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:50.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:50.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:50.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:50.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:50.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:50.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:50.438+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:51.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:51.026+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:51.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:51.026+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:51.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:51.026+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:51.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:51.026+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:51.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:50.824+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:51.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:50.824+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:51.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:50.824+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:51.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:50.826+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:51.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:51.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:51.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:51.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:51.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:51.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:51.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:51.462+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:52.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:52.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:52.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:52.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:52.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:52.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:52.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:52.014+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:52.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:51.803+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:52.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:51.803+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:52.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:51.803+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:52.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:51.803+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:52.722 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:56:52.625+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 47479 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:56:52.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:52.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:52.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:52.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:52.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:52.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:52.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:52.438+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:53.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:52.819+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:53.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:52.819+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:53.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:52.819+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:53.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:52.821+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:53.429 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:53.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:53.430 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:53.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:53.430 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:53.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:53.430 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:53.006+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:53.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:53.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:53.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:53.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:53.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:53.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:53.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:53.428+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:53.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:53.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:53.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:53.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:53.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:53.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:53.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:53.824+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:54.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:53.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:54.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:53.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:54.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:53.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:54.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:53.988+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:54.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:54.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:54.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:54.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:54.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:54.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:54.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:54.414+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:55.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:54.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:55.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:54.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:55.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:54.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:55.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:54.962+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:55.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:54.871+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:55.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:54.871+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:55.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:54.871+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:55.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:54.872+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:55.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:55.374+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:55.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:55.374+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:55.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:55.374+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:55.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:55.375+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:56.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:55.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:56.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:55.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:56.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:55.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:56.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:55.913+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:56.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:55.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:56.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:55.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:56.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:55.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:56.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:55.842+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:56.499 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:56.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:56.499 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:56.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:56.499 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:56.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:56.499 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:56.403+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:57.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:56.872+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:57.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:56.872+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:57.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:56.872+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:57.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:56.873+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:57.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:56.882+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:57.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:56.882+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:57.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:56.882+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:57.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:56.883+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:57.658 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:56:57.635+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 47524 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:56:57.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:57.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:57.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:57.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:57.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:57.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:57.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:57.398+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:57.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:57.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:57.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:57.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:57.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:57.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:57.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:57.847+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:57.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:57.900+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:57.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:57.900+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:57.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:57.900+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:57.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:57.901+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:58.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:58.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:58.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:58.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:58.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:58.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:58.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:58.418+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:59.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:58.837+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:59.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:58.837+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:59.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:58.837+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:56:59.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:58.837+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:59.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:58.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:59.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:58.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:59.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:58.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:56:59.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:58.950+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:56:59.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:59.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:59.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:59.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:59.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:59.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:56:59.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:56:59.416+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:00.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:59.909+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:00.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:59.909+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:00.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:59.909+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:00.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:56:59.910+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:00.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:59.790+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:00.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:59.790+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:00.085 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:59.790+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:00.085 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:56:59.790+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:00.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:00.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:00.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:00.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:00.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:00.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:00.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:00.396+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:01.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:00.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:01.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:00.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:01.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:00.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:01.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:00.951+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:01.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:00.804+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:01.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:00.804+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:01.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:00.804+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:01.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:00.804+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:01.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:01.355+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:01.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:01.355+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:01.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:01.355+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:01.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:01.356+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:02.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:01.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:02.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:01.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:02.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:01.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:02.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:01.912+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:02.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:01.816+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:02.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:01.816+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:02.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:01.816+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:02.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:01.816+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:02.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:02.314+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:02.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:02.314+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:02.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:02.314+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:02.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:02.314+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:02.932 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:57:02.641+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 47569 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:57:02.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:02.830+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:02.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:02.830+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:02.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:02.830+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:02.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:02.831+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:03.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:02.930+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:03.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:02.930+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:03.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:02.930+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:03.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:02.931+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:03.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:03.286+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:03.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:03.286+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:03.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:03.286+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:03.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:03.287+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:03.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:03.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:03.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:03.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:03.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:03.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:03.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:03.912+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:03.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:03.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:03.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:03.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:03.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:03.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:03.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:03.842+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:04.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:04.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:04.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:04.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:04.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:04.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:04.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:04.253+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:05.132 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:04.903+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:05.132 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:04.903+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:05.132 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:04.903+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:05.133 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:04.904+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:05.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:04.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:05.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:04.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:05.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:04.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:05.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:04.821+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:05.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:05.224+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:05.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:05.224+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:05.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:05.224+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:05.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:05.224+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:06.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:05.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:06.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:05.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:06.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:05.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:06.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:05.851+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:06.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:05.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:06.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:05.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:06.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:05.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:06.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:05.856+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:06.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:06.264+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:06.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:06.264+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:06.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:06.264+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:06.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:06.265+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:07.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:06.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:07.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:06.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:07.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:06.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:07.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:06.866+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:07.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:06.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:07.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:06.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:07.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:06.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:07.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:06.813+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:07.650 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:07.246+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:07.650 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:07.246+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:07.650 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:07.246+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:07.650 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:07.246+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:07.972 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:57:07.646+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 47614 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:57:07.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:07.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:07.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:07.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:07.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:07.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:07.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:07.828+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:07.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:07.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:07.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:07.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:07.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:07.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:07.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:07.821+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:08.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:08.271+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:08.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:08.271+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:08.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:08.271+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:08.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:08.272+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:09.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:08.819+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:09.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:08.819+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:09.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:08.819+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:09.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:08.820+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:09.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:08.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:09.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:08.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:09.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:08.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:09.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:08.788+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:09.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:09.246+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:09.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:09.246+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:09.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:09.246+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:09.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:09.246+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:10.173 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:09.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:10.173 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:09.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:10.173 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:09.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:10.174 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:09.826+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:10.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:09.755+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:10.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:09.755+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:10.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:09.755+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:10.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:09.755+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:10.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:10.290+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:10.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:10.290+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:10.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:10.290+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:10.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:10.291+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:11.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:10.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:11.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:10.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:11.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:10.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:11.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:10.835+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:11.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:10.765+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:11.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:10.766+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:11.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:10.766+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:11.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:10.766+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:11.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:11.299+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:11.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:11.299+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:11.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:11.299+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:11.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:11.300+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:12.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:11.778+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:12.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:11.778+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:12.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:11.778+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:12.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:11.778+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:12.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:11.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:12.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:11.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:12.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:11.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:12.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:11.794+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:12.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:12.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:12.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:12.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:12.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:12.341+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:12.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:12.342+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:12.972 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:57:12.653+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 47660 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:57:12.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:12.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:12.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:12.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:12.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:12.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:12.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:12.772+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:12.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:12.813+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:12.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:12.813+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:12.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:12.813+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:12.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:12.813+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:13.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:13.303+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:13.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:13.303+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:13.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:13.303+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:13.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:13.305+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:13.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:13.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:13.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:13.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:13.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:13.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:13.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:13.821+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:13.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:13.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:13.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:13.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:13.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:13.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:13.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:13.813+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:14.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:14.260+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:14.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:14.260+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:14.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:14.260+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:14.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:14.261+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:15.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:14.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:15.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:14.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:15.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:14.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:15.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:14.802+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:15.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:14.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:15.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:14.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:15.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:14.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:15.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:14.848+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:15.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:15.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:15.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:15.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:15.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:15.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:15.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:15.281+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:16.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:15.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:16.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:15.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:16.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:15.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:16.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:15.802+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:16.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:15.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:16.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:15.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:16.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:15.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:16.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:15.861+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:16.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:16.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:16.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:16.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:16.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:16.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:16.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:16.323+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:17.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:16.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:17.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:16.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:17.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:16.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:17.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:16.756+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:17.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:16.828+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:17.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:16.828+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:17.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:16.828+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:17.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:16.828+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:17.662 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:17.296+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:17.662 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:17.296+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:17.662 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:17.296+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:17.662 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:17.297+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:17.972 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:57:17.659+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 47705 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:57:17.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:17.815+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:17.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:17.815+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:17.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:17.815+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:17.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:17.815+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:17.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:17.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:17.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:17.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:17.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:17.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:17.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:17.799+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:18.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:18.250+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:18.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:18.250+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:18.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:18.250+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:18.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:18.251+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:19.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:18.797+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:19.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:18.797+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:19.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:18.797+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:19.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:18.798+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:19.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:18.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:19.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:18.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:19.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:18.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:19.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:18.799+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:19.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:19.228+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:19.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:19.228+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:19.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:19.228+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:19.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:19.229+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:20.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:19.767+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:20.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:19.767+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:20.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:19.767+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:20.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:19.767+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:20.174 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:19.824+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:20.175 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:19.824+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:20.175 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:19.824+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:20.175 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:19.825+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:20.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:20.235+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:20.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:20.235+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:20.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:20.235+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:20.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:20.237+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:21.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:20.770+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:21.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:20.770+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:21.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:20.770+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:21.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:20.771+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:21.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:20.847+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:21.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:20.847+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:21.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:20.847+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:21.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:20.848+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:21.535 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:21.272+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:21.535 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:21.272+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:21.535 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:21.272+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:21.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:21.273+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:22.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:21.881+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:22.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:21.881+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:22.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:21.881+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:22.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:21.882+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:22.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:21.796+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:22.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:21.796+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:22.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:21.796+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:22.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:21.796+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:22.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:22.281+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:22.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:22.281+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:22.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:22.281+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:22.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:22.282+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:22.972 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:57:22.666+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 47750 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:57:22.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:22.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:22.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:22.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:22.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:22.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:22.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:22.847+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:22.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:22.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:22.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:22.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:22.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:22.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:22.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:22.862+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:23.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:23.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:23.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:23.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:23.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:23.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:23.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:23.322+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:23.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:23.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:23.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:23.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:23.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:23.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:23.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:23.834+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:23.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:23.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:23.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:23.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:23.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:23.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:23.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:23.835+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:24.607 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:24.301+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:24.608 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:24.301+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:24.608 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:24.301+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:24.608 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:24.302+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:24.883 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:24.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:24.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:24.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:24.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:24.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:24.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:24.785+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:25.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:24.881+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:25.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:24.881+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:25.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:24.881+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:25.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:24.882+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:25.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:25.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:25.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:25.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:25.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:25.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:25.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:25.254+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:26.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:25.802+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:26.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:25.802+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:26.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:25.802+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:26.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:25.802+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:26.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:25.896+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:26.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:25.896+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:26.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:25.896+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:26.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:25.896+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:26.504 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:26.265+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:26.504 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:26.265+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:26.504 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:26.265+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:26.504 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:26.266+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:27.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:26.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:27.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:26.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:27.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:26.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:27.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:26.850+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:27.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:26.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:27.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:26.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:27.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:26.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:27.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:26.876+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:27.497 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:27.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:27.497 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:27.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:27.497 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:27.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:27.497 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:27.234+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:27.798 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:57:27.672+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 47795 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:57:28.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:27.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:28.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:27.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:28.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:27.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:28.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:27.851+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:28.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:27.842+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:28.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:27.842+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:28.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:27.842+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:28.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:27.843+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:28.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:28.240+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:28.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:28.240+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:28.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:28.240+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:28.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:28.241+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:29.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:28.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:29.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:28.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:29.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:28.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:29.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:28.820+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:29.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:28.865+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:29.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:28.865+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:29.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:28.865+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:29.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:28.866+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:29.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:29.286+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:29.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:29.286+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:29.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:29.286+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:29.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:29.287+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:30.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:29.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:30.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:29.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:30.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:29.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:30.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:29.869+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:30.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:29.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:30.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:29.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:30.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:29.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:30.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:29.855+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:30.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:30.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:30.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:30.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:30.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:30.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:30.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:30.290+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:31.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:30.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:31.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:30.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:31.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:30.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:31.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:30.866+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:31.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:30.852+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:31.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:30.852+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:31.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:30.852+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:31.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:30.852+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:31.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:31.261+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:31.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:31.261+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:31.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:31.261+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:31.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:31.263+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:32.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:31.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:32.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:31.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:32.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:31.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:32.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:31.897+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:32.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:31.824+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:32.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:31.824+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:32.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:31.824+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:32.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:31.824+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:32.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:32.259+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:32.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:32.259+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:32.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:32.259+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:32.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:32.260+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:32.937 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:57:32.678+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 47841 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:57:32.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:32.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:32.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:32.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:32.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:32.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:32.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:32.813+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:33.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:32.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:33.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:32.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:33.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:32.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:33.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:32.935+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:33.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:33.307+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:33.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:33.307+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:33.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:33.307+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:33.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:33.308+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:33.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:33.890+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:33.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:33.890+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:33.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:33.890+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:33.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:33.891+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:33.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:33.826+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:33.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:33.826+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:33.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:33.826+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:33.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:33.828+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:34.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:34.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:34.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:34.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:34.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:34.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:34.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:34.346+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:35.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:34.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:35.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:34.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:35.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:34.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:35.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:34.848+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:35.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:34.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:35.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:34.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:35.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:34.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:35.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:34.799+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:35.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:35.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:35.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:35.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:35.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:35.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:35.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:35.390+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:36.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:35.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:36.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:35.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:36.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:35.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:36.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:35.825+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:36.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:35.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:36.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:35.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:36.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:35.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:36.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:35.755+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:36.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:36.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:36.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:36.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:36.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:36.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:36.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:36.421+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:36.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:36.779+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:36.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:36.779+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:36.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:36.779+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:36.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:36.780+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:36.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:36.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:36.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:36.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:36.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:36.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:36.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:36.713+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:37.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:37.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:37.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:37.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:37.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:37.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:37.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:37.402+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:37.972 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:57:37.684+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 47886 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:57:37.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:37.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:37.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:37.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:37.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:37.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:37.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:37.777+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:37.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:37.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:37.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:37.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:37.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:37.673+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:37.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:37.674+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:38.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:38.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:38.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:38.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:38.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:38.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:38.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:38.411+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:38.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:38.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:38.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:38.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:38.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:38.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:38.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:38.699+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:38.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:38.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:38.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:38.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:38.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:38.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:38.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:38.730+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:39.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:39.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:39.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:39.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:39.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:39.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:39.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:39.419+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:40.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:39.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:40.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:39.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:40.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:39.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:40.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:39.738+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:40.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:39.768+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:40.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:39.768+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:40.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:39.768+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:40.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:39.768+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:40.453 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:40.372+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:40.454 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:40.372+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:40.454 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:40.372+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:40.454 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:40.373+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:41.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:40.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:41.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:40.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:41.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:40.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:41.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:40.752+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:41.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:40.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:41.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:40.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:41.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:40.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:41.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:40.763+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:41.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:41.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:41.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:41.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:41.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:41.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:41.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:41.404+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:42.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:41.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:42.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:41.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:42.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:41.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:42.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:41.814+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:42.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:41.727+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:42.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:41.727+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:42.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:41.727+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:42.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:41.727+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:42.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:42.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:42.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:42.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:42.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:42.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:42.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:42.405+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:42.972 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:57:42.691+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 47932 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:57:42.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:42.686+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:42.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:42.686+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:42.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:42.686+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:42.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:42.687+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:42.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:42.779+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:42.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:42.779+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:42.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:42.779+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:42.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:42.780+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:43.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:43.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:43.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:43.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:43.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:43.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:43.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:43.436+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:43.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:43.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:43.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:43.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:43.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:43.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:43.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:43.733+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:43.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:43.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:43.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:43.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:43.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:43.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:43.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:43.772+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:44.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:44.407+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:44.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:44.407+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:44.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:44.407+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:44.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:44.407+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:45.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:44.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:45.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:44.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:45.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:44.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:45.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:44.821+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:45.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:44.775+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:45.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:44.775+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:45.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:44.775+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:45.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:44.776+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:45.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:45.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:45.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:45.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:45.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:45.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:45.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:45.443+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:46.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:45.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:46.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:45.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:46.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:45.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:46.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:45.845+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:46.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:45.769+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:46.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:45.769+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:46.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:45.769+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:46.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:45.770+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:46.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:46.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:46.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:46.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:46.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:46.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:46.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:46.468+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:47.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:46.780+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:47.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:46.780+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:47.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:46.780+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:47.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:46.781+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:47.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:46.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:47.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:46.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:47.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:46.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:47.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:46.805+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:47.767 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:57:47.697+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 47976 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:57:47.767 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:47.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:47.767 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:47.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:47.767 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:47.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:47.767 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:47.482+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:48.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:47.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:48.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:47.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:48.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:47.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:48.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:47.765+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:48.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:47.816+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:48.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:47.816+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:48.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:47.816+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:48.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:47.816+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:48.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:48.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:48.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:48.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:48.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:48.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:48.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:48.446+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:49.184 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:48.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:49.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:48.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:49.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:48.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:49.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:48.836+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:49.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:48.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:49.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:48.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:49.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:48.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:49.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:48.748+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:49.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:49.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:49.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:49.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:49.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:49.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:49.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:49.401+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:50.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:49.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:50.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:49.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:50.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:49.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:50.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:49.875+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:50.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:49.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:50.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:49.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:50.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:49.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:50.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:49.792+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:50.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:50.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:50.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:50.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:50.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:50.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:50.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:50.376+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:50.833 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:50.784+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:50.833 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:50.784+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:50.833 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:50.784+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:50.833 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:50.784+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:51.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:50.830+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:51.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:50.830+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:51.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:50.830+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:51.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:50.830+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:51.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:51.344+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:51.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:51.344+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:51.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:51.344+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:51.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:51.345+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:52.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:51.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:52.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:51.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:52.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:51.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:52.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:51.763+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:52.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:51.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:52.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:51.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:52.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:51.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:52.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:51.826+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:52.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:52.305+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:52.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:52.305+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:52.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:52.305+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:52.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:52.306+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:52.972 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:57:52.703+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 48023 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:57:52.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:52.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:52.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:52.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:52.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:52.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:52.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:52.725+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:52.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:52.801+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:52.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:52.801+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:52.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:52.801+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:52.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:52.801+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:53.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:53.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:53.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:53.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:53.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:53.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:53.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:53.335+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:53.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:53.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:53.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:53.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:53.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:53.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:53.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:53.847+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:53.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:53.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:53.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:53.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:53.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:53.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:53.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:53.771+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:54.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:54.288+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:54.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:54.288+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:54.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:54.288+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:54.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:54.289+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:55.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:54.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:55.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:54.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:55.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:54.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:55.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:54.812+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:55.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:54.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:55.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:54.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:55.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:54.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:55.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:54.851+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:55.597 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:55.319+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:55.597 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:55.319+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:55.598 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:55.319+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:55.598 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:55.321+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:55.868 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:55.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:55.869 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:55.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:55.869 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:55.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:55.869 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:55.827+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:56.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:55.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:56.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:55.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:56.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:55.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:56.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:55.866+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:56.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:56.288+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:56.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:56.288+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:56.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:56.288+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:56.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:56.290+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:57.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:56.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:57.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:56.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:57.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:56.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:57.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:56.829+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:57.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:56.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:57.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:56.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:57.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:56.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:57.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:56.847+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:57.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:57.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:57.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:57.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:57.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:57.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:57.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:57.282+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:57.973 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:57:57.709+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 48067 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:57:57.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:57.839+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:57.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:57.839+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:57.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:57.839+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:57.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:57.839+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:57.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:57.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:57.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:57.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:57.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:57.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:57.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:57.786+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:58.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:58.317+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:58.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:58.317+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:58.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:58.317+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:58.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:58.318+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:59.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:58.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:59.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:58.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:59.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:58.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:57:59.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:58.835+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:59.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:58.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:59.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:58.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:59.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:58.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:57:59.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:58.888+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:57:59.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:59.344+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:59.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:59.344+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:59.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:59.344+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:57:59.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:57:59.345+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:00.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:59.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:00.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:59.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:00.178 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:59.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:00.178 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:57:59.850+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:00.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:59.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:00.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:59.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:00.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:59.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:00.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:57:59.900+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:00.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:00.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:00.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:00.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:00.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:00.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:00.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:00.368+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:00.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:00.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:00.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:00.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:00.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:00.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:00.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:00.812+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:00.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:00.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:00.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:00.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:00.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:00.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:00.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:00.857+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:01.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:01.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:01.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:01.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:01.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:01.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:01.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:01.416+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:02.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:01.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:02.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:01.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:02.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:01.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:02.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:01.807+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:02.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:01.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:02.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:01.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:02.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:01.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:02.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:01.812+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:02.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:02.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:02.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:02.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:02.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:02.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:02.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:02.443+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:02.973 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:58:02.715+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 48113 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:58:02.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:02.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:02.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:02.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:02.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:02.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:02.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:02.828+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:02.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:02.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:02.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:02.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:02.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:02.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:02.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:02.861+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:03.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:03.404+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:03.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:03.404+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:03.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:03.404+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:03.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:03.405+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:03.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:03.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:03.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:03.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:03.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:03.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:03.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:03.869+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:03.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:03.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:03.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:03.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:03.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:03.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:03.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:03.895+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:04.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:04.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:04.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:04.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:04.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:04.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:04.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:04.368+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:05.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:04.875+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:05.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:04.875+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:05.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:04.875+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:05.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:04.877+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:05.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:04.903+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:05.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:04.903+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:05.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:04.903+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:05.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:04.904+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:05.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:05.343+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:05.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:05.343+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:05.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:05.343+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:05.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:05.343+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:05.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:05.900+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:05.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:05.900+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:05.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:05.900+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:05.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:05.901+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:05.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:05.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:05.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:05.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:05.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:05.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:05.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:05.897+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:06.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:06.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:06.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:06.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:06.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:06.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:06.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:06.380+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:07.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:06.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:07.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:06.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:07.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:06.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:07.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:06.887+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:07.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:06.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:07.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:06.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:07.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:06.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:07.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:06.890+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:07.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:07.391+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:07.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:07.391+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:07.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:07.391+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:07.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:07.391+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:08.223 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:58:07.720+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 48157 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:58:08.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:07.896+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:08.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:07.896+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:08.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:07.896+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:08.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:07.897+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:08.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:07.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:08.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:07.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:08.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:07.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:08.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:07.922+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:08.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:08.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:08.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:08.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:08.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:08.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:08.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:08.348+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:09.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:08.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:09.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:08.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:09.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:08.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:09.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:08.925+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:09.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:08.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:09.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:08.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:09.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:08.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:09.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:08.900+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:09.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:09.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:09.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:09.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:09.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:09.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:09.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:09.338+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:10.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:09.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:10.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:09.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:10.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:09.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:10.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:09.958+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:10.178 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:09.887+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:10.178 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:09.887+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:10.178 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:09.887+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:10.178 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:09.888+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:10.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:10.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:10.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:10.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:10.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:10.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:10.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:10.368+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:10.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:10.926+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:10.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:10.926+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:10.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:10.926+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:10.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:10.927+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:10.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:10.921+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:10.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:10.921+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:10.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:10.921+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:10.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:10.922+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:11.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:11.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:11.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:11.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:11.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:11.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:11.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:11.366+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:12.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:11.905+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:12.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:11.905+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:12.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:11.905+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:12.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:11.906+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:12.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:11.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:12.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:11.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:12.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:11.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:12.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:11.936+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:12.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:12.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:12.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:12.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:12.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:12.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:12.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:12.355+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:13.223 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:58:12.727+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 48204 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:58:13.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:12.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:13.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:12.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:13.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:12.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:13.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:12.957+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:13.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:12.912+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:13.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:12.912+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:13.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:12.912+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:13.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:12.913+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:13.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:13.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:13.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:13.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:13.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:13.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:13.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:13.306+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:13.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:13.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:13.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:13.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:13.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:13.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:13.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:13.961+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:13.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:13.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:13.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:13.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:13.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:13.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:13.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:13.867+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:14.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:14.350+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:14.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:14.350+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:14.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:14.350+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:14.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:14.351+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:15.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:14.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:15.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:14.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:15.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:14.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:15.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:14.863+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:15.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:14.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:15.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:14.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:15.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:14.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:15.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:14.921+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:15.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:15.324+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:15.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:15.324+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:15.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:15.324+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:15.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:15.326+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:16.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:15.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:16.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:15.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:16.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:15.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:16.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:15.842+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:16.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:15.919+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:16.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:15.919+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:16.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:15.919+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:16.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:15.920+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:16.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:16.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:16.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:16.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:16.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:16.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:16.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:16.354+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:17.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:16.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:17.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:16.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:17.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:16.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:17.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:16.874+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:17.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:16.877+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:17.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:16.877+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:17.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:16.877+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:17.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:16.878+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:17.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:17.319+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:17.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:17.319+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:17.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:17.319+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:17.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:17.320+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:18.223 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:58:17.734+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 48248 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:58:18.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:17.915+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:18.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:17.915+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:18.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:17.915+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:18.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:17.917+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:18.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:17.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:18.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:17.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:18.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:17.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:18.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:17.887+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:18.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:18.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:18.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:18.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:18.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:18.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:18.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:18.322+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:19.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:18.959+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:19.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:18.959+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:19.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:18.959+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:19.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:18.960+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:19.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:18.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:19.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:18.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:19.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:18.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:19.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:18.843+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:19.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:19.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:19.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:19.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:19.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:19.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:19.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:19.368+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:20.178 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:19.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:20.179 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:19.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:20.179 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:19.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:20.179 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:19.929+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:20.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:19.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:20.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:19.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:20.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:19.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:20.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:19.888+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:20.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:20.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:20.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:20.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:20.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:20.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:20.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:20.419+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:21.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:20.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:21.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:20.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:21.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:20.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:21.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:20.914+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:21.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:20.909+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:21.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:20.909+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:21.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:20.909+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:21.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:20.909+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:21.391 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:21.387+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:21.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:21.387+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:21.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:21.387+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:21.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:21.389+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:22.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:21.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:22.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:21.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:22.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:21.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:22.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:21.916+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:22.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:21.877+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:22.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:21.877+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:22.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:21.877+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:22.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:21.878+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:22.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:22.421+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:22.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:22.421+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:22.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:22.421+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:22.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:22.422+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:23.223 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:58:22.740+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 48294 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:58:23.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:22.858+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:23.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:22.858+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:23.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:22.858+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:23.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:22.859+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:23.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:22.890+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:23.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:22.890+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:23.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:22.890+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:23.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:22.891+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:23.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:23.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:23.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:23.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:23.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:23.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:23.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:23.428+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:23.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:23.849+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:23.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:23.849+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:23.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:23.849+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:23.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:23.851+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:23.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:23.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:23.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:23.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:23.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:23.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:23.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:23.937+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:24.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:24.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:24.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:24.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:24.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:24.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:24.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:24.452+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:25.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:24.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:25.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:24.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:25.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:24.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:25.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:24.823+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:25.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:24.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:25.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:24.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:25.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:24.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:25.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:24.934+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:25.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:25.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:25.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:25.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:25.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:25.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:25.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:25.483+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:26.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:25.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:26.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:25.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:26.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:25.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:26.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:25.818+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:26.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:25.964+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:26.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:25.964+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:26.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:25.964+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:26.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:25.964+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:26.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:26.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:26.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:26.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:26.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:26.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:26.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:26.490+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:27.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:26.840+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:27.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:26.840+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:27.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:26.840+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:27.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:26.841+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:27.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:26.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:27.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:26.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:27.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:26.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:27.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:26.989+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:27.749 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:27.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:27.749 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:27.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:27.750 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:27.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:27.750 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:27.495+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:28.223 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:58:27.746+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 48338 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:58:28.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:27.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:28.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:27.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:28.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:27.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:28.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:27.807+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:28.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:27.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:28.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:27.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:28.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:27.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:28.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:27.961+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:28.797 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:28.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:28.797 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:28.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:28.797 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:28.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:28.797 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:28.543+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:29.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:28.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:29.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:28.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:29.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:28.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:29.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:28.795+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:29.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:28.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:29.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:28.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:29.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:28.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:29.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:28.938+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:29.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:29.539+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:29.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:29.539+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:29.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:29.539+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:29.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:29.540+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:29.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:29.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:29.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:29.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:29.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:29.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:29.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:29.771+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:30.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:29.942+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:30.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:29.942+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:30.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:29.942+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:30.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:29.943+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:30.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:30.532+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:30.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:30.532+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:30.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:30.532+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:30.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:30.532+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:31.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:30.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:31.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:30.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:31.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:30.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:31.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:30.819+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:31.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:30.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:31.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:30.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:31.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:30.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:31.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:30.927+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:31.774 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:31.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:31.774 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:31.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:31.774 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:31.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:31.774 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:31.513+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:32.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:31.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:32.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:31.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:32.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:31.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:32.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:31.771+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:32.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:31.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:32.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:31.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:32.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:31.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:32.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:31.897+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:32.755 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:32.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:32.756 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:32.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:32.756 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:32.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:32.756 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:32.502+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:33.224 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:58:32.752+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 48385 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:58:33.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:32.768+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:33.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:32.768+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:33.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:32.768+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:33.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:32.769+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:33.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:32.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:33.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:32.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:33.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:32.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:33.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:32.864+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:33.737 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:33.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:33.738 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:33.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:33.738 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:33.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:33.738 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:33.483+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:34.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:33.881+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:34.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:33.881+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:34.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:33.881+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:34.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:33.881+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:34.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:33.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:34.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:33.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:34.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:33.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:34.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:33.735+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:34.868 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:34.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:34.868 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:34.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:34.868 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:34.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:34.868 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:34.520+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:34.868 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:34.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:34.869 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:34.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:34.869 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:34.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:34.869 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:34.722+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:35.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:34.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:35.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:34.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:35.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:34.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:35.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:34.865+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:35.841 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:35.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:35.841 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:35.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:35.842 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:35.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:35.842 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:35.565+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:35.842 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:35.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:35.842 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:35.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:35.842 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:35.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:35.842 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:35.764+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:36.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:35.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:36.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:35.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:36.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:35.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:36.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:35.888+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:36.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:36.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:36.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:36.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:36.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:36.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:36.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:36.603+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:37.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:36.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:37.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:36.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:37.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:36.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:37.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:36.862+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:37.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:36.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:37.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:36.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:37.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:36.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:37.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:36.753+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:37.882 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:58:37.758+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 48429 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:58:37.882 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:37.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:37.883 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:37.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:37.883 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:37.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:37.883 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:37.825+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:37.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:37.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:37.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:37.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:37.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:37.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:37.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:37.610+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:37.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:37.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:37.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:37.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:37.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:37.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:37.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:37.803+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:38.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:38.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:38.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:38.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:38.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:38.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:38.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:38.831+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:38.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:38.792+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:38.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:38.792+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:38.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:38.792+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:38.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:38.792+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:38.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:38.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:38.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:38.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:38.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:38.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:38.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:38.592+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:39.834 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:39.580+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:39.834 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:39.580+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:39.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:39.580+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:39.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:39.581+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:39.835 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:39.766+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:39.835 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:39.766+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:39.835 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:39.766+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:39.835 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:39.766+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:39.836 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:39.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:39.836 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:39.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:39.836 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:39.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:39.836 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:39.831+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:40.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:40.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:40.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:40.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:40.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:40.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:40.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:40.861+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:40.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:40.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:40.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:40.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:40.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:40.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:40.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:40.621+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:40.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:40.719+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:40.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:40.719+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:40.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:40.719+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:40.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:40.719+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:41.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:41.767+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:41.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:41.767+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:41.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:41.767+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:41.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:41.767+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:41.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:41.668+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:41.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:41.668+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:41.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:41.668+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:41.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:41.670+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:41.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:41.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:41.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:41.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:41.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:41.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:41.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:41.893+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:42.973 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:58:42.764+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 48475 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:58:42.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:42.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:42.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:42.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:42.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:42.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:42.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:42.702+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:42.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:42.747+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:42.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:42.747+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:42.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:42.747+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:42.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:42.748+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:42.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:42.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:42.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:42.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:42.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:42.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:42.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:42.893+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:44.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:43.740+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:44.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:43.740+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:44.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:43.740+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:44.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:43.741+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:44.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:43.721+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:44.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:43.721+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:44.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:43.721+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:44.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:43.721+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:44.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:43.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:44.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:43.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:44.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:43.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:44.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:43.930+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:44.757 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:44.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:44.758 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:44.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:44.758 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:44.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:44.758 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:44.684+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:45.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:44.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:45.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:44.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:45.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:44.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:45.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:44.754+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:45.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:44.961+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:45.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:44.961+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:45.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:44.961+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:45.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:44.962+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:45.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:45.795+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:45.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:45.795+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:45.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:45.795+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:45.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:45.795+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:45.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:45.714+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:45.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:45.714+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:45.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:45.714+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:45.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:45.715+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:45.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:45.923+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:45.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:45.923+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:45.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:45.923+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:45.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:45.924+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:47.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:46.756+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:47.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:46.756+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:47.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:46.756+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:47.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:46.757+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:47.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:46.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:47.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:46.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:47.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:46.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:47.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:46.892+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:47.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:46.765+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:47.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:46.765+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:47.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:46.765+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:47.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:46.765+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:48.223 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:58:47.770+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 48519 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:58:48.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:47.806+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:48.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:47.806+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:48.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:47.806+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:48.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:47.807+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:48.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:47.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:48.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:47.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:48.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:47.745+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:48.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:47.745+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:48.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:47.934+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:48.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:47.934+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:48.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:47.934+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:48.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:47.935+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:49.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:48.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:49.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:48.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:49.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:48.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:49.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:48.831+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:49.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:48.887+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:49.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:48.887+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:49.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:48.887+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:49.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:48.888+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:49.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:48.759+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:49.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:48.759+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:49.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:48.759+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:49.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:48.760+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:50.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:49.922+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:50.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:49.922+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:50.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:49.922+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:50.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:49.923+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:50.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:49.753+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:50.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:49.753+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:50.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:49.753+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:50.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:49.754+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:50.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:49.810+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:50.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:49.810+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:50.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:49.810+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:50.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:49.810+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:50.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:50.783+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:50.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:50.783+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:50.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:50.783+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:50.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:50.783+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:50.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:50.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:50.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:50.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:50.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:50.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:50.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:50.761+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:51.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:50.969+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:51.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:50.969+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:51.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:50.969+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:51.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:50.970+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:52.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:51.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:52.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:51.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:52.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:51.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:52.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:51.735+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:52.016 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:51.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:52.017 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:51.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:52.017 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:51.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:52.017 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:51.780+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:52.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:52.012+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:52.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:52.012+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:52.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:52.012+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:52.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:52.013+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:53.025 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:58:52.776+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 48566 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:58:53.025 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:52.746+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:53.025 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:52.746+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:53.025 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:52.746+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:53.025 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:52.747+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:53.025 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:52.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:53.025 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:52.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:53.026 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:52.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:53.026 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:52.755+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:53.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:53.021+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:53.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:53.021+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:53.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:53.021+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:53.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:53.022+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:53.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:53.769+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:53.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:53.769+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:53.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:53.769+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:53.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:53.770+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:53.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:53.716+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:53.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:53.716+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:53.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:53.716+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:53.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:53.717+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:54.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:53.993+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:54.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:53.993+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:54.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:53.993+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:54.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:53.994+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:55.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:54.754+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:55.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:54.754+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:55.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:54.754+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:55.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:54.755+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:55.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:55.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:55.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:55.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:55.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:55.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:55.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:55.001+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:55.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:54.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:55.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:54.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:55.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:54.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:55.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:54.819+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:56.125 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:55.796+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:56.126 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:55.796+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:56.126 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:55.796+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:56.126 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:55.797+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:56.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:55.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:56.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:55.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:56.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:55.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:56.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:55.822+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:56.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:56.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:56.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:56.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:56.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:56.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:56.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:56.023+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:57.058 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:56.832+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:57.058 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:56.832+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:57.058 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:56.832+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:57.058 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:56.833+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:57.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:56.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:57.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:56.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:57.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:56.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:57.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:56.862+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:57.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:57.054+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:57.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:57.054+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:57.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:57.054+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:57.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:57.056+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:58.104 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:58:57.782+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 48610 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:58:58.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:57.872+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:58.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:57.872+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:58.105 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:57.872+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:58.105 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:57.873+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:58.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:57.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:58.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:57.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:58.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:57.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:58.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:57.837+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:58.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:58.100+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:58.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:58.100+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:58.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:58.100+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:58.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:58.101+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:59.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:58.914+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:59.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:58.914+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:59.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:58.914+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:58:59.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:58.915+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:59.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:58.882+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:59.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:58.882+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:59.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:58.882+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:58:59.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:58.883+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:58:59.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:59.104+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:59.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:59.104+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:59.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:59.104+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:58:59.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:58:59.106+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:00.132 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:59.950+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:00.132 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:59.950+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:00.132 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:59.950+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:00.132 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:58:59.951+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:00.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:59.896+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:00.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:59.896+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:00.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:59.896+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:00.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:58:59.896+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:00.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:00.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:59:00.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:00.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:59:00.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:00.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:59:00.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:00.129+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:01.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:01.133+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:59:01.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:01.133+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:59:01.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:01.133+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:59:01.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:01.134+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:01.216 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:00.942+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:01.216 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:00.942+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:01.216 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:00.942+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:01.216 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:00.943+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:01.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:00.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:01.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:00.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:01.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:00.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:01.217 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:00.911+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:02.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:01.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:02.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:01.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:02.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:01.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:02.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:01.866+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:02.186 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:01.911+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:02.187 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:01.911+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:02.187 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:01.911+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:02.187 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:01.912+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:02.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:02.182+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:59:02.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:02.182+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:59:02.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:02.182+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:59:02.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:02.183+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:03.205 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:59:02.788+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 48656 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:59:03.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:02.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:03.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:02.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:03.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:02.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:03.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:02.846+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:03.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:02.936+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:03.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:02.936+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:03.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:02.936+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:03.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:02.938+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:03.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:03.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:59:03.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:03.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:59:03.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:03.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:31.150790+0000 (oldest deadline 2024-04-27T02:48:51.150790+0000) 2024-04-27T02:59:03.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:03.202+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:03.983 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:03.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:03.984 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:03.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:03.984 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:03.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:03.984 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:03.843+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:04.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:04.171+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:04.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:03.979+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:04.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:03.979+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:04.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:03.979+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:04.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:03.980+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:05.146 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:04.956+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:05.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:04.956+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:05.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:04.956+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:05.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:04.957+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:05.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:04.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:05.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:04.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:05.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:04.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:05.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:04.822+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:05.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:05.142+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:06.172 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:05.978+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:06.172 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:05.978+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:06.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:05.978+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:06.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:05.980+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:06.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:05.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:06.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:05.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:06.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:05.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:06.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:05.836+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:06.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:06.167+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:07.131 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:07.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:07.131 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:07.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:07.131 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:07.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:07.131 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:07.003+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:07.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:06.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:07.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:06.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:07.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:06.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:07.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:06.871+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:07.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:07.126+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:08.133 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:59:07.793+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 48700 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:59:08.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:07.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:08.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:07.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:08.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:07.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:08.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:07.884+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:08.134 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:07.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:08.134 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:07.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:08.134 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:07.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:08.134 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:07.976+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:08.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:08.128+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:09.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:09.126+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:09.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:08.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:09.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:08.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:09.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:08.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:09.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:08.977+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:09.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:08.928+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:09.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:08.928+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:09.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:08.928+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:35.587098+0000 (oldest deadline 2024-04-27T02:48:55.587098+0000) 2024-04-27T02:59:09.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:08.928+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:10.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:09.953+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:10.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:10.156+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:10.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:10.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:10.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:10.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:10.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:10.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:10.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:10.000+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:11.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:10.954+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:11.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:11.195+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:11.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:10.982+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:11.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:10.982+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:11.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:10.982+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:11.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:10.983+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:12.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:11.908+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:12.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:11.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:12.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:11.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:12.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:11.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:12.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:11.986+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:12.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:12.220+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:13.188 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:59:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:59:12.799+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 48747 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:59:13.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:12.934+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:13.189 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:12.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:13.189 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:12.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:13.189 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:12.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:13.189 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:12.955+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:13.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:13.184+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:14.202 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:13.916+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:14.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:13.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:14.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:13.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:14.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:13.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:14.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:13.921+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:14.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:14.198+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:15.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:14.905+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:15.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:14.929+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:15.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:14.929+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:15.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:14.929+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:15.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:14.931+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:15.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:15.193+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:16.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:15.885+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:16.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:15.940+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:16.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:15.940+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:16.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:15.940+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:48:40.041617+0000 (oldest deadline 2024-04-27T02:49:00.041617+0000) 2024-04-27T02:59:16.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:15.941+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:16.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:16.224+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:17.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:16.975+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:17.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:16.933+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:17.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:17.191+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:18.157 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:59:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:59:17.805+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 48791 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:59:18.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:17.968+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:18.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:17.959+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:18.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:18.153+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:19.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:18.996+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:19.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:19.148+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:19.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:18.939+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:20.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:20.114+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:20.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:20.032+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:20.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:19.981+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:21.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:20.932+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:21.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:21.144+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:21.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:21.015+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:22.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:21.983+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:22.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:21.887+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:22.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:22.153+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:23.131 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:23.029+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:23.131 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:59:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:59:22.811+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 48837 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:59:23.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:22.859+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:23.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:23.127+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:24.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:23.860+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:24.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:24.020+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:24.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:24.103+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:25.148 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:25.009+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:25.148 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:24.849+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:25.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:25.144+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:26.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:26.049+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:26.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:25.879+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:26.451 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:26.177+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:26.451 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:26.177+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:26.451 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:26.177+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:26.451 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:26.178+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:27.160 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:27.049+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:27.161 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:26.846+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:27.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:27.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:27.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:27.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:27.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:27.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:27.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:27.157+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:28.201 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:59:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:59:27.818+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 48881 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:59:28.201 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:28.006+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:28.202 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:27.865+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:28.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:28.197+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:28.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:28.197+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:28.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:28.197+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:28.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:28.198+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:29.201 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:28.977+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:29.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:28.862+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:29.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:29.197+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:29.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:29.197+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:29.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:29.197+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:29.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:29.197+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:30.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:29.939+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:30.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:29.888+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:30.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:30.210+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:30.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:30.210+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:30.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:30.210+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:30.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:30.211+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:31.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:30.962+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:31.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:30.856+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:31.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:31.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:31.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:31.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:31.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:31.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:31.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:31.259+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:32.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:31.964+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:32.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:31.898+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:32.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:32.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:32.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:32.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:32.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:32.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:32.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:32.265+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:33.224 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:59:32.824+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 48928 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:59:33.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:32.982+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:33.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:32.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:33.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:32.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:33.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:32.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:33.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:32.880+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:33.634 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:33.283+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:33.634 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:33.283+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:33.634 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:33.283+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:33.634 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:33.284+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:34.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:33.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:34.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:33.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:34.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:33.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:34.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:33.841+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:34.335 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:34.026+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:34.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:34.331+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:34.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:34.331+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:34.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:34.331+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:34.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:34.332+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:35.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:34.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:35.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:34.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:35.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:34.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:35.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:34.793+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:35.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:35.071+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:35.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:35.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:35.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:35.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:35.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:35.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:35.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:35.310+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:36.103 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:35.767+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:36.104 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:35.767+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:36.104 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:35.767+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:36.104 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:35.768+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:36.371 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:36.099+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:36.371 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:36.332+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:36.371 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:36.332+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:36.371 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:36.332+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:36.371 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:36.333+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:36.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:36.816+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:36.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:36.816+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:36.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:36.816+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:36.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:36.817+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:37.288 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:37.135+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:37.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:37.284+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:37.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:37.284+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:37.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:37.284+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:37.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:37.285+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:38.128 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:59:37.830+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 48972 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:59:38.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:37.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:38.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:37.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:38.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:37.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:38.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:37.798+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:38.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:38.124+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:38.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:38.294+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:38.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:38.294+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:38.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:38.294+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:38.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:38.294+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:39.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:38.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:39.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:38.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:39.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:38.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:39.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:38.846+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:39.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:39.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:39.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:39.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:39.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:39.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:39.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:39.292+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:39.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:39.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:39.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:39.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:39.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:39.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:39.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:39.104+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:40.120 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:39.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:40.120 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:39.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:40.120 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:39.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:40.120 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:39.801+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:40.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:40.115+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:40.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:40.115+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:40.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:40.115+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:40.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:40.116+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:40.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:40.317+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:40.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:40.317+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:40.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:40.317+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:40.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:40.318+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:41.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:40.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:41.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:40.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:41.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:40.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:41.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:40.787+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:41.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:41.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:41.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:41.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:41.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:41.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:41.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:41.104+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:41.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:41.323+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:41.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:41.323+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:41.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:41.323+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:41.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:41.324+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:41.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:41.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:41.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:41.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:41.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:41.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:41.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:41.800+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:42.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:42.142+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:42.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:42.142+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:42.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:42.142+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:42.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:42.142+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:42.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:42.278+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:42.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:42.278+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:42.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:42.278+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:42.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:42.279+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:43.105 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:59:42.836+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49018 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:59:43.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:42.757+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:43.106 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:42.757+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:43.106 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:42.757+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:43.106 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:42.758+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:43.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:43.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:43.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:43.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:43.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:43.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:43.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:43.101+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:43.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:43.241+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:43.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:43.241+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:43.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:43.241+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:43.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:43.242+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:44.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:43.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:44.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:43.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:44.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:43.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:44.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:43.763+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:44.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:44.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:44.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:44.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:44.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:44.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:44.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:44.104+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:44.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:44.216+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:44.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:44.216+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:44.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:44.216+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:44.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:44.218+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:45.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:44.767+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:45.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:44.767+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:45.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:44.767+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:45.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:44.768+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:45.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:45.227+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:45.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:45.227+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:45.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:45.227+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:45.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:45.228+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:45.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:45.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:45.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:45.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:45.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:45.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:45.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:45.056+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:46.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:45.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:46.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:45.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:46.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:45.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:46.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:45.775+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:46.371 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:46.276+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:46.371 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:46.276+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:46.371 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:46.276+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:46.371 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:46.277+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:46.371 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:46.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:46.372 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:46.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:46.372 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:46.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:46.372 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:46.061+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:46.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:46.815+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:46.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:46.815+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:46.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:46.815+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:46.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:46.815+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:47.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:47.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:47.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:47.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:47.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:47.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:47.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:47.074+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:47.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:47.241+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:47.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:47.241+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:47.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:47.241+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:47.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:47.242+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:48.050 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:59:47.842+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49062 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:59:48.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:47.795+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:48.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:47.795+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:48.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:47.795+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:48.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:47.796+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:48.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:48.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:48.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:48.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:48.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:48.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:48.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:48.047+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:48.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:48.283+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:48.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:48.283+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:48.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:48.283+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:48.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:48.283+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:49.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:49.043+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:49.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:49.043+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:49.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:49.043+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:49.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:49.045+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:49.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:48.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:49.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:48.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:49.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:48.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:49.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:48.817+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:49.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:49.287+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:49.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:49.287+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:49.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:49.287+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:49.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:49.288+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:50.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:49.768+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:50.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:49.768+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:50.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:49.768+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:50.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:49.768+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:50.334 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:50.067+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:50.334 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:50.067+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:50.334 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:50.067+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:50.334 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:50.068+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:50.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:50.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:50.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:50.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:50.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:50.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:50.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:50.330+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:51.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:51.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:51.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:51.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:51.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:51.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:51.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:51.027+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:51.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:50.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:51.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:50.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:51.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:50.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:51.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:50.793+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:51.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:51.332+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:51.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:51.332+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:51.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:51.332+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:51.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:51.333+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:51.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:51.806+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:51.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:51.806+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:51.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:51.806+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:51.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:51.807+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:52.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:52.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:52.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:52.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:52.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:52.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:52.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:52.012+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:52.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:52.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:52.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:52.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:52.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:52.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:52.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:52.375+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:53.154 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:59:52.848+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49109 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:59:53.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:52.809+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:53.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:52.809+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:53.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:52.809+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:53.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:52.809+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:53.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:52.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:53.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:52.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:53.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:52.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:53.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:52.999+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:53.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:53.333+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:53.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:53.333+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:53.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:53.333+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:53.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:53.334+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:54.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:53.810+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:54.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:53.810+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:54.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:53.810+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:54.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:53.810+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:54.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:53.950+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:54.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:53.950+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:54.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:53.950+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:54.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:53.951+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:54.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:54.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:54.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:54.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:54.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:54.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:54.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:54.364+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:55.137 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:54.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:55.137 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:54.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:55.137 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:54.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:55.138 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:54.989+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:55.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:54.796+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:55.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:54.796+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:55.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:54.796+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:55.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:54.797+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:55.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:55.327+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:55.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:55.327+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:55.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:55.327+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:55.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:55.328+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:56.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:55.809+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:56.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:55.809+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:56.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:55.809+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:56.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:55.810+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:56.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:55.949+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:56.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:55.949+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:56.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:55.949+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:56.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:55.950+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:56.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:56.335+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:56.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:56.335+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:56.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:56.335+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:56.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:56.336+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:56.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:56.918+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:56.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:56.918+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:56.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:56.918+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:56.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:56.918+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:56.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:56.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:56.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:56.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:56.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:56.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:56.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:56.786+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:57.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:57.336+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:57.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:57.336+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:57.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:57.336+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:57.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:57.337+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:58.224 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 02:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T02:59:57.854+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49153 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T02:59:58.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:57.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:58.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:57.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:58.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:57.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:58.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:57.832+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:58.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:57.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:58.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:57.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:58.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:57.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:58.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:57.875+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:58.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:58.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:58.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:58.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:58.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:58.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:58.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:58.309+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:59.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:58.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:59.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:58.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:59.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:58.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T02:59:59.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:58.903+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:59.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:58.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:59.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:58.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:59.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:58.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T02:59:59.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:58.818+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T02:59:59.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:59.350+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:59.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:59.350+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:59.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:59.350+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T02:59:59.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 02:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T02:59:59.352+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:00.187 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:59.940+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:00.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:59.940+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:00.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:59.940+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:00.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 02:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T02:59:59.941+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:00.188 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:59.779+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:00.188 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:59.779+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:00.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:59.779+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:00.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 02:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T02:59:59.780+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:00.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:00.345+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:00.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:00.345+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:00.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:00.345+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:00.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:00.347+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:01.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:00.765+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:01.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:00.765+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:01.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:00.765+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:01.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:00.765+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:01.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:00.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:01.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:00.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:01.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:00.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:01.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:00.894+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:01.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:01.343+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:01.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:01.343+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:01.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:01.343+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:01.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:01.343+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:02.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:01.809+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:02.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:01.810+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:02.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:01.810+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:02.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:01.810+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:02.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:01.918+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:02.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:01.918+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:02.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:01.918+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:02.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:01.919+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:02.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:02.318+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:02.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:02.318+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:02.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:02.318+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:02.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:02.319+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:03.224 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:00:02.859+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49199 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:00:03.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:02.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:03.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:02.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:03.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:02.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:03.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:02.815+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:03.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:02.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:03.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:02.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:03.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:02.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:03.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:02.934+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:03.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:03.338+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:03.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:03.338+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:03.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:03.338+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:03.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:03.338+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:04.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:03.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:04.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:03.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:04.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:03.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:04.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:03.939+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:04.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:03.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:04.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:03.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:04.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:03.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:04.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:03.841+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:04.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:04.387+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:04.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:04.387+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:04.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:04.387+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:04.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:04.388+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:05.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:04.825+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:05.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:04.825+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:05.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:04.825+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:05.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:04.825+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:05.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:04.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:05.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:04.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:05.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:04.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:05.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:04.926+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:05.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:05.372+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:05.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:05.372+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:05.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:05.372+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:05.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:05.373+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:06.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:05.935+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:06.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:05.935+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:06.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:05.935+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:06.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:05.936+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:06.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:05.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:06.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:05.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:06.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:05.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:06.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:05.864+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:06.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:06.338+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:06.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:06.338+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:06.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:06.338+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:06.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:06.338+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:07.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:06.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:07.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:06.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:07.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:06.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:07.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:06.897+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:07.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:06.924+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:07.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:06.924+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:07.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:06.924+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:07.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:06.924+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:07.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:07.360+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:07.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:07.360+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:07.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:07.360+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:07.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:07.361+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:08.224 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:00:07.865+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49243 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:00:08.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:07.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:08.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:07.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:08.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:07.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:08.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:07.894+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:08.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:07.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:08.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:07.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:08.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:07.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:08.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:07.856+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:08.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:08.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:08.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:08.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:08.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:08.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:08.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:08.407+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:09.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:08.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:09.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:08.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:09.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:08.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:09.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:08.932+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:09.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:08.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:09.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:08.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:09.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:08.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:09.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:08.878+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:09.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:09.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:09.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:09.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:09.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:09.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:09.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:09.435+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:10.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:09.978+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:10.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:09.978+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:10.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:09.978+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:10.189 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:09.979+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:10.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:09.838+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:10.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:09.838+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:10.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:09.838+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:10.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:09.839+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:10.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:10.468+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:10.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:10.468+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:10.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:10.468+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:10.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:10.468+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:11.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:10.953+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:11.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:10.953+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:11.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:10.953+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:11.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:10.954+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:11.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:10.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:11.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:10.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:11.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:10.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:11.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:10.870+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:11.871 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:11.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:11.871 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:11.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:11.872 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:11.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:11.872 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:11.495+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:12.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:11.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:12.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:11.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:12.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:11.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:12.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:11.969+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:12.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:11.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:12.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:11.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:12.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:11.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:12.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:11.894+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:12.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:12.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:12.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:12.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:12.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:12.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:12.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:12.454+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:13.224 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:00:12.871+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49290 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:00:13.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:12.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:13.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:12.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:13.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:12.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:13.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:12.873+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:13.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:12.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:13.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:12.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:13.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:12.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:13.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:12.999+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:13.859 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:13.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:13.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:13.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:13.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:13.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:13.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:13.502+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:14.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:13.969+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:14.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:13.969+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:14.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:13.969+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:14.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:13.970+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:14.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:13.854+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:14.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:13.854+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:14.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:13.854+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:14.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:13.855+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:14.851 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:14.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:14.851 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:14.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:14.851 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:14.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:14.851 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:14.551+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:15.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:15.019+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:15.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:15.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:15.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:15.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:15.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:15.020+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:15.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:14.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:15.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:14.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:15.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:14.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:15.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:14.847+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:15.859 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:15.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:15.859 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:15.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:15.859 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:15.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:15.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:15.551+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:16.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:16.003+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:16.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:16.003+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:16.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:16.003+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:16.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:16.004+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:16.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:15.855+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:16.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:15.855+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:16.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:15.855+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:16.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:15.855+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:16.642 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:16.561+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:16.642 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:16.561+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:16.642 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:16.561+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:16.642 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:16.562+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:16.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:16.858+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:16.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:16.858+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:16.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:16.858+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:16.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:16.858+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:17.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:17.030+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:17.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:17.030+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:17.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:17.030+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:17.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:17.031+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:17.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:17.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:17.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:17.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:17.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:17.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:17.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:17.520+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:18.224 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:00:17.878+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49334 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:00:18.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:18.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:18.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:18.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:18.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:18.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:18.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:18.051+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:18.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:17.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:18.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:17.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:18.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:17.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:18.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:17.898+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:18.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:18.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:18.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:18.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:18.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:18.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:18.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:18.535+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:19.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:18.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:19.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:18.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:19.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:18.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:19.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:18.917+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:19.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:19.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:19.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:19.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:19.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:19.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:19.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:19.003+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:19.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:19.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:19.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:19.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:19.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:19.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:19.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:19.579+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:20.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:19.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:20.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:19.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:20.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:19.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:20.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:19.936+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:20.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:20.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:20.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:20.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:20.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:20.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:20.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:20.005+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:20.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:20.587+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:20.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:20.587+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:20.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:20.587+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:20.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:20.588+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:21.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:20.970+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:21.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:20.970+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:21.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:20.970+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:21.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:20.971+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:21.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:21.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:21.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:21.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:21.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:21.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:21.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:21.040+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:21.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:21.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:21.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:21.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:21.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:21.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:21.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:21.632+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:22.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:22.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:22.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:22.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:22.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:22.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:22.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:22.062+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:22.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:21.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:22.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:21.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:22.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:21.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:22.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:21.986+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:22.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:22.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:22.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:22.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:22.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:22.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:22.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:22.593+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:23.224 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:00:22.884+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49380 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:00:23.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:23.078+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:23.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:23.078+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:23.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:23.078+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:23.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:23.079+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:23.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:23.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:23.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:23.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:23.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:23.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:23.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:23.005+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:23.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:23.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:23.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:23.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:23.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:23.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:23.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:23.591+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:24.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:24.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:24.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:24.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:24.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:24.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:24.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:24.031+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:24.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:24.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:24.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:24.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:24.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:24.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:24.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:24.090+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:24.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:24.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:24.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:24.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:24.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:24.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:24.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:24.551+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:25.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:25.117+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:25.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:25.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:25.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:25.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:25.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:25.118+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:25.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:25.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:25.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:25.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:25.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:25.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:25.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:25.021+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:25.858 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:25.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:25.858 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:25.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:25.858 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:25.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:25.858 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:25.568+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:26.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:26.096+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:26.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:26.096+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:26.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:26.097+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:26.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:26.097+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:26.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:26.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:26.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:26.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:26.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:26.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:26.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:26.008+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:26.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:26.553+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:26.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:26.553+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:26.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:26.553+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:26.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:26.554+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:27.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:27.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:27.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:27.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:27.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:27.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:27.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:27.067+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:27.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:27.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:27.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:27.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:27.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:27.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:27.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:27.004+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:27.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:27.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:27.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:27.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:27.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:27.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:27.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:27.595+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:28.224 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:00:27.891+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49424 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:00:28.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:28.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:28.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:28.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:28.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:28.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:28.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:28.023+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:28.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:28.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:28.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:28.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:28.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:28.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:28.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:28.020+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:28.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:28.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:28.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:28.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:28.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:28.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:28.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:28.581+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:29.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:28.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:29.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:28.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:29.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:28.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:29.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:28.983+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:29.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:29.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:29.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:29.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:29.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:29.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:29.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:29.061+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:29.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:29.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:29.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:29.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:29.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:29.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:29.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:29.549+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:30.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:30.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:30.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:30.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:30.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:30.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:30.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:30.038+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:30.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:30.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:30.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:30.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:30.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:30.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:30.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:30.010+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:30.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:30.509+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:30.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:30.509+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:30.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:30.509+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:30.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:30.509+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:31.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:31.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:31.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:31.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:31.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:31.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:31.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:31.009+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:31.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:31.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:31.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:31.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:31.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:31.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:31.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:31.037+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:31.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:31.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:31.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:31.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:31.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:31.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:31.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:31.525+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:32.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:32.029+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:32.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:32.029+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:32.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:32.029+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:32.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:32.031+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:32.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:32.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:32.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:32.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:32.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:32.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:32.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:32.049+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:32.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:32.565+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:32.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:32.565+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:32.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:32.565+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:32.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:32.566+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:33.224 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:00:32.897+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49471 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:00:33.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:33.078+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:33.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:33.078+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:33.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:33.078+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:33.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:33.080+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:33.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:33.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:33.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:33.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:33.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:33.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:33.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:33.024+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:33.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:33.613+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:33.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:33.613+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:33.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:33.613+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:33.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:33.614+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:34.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:34.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:34.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:34.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:34.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:34.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:34.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:34.114+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:34.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:33.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:34.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:33.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:34.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:33.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:34.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:33.995+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:34.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:34.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:34.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:34.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:34.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:34.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:34.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:34.665+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:35.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:35.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:35.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:35.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:35.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:35.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:35.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:35.082+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:35.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:34.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:35.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:34.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:35.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:34.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:35.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:34.964+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:35.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:35.626+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:35.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:35.626+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:35.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:35.626+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:35.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:35.627+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:36.372 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:36.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:36.372 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:36.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:36.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:36.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:36.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:36.062+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:36.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:36.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:36.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:36.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:36.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:36.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:36.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:36.013+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:36.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:36.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:36.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:36.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:36.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:36.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:36.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:36.672+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:37.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:37.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:37.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:37.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:37.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:37.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:37.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:37.021+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:37.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:37.106+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:37.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:37.106+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:37.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:37.106+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:37.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:37.107+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:37.974 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:00:37.903+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49515 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:00:37.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:37.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:37.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:37.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:37.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:37.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:37.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:37.694+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:38.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:38.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:38.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:38.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:38.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:38.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:38.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:38.124+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:38.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:38.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:38.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:38.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:38.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:38.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:38.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:38.030+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:38.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:38.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:38.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:38.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:38.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:38.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:38.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:38.679+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:39.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:39.070+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:39.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:39.070+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:39.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:39.070+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:39.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:39.070+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:39.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:39.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:39.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:39.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:39.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:39.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:39.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:39.173+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:39.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:39.703+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:39.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:39.703+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:39.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:39.703+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:39.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:39.704+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:40.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:40.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:40.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:40.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:40.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:40.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:40.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:40.084+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:40.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:40.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:40.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:40.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:40.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:40.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:40.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:40.155+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:40.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:40.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:40.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:40.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:40.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:40.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:40.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:40.714+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:41.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:41.044+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:41.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:41.044+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:41.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:41.044+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:41.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:41.045+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:41.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:41.178+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:41.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:41.178+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:41.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:41.178+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:41.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:41.179+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:41.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:41.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:41.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:41.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:41.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:41.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:41.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:41.700+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:42.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:42.051+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:42.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:42.051+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:42.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:42.051+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:42.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:42.051+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:42.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:42.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:42.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:42.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:42.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:42.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:42.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:42.142+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:42.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:00:42.909+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49561 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:00:42.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:42.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:42.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:42.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:42.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:42.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:42.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:42.678+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:43.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:43.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:43.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:43.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:43.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:43.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:43.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:43.104+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:43.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:43.073+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:43.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:43.073+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:43.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:43.073+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:43.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:43.073+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:43.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:43.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:43.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:43.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:43.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:43.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:43.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:43.677+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:44.291 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:44.035+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:44.291 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:44.035+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:44.291 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:44.035+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:44.291 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:44.036+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:44.291 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:44.092+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:44.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:44.092+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:44.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:44.092+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:44.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:44.093+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:44.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:44.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:44.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:44.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:44.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:44.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:44.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:44.649+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:45.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:45.068+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:45.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:45.068+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:45.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:45.068+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:45.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:45.069+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:45.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:45.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:45.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:45.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:45.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:45.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:45.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:45.085+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:45.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:45.670+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:45.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:45.670+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:45.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:45.670+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:45.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:45.671+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:46.372 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:46.078+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:46.372 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:46.078+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:46.372 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:46.078+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:46.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:46.079+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:46.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:46.114+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:46.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:46.115+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:46.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:46.115+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:46.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:46.115+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:46.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:46.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:46.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:46.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:46.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:46.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:46.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:46.638+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:47.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:47.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:47.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:47.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:47.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:47.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:47.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:47.141+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:47.363 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:47.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:47.363 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:47.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:47.363 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:47.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:47.363 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:47.061+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:47.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:47.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:47.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:47.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:47.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:47.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:47.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:47.611+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:48.224 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:00:47.914+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49605 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:00:48.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:48.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:48.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:48.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:48.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:48.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:48.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:48.102+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:48.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:48.131+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:48.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:48.131+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:48.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:48.131+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:48.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:48.131+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:48.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:48.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:48.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:48.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:48.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:48.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:48.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:48.573+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:49.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:49.097+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:49.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:49.097+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:49.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:49.097+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:49.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:49.097+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:49.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:49.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:49.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:49.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:49.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:49.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:49.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:49.156+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:49.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:49.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:49.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:49.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:49.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:49.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:49.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:49.534+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:50.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:50.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:50.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:50.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:50.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:50.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:50.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:50.147+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:50.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:50.131+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:50.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:50.131+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:50.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:50.131+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:50.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:50.133+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:50.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:50.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:50.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:50.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:50.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:50.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:50.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:50.577+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:51.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:51.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:51.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:51.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:51.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:51.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:51.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:51.158+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:51.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:51.121+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:51.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:51.121+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:51.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:51.121+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:51.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:51.123+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:51.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:51.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:51.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:51.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:51.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:51.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:51.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:51.618+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:52.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:52.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:52.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:52.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:52.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:52.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:52.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:52.165+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:52.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:52.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:52.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:52.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:52.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:52.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:52.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:52.119+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:52.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:52.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:52.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:52.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:52.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:52.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:52.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:52.591+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:53.224 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:00:52.921+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49652 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:00:53.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:53.149+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:53.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:53.149+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:53.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:53.149+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:53.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:53.150+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:53.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:53.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:53.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:53.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:53.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:53.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:53.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:53.121+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:53.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:53.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:53.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:53.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:53.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:53.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:53.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:53.581+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:54.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:54.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:54.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:54.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:54.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:54.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:54.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:54.119+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:54.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:54.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:54.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:54.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:54.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:54.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:54.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:54.100+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:54.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:54.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:54.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:54.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:54.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:54.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:54.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:54.584+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:55.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:55.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:55.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:55.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:55.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:55.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:55.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:55.082+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:55.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:55.088+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:55.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:55.088+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:55.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:55.088+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:55.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:55.089+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:55.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:55.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:55.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:55.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:55.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:55.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:55.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:55.581+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:56.371 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:56.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:56.372 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:56.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:56.372 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:56.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:56.372 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:56.130+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:56.372 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:56.044+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:56.372 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:56.044+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:56.372 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:56.044+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:56.372 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:56.045+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:56.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:56.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:56.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:56.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:56.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:56.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:56.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:56.599+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:57.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:57.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:57.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:57.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:57.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:57.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:57.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:57.147+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:57.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:57.047+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:57.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:57.047+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:57.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:57.047+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:57.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:57.047+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:57.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:57.587+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:57.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:57.587+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:57.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:57.587+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:57.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:57.588+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:58.225 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:00:57.927+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49696 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:00:58.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:58.151+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:58.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:58.151+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:58.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:58.151+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:58.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:58.152+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:58.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:58.044+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:58.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:58.044+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:58.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:58.044+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:58.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:58.045+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:58.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:58.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:58.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:58.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:58.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:58.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:58.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:58.637+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:59.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:59.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:59.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:59.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:59.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:59.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:00:59.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:00:59.192+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:59.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:59.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:59.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:59.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:59.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:59.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:00:59.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:00:59.043+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:00:59.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:59.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:59.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:59.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:59.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:59.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:00:59.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:00:59.632+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:00.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:00.144+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:00.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:00.144+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:00.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:00.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:00.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:00.145+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:00.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:00.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:00.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:00.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:00.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:00.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:00.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:00.090+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:00.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:00.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:00.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:00.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:00.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:00.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:00.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:00.680+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:01.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:01.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:01.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:01.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:01.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:01.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:01.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:01.151+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:01.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:01.129+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:01.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:01.129+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:01.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:01.129+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:01.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:01.130+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:01.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:01.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:01.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:01.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:01.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:01.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:01.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:01.664+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:02.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:02.129+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:02.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:02.129+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:02.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:02.129+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:02.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:02.129+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:02.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:02.179+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:02.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:02.179+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:02.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:02.179+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:02.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:02.180+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:02.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:02.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:02.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:02.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:02.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:02.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:02.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:02.629+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:03.224 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:01:02.933+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49742 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:01:03.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:03.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:03.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:03.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:03.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:03.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:03.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:03.148+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:03.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:03.230+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:03.637 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:03.230+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:03.637 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:03.230+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:03.637 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:03.230+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:03.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:03.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:03.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:03.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:03.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:03.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:03.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:03.632+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:04.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:04.264+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:04.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:04.264+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:04.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:04.264+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:04.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:04.265+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:04.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:04.189+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:04.481 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:04.189+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:04.481 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:04.189+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:04.481 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:04.189+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:04.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:04.605+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:04.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:04.605+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:04.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:04.605+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:04.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:04.606+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:05.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:05.255+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:05.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:05.255+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:05.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:05.255+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:05.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:05.256+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:05.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:05.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:05.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:05.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:05.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:05.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:05.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:05.146+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:05.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:05.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:05.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:05.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:05.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:05.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:05.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:05.651+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:06.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:06.226+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:06.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:06.226+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:06.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:06.226+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:06.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:06.226+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:06.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:06.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:06.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:06.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:06.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:06.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:06.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:06.191+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:06.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:06.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:06.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:06.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:06.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:06.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:06.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:06.659+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:07.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:07.231+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:07.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:07.231+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:07.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:07.231+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:07.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:07.232+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:07.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:07.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:07.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:07.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:07.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:07.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:07.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:07.150+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:07.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:07.613+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:07.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:07.613+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:07.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:07.613+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:07.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:07.613+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:08.225 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:01:07.939+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49786 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:01:08.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:08.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:08.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:08.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:08.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:08.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:08.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:08.162+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:08.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:08.268+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:08.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:08.268+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:08.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:08.268+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:08.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:08.269+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:08.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:08.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:08.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:08.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:08.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:08.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:08.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:08.615+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:09.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:09.305+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:09.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:09.305+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:09.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:09.305+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:09.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:09.306+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:09.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:09.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:09.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:09.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:09.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:09.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:09.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:09.209+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:09.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:09.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:09.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:09.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:09.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:09.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:09.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:09.608+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:10.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:10.329+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:10.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:10.329+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:10.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:10.329+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:10.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:10.331+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:10.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:10.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:10.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:10.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:10.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:10.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:10.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:10.174+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:10.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:10.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:10.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:10.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:10.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:10.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:10.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:10.576+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:11.311 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:11.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:11.311 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:11.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:11.311 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:11.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:11.311 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:11.163+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:11.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:11.305+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:11.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:11.305+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:11.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:11.305+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:11.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:11.306+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:11.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:11.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:11.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:11.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:11.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:11.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:11.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:11.578+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:12.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:12.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:12.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:12.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:12.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:12.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:12.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:12.179+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:12.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:12.313+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:12.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:12.313+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:12.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:12.313+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:12.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:12.314+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:12.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:12.552+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:12.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:12.552+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:12.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:12.552+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:12.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:12.553+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:13.225 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:01:12.945+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49833 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:01:13.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:13.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:13.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:13.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:13.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:13.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:13.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:13.168+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:13.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:13.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:13.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:13.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:13.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:13.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:13.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:13.527+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:13.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:13.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:13.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:13.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:13.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:13.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:13.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:13.355+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:14.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:14.320+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:14.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:14.320+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:14.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:14.320+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:14.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:14.321+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:14.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:14.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:14.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:14.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:14.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:14.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:14.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:14.160+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:14.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:14.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:14.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:14.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:14.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:14.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:14.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:14.547+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:15.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:15.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:15.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:15.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:15.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:15.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:15.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:15.128+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:15.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:15.275+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:15.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:15.275+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:15.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:15.275+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:15.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:15.276+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:15.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:15.538+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:15.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:15.538+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:15.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:15.538+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:15.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:15.539+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:16.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:16.281+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:16.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:16.281+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:16.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:16.281+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:16.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:16.282+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:16.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:16.094+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:16.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:16.094+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:16.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:16.094+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:16.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:16.095+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:16.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:16.585+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:16.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:16.585+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:16.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:16.585+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:16.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:16.585+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:17.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:17.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:17.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:17.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:17.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:17.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:17.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:17.284+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:17.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:17.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:17.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:17.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:17.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:17.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:17.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:17.101+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:17.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:17.626+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:17.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:17.626+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:17.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:17.626+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:17.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:17.627+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:18.225 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:01:17.952+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49877 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:01:18.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:18.051+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:18.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:18.051+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:18.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:18.051+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:18.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:18.052+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:18.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:18.276+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:18.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:18.276+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:18.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:18.276+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:18.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:18.276+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:18.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:18.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:18.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:18.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:18.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:18.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:18.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:18.580+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:19.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:19.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:19.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:19.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:19.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:19.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:19.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:19.014+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:19.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:19.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:19.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:19.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:19.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:19.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:19.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:19.302+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:19.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:19.608+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:19.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:19.608+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:19.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:19.608+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:19.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:19.608+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:20.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:20.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:20.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:20.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:20.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:20.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:20.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:20.009+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:20.602 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:20.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:20.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:20.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:20.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:20.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:20.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:20.278+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:20.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:20.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:20.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:20.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:20.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:20.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:20.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:20.598+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:21.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:21.262+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:21.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:21.262+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:21.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:21.262+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:21.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:21.263+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:21.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:21.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:21.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:21.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:21.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:21.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:21.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:21.017+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:21.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:21.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:21.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:21.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:21.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:21.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:21.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:21.626+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:22.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:21.984+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:22.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:21.984+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:22.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:21.984+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:22.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:21.984+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:22.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:22.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:22.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:22.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:22.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:22.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:22.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:22.305+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:22.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:22.605+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:22.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:22.605+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:22.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:22.605+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:22.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:22.606+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:23.225 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:01:22.959+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49923 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:01:23.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:22.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:23.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:22.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:23.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:22.937+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:23.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:22.938+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:23.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:23.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:23.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:23.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:23.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:23.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:23.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:23.347+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:23.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:23.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:23.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:23.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:23.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:23.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:23.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:23.896+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:23.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:23.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:23.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:23.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:23.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:23.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:23.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:23.650+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:24.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:24.311+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:24.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:24.311+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:24.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:24.311+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:24.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:24.312+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:24.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:24.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:24.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:24.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:24.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:24.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:24.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:24.690+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:24.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:24.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:24.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:24.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:24.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:24.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:24.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:24.885+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:25.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:25.329+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:25.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:25.329+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:25.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:25.329+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:25.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:25.330+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:26.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:25.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:26.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:25.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:26.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:25.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:26.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:25.732+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:26.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:25.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:26.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:25.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:26.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:25.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:26.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:25.895+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:26.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:26.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:26.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:26.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:26.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:26.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:26.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:26.359+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:27.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:26.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:27.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:26.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:27.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:26.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:27.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:26.773+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:27.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:26.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:27.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:26.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:27.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:26.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:27.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:26.930+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:27.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:27.310+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:27.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:27.310+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:27.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:27.310+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:27.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:27.312+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:27.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:27.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:27.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:27.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:27.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:27.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:27.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:27.777+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:28.225 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:01:27.965+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 49967 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:01:28.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:27.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:28.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:27.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:28.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:27.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:28.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:27.922+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:28.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:28.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:28.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:28.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:28.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:28.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:28.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:28.354+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:29.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:28.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:29.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:28.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:29.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:28.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:29.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:28.771+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:29.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:28.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:29.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:28.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:29.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:28.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:29.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:28.924+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:29.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:29.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:29.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:29.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:29.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:29.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:29.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:29.368+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:30.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:29.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:30.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:29.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:30.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:29.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:30.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:29.803+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:30.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:29.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:30.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:29.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:30.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:29.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:30.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:29.948+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:30.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:30.351+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:30.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:30.351+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:30.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:30.351+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:30.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:30.351+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:31.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:30.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:31.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:30.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:31.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:30.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:31.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:30.787+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:31.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:30.996+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:31.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:30.996+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:31.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:30.996+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:31.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:30.997+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:31.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:31.376+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:31.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:31.376+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:31.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:31.376+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:31.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:31.377+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:32.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:32.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:32.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:32.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:32.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:32.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:32.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:32.022+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:32.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:31.783+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:32.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:31.783+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:32.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:31.783+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:32.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:31.784+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:32.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:32.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:32.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:32.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:32.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:32.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:32.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:32.401+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:33.098 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:01:32.972+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 50014 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:01:33.098 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:33.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:33.098 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:33.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:33.098 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:33.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:33.099 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:33.005+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:33.099 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:32.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:33.099 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:32.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:33.099 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:32.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:33.099 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:32.823+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:33.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:33.416+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:33.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:33.416+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:33.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:33.416+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:33.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:33.417+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:34.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:33.863+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:34.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:33.863+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:34.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:33.863+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:34.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:33.864+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:34.401 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:34.050+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:34.401 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:34.050+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:34.401 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:34.050+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:34.401 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:34.051+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:34.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:34.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:34.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:34.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:34.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:34.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:34.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:34.397+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:35.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:34.902+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:35.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:34.902+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:35.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:34.902+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:35.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:34.903+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:35.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:35.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:35.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:35.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:35.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:35.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:35.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:35.021+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:35.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:35.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:35.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:35.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:35.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:35.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:35.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:35.370+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:36.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:35.949+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:36.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:35.949+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:36.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:35.949+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:36.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:35.950+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:36.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:35.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:36.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:35.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:36.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:35.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:36.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:35.976+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:36.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:36.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:36.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:36.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:36.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:36.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:36.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:36.366+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:37.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:36.915+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:37.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:36.915+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:37.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:36.915+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:37.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:36.917+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:37.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:36.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:37.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:36.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:37.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:36.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:37.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:36.981+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:37.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:37.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:37.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:37.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:37.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:37.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:37.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:37.365+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:37.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:37.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:37.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:37.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:37.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:37.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:37.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:37.901+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:38.359 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:01:37.978+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 50058 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:01:38.360 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:38.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:38.360 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:38.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:38.360 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:38.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:38.360 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:38.015+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:38.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:38.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:38.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:38.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:38.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:38.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:38.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:38.355+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:39.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:38.983+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:39.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:38.983+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:39.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:38.983+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:39.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:38.983+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:39.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:38.897+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:39.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:38.897+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:39.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:38.897+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:39.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:38.898+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:39.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:39.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:39.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:39.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:39.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:39.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:39.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:39.402+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:40.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:39.871+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:40.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:39.871+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:40.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:39.871+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:40.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:39.872+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:40.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:39.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:40.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:39.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:40.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:39.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:40.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:39.986+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:40.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:40.361+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:40.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:40.361+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:40.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:40.361+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:40.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:40.361+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:41.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:40.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:41.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:40.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:41.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:40.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:41.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:40.836+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:41.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:41.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:41.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:41.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:41.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:41.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:41.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:41.031+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:41.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:41.404+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:41.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:41.404+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:41.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:41.404+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:41.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:41.405+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:41.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:41.857+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:41.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:41.857+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:41.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:41.857+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:41.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:41.858+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:42.424 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:42.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:42.424 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:42.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:42.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:42.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:42.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:42.024+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:42.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:42.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:42.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:42.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:42.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:42.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:42.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:42.420+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:42.990 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:42.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:42.990 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:42.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:42.990 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:42.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:42.990 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:42.899+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:43.381 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:01:42.984+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 50104 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:01:43.381 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:43.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:43.381 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:43.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:43.381 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:43.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:43.381 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:43.008+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:43.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:43.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:43.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:43.376+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:43.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:43.376+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:43.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:43.376+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:44.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:43.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:44.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:43.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:44.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:43.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:44.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:43.969+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:44.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:43.927+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:44.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:43.927+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:44.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:43.927+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:44.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:43.928+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:44.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:44.423+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:44.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:44.423+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:44.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:44.423+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:44.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:44.423+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:45.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:45.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:45.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:45.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:45.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:45.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:45.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:45.004+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:45.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:44.883+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:45.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:44.883+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:45.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:44.883+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:45.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:44.883+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:45.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:45.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:45.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:45.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:45.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:45.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:45.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:45.446+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:46.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:45.887+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:46.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:45.887+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:46.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:45.887+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:46.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:45.888+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:46.372 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:46.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:46.372 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:46.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:46.372 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:46.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:46.372 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:46.037+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:46.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:46.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:46.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:46.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:46.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:46.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:46.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:46.482+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:47.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:46.927+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:47.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:46.927+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:47.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:46.927+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:47.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:46.928+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:47.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:47.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:47.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:47.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:47.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:47.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:47.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:47.028+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:47.780 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:47.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:47.780 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:47.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:47.780 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:47.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:47.780 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:47.482+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:48.030 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:01:47.990+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 50148 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:01:48.030 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:47.951+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:48.030 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:47.951+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:48.030 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:47.951+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:48.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:47.952+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:48.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:48.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:48.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:48.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:48.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:48.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:48.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:48.025+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:48.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:48.499+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:48.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:48.499+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:48.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:48.499+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:48.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:48.499+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:49.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:49.040+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:49.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:49.040+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:49.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:49.040+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:49.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:49.040+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:49.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:48.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:49.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:48.978+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:49.462 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:48.978+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:49.462 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:48.978+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:49.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T03:01:49.607+0000 7ff0e4ed9700 0 monclient(hunting): authenticate timed out after 300 2024-04-27T03:01:49.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:49.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:49.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:49.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:49.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:49.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:49.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:49.456+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:50.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:49.988+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:50.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:49.988+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:50.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:49.988+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:50.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:49.989+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:50.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:50.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:50.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:50.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:50.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:50.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:50.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:50.010+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:50.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:50.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:50.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:50.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:50.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:50.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:50.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:50.502+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:51.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:51.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:51.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:51.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:51.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:51.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:51.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:51.026+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:51.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:51.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:51.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:51.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:51.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:51.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:51.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:51.024+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:51.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:51.480+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:51.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:51.480+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:51.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:51.480+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:51.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:51.480+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:52.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:52.060+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:52.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:52.060+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:52.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:52.060+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:52.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:52.061+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:52.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:52.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:52.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:52.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:52.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:52.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:52.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:52.021+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:52.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:52.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:52.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:52.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:52.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:52.452+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:53.123 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:01:52.996+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 50196 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:01:53.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:53.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:53.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:53.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:53.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:53.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:53.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:53.027+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:53.124 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:53.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:53.124 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:53.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:53.124 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:53.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:53.124 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:53.031+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:53.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:53.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:53.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:53.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:53.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:53.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:53.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:53.496+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:54.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:54.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:54.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:54.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:54.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:54.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:54.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:54.029+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:54.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:54.021+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:54.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:54.021+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:54.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:54.021+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:54.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:54.022+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:54.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:54.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:54.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:54.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:54.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:54.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:54.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:54.494+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:55.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:55.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:55.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:55.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:55.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:55.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:55.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:55.066+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:55.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:55.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:55.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:55.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:55.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:55.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:55.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:55.028+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:55.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:55.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:55.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:55.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:55.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:55.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:55.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:55.529+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:56.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:56.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:56.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:56.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:56.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:56.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:56.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:56.076+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:56.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:56.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:56.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:56.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:56.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:56.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:56.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:56.009+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:56.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:56.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:56.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:56.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:56.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:56.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:56.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:56.480+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:57.457 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:57.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:57.457 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:57.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:57.457 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:57.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:57.457 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:57.099+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:57.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:57.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:57.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:57.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:57.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:57.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:57.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:57.017+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:57.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:57.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:57.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:57.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:57.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:57.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:57.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:57.452+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:58.129 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:01:58.003+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 50240 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:01:58.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:58.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:58.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:58.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:58.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:58.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:58.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:58.031+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:58.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:58.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:58.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:58.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:58.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:58.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:58.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:58.125+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:58.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:58.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:58.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:58.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:58.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:58.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:58.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:58.480+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:59.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:59.147+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:59.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:59.147+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:59.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:59.147+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:01:59.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:01:59.148+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:59.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:59.046+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:59.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:59.046+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:59.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:59.046+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:01:59.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:01:59.047+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:01:59.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:59.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:59.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:59.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:59.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:59.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:01:59.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:01:59.492+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:00.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:00.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:00.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:00.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:00.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:00.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:00.467 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:00.049+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:00.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:00.186+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:00.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:00.186+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:00.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:00.186+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:00.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:00.187+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:00.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:00.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:00.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:00.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:00.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:00.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:00.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:00.462+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:01.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:01.177+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:01.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:01.177+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:01.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:01.177+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:01.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:01.178+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:01.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:01.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:01.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:01.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:01.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:01.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:01.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:01.034+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:01.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:01.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:01.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:01.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:01.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:01.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:01.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:01.481+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:02.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:02.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:02.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:02.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:02.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:02.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:02.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:02.024+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:02.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:02.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:02.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:02.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:02.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:02.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:02.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:02.219+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:02.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:02.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:02.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:02.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:02.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:02.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:02.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:02.480+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:03.176 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:02:03.008+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 50286 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:02:03.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:03.067+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:03.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:03.067+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:03.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:03.067+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:03.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:03.068+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:03.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:03.169+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:03.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:03.169+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:03.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:03.169+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:03.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:03.170+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:03.738 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:03.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:03.738 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:03.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:03.738 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:03.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:03.738 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:03.478+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:04.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:04.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:04.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:04.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:04.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:04.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:04.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:04.085+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:04.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:04.166+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:04.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:04.166+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:04.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:04.166+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:04.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:04.166+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:04.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:04.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:04.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:04.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:04.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:04.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:04.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:04.528+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:05.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:05.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:05.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:05.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:05.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:05.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:05.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:05.162+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:05.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:05.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:05.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:05.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:05.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:05.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:05.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:05.084+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:05.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:05.532+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:05.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:05.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:05.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:05.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:05.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:05.533+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:06.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:06.081+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:06.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:06.081+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:06.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:06.081+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:06.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:06.082+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:06.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:06.166+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:06.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:06.166+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:06.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:06.166+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:06.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:06.166+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:06.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:06.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:06.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:06.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:06.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:06.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:06.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:06.549+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:07.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:07.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:07.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:07.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:07.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:07.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:07.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:07.033+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:07.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:07.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:07.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:07.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:07.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:07.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:07.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:07.201+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:07.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:07.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:07.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:07.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:07.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:07.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:07.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:07.515+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:08.226 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:02:08.015+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 50330 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:02:08.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:08.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:08.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:08.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:08.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:08.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:08.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:08.022+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:08.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:08.171+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:08.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:08.171+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:08.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:08.171+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:08.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:08.172+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:08.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:08.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:08.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:08.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:08.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:08.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:08.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:08.495+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:09.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:09.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:09.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:09.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:09.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:09.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:09.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:09.053+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:09.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:09.185+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:09.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:09.185+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:09.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:09.185+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:09.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:09.186+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:09.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:09.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:09.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:09.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:09.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:09.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:09.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:09.468+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:10.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:10.216+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:10.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:10.216+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:10.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:10.216+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:10.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:10.217+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:10.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:10.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:10.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:10.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:10.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:10.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:10.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:10.096+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:10.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:10.510+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:10.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:10.510+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:10.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:10.510+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:10.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:10.510+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:11.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:11.179+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:11.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:11.179+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:11.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:11.179+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:11.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:11.180+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:11.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:11.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:11.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:11.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:11.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:11.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:11.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:11.058+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:11.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:11.480+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:11.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:11.480+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:11.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:11.480+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:11.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:11.480+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:12.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:12.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:12.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:12.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:12.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:12.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:12.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:12.024+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:12.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:12.223+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:12.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:12.223+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:12.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:12.223+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:12.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:12.223+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:12.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:12.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:12.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:12.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:12.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:12.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:12.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:12.513+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:13.225 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:02:13.021+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 50377 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:02:13.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:13.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:13.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:13.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:13.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:13.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:13.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:13.049+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:13.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:13.244+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:13.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:13.244+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:13.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:13.244+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:13.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:13.244+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:13.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:13.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:13.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:13.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:13.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:13.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:13.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:13.481+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:14.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:14.283+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:14.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:14.284+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:14.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:14.284+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:14.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:14.285+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:14.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:14.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:14.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:14.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:14.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:14.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:14.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:14.056+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:14.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:14.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:14.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:14.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:14.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:14.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:14.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:14.513+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:15.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:15.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:15.304 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:15.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:15.304 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:15.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:15.304 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:15.022+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:15.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:15.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:15.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:15.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:15.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:15.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:15.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:15.510+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:15.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:15.298+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:15.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:15.298+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:15.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:15.298+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:15.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:15.299+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:16.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:15.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:16.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:15.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:16.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:15.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:16.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:15.998+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:16.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:16.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:16.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:16.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:16.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:16.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:16.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:16.534+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:16.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:16.303+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:16.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:16.303+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:16.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:16.303+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:16.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:16.305+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:17.311 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:17.034+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:17.311 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:17.034+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:17.312 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:17.034+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:17.312 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:17.035+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:17.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:17.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:17.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:17.549+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:17.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:17.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:17.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:17.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:17.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:17.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:17.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:17.307+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:18.324 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:02:18.029+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 50421 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:02:18.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:18.050+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:18.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:18.050+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:18.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:18.050+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:18.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:18.050+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:18.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:18.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:18.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:18.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:18.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:18.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:18.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:18.530+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:18.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:18.319+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:18.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:18.319+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:18.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:18.319+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:18.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:18.320+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:19.315 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:19.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:19.316 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:19.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:19.316 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:19.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:19.316 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:19.028+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:19.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:19.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:19.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:19.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:19.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:19.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:19.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:19.311+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:19.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:19.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:19.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:19.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:19.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:19.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:19.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:19.497+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:20.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:19.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:20.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:19.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:20.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:19.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:20.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:19.993+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:20.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:20.335+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:20.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:20.335+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:20.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:20.335+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:20.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:20.336+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:20.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:20.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:20.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:20.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:20.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:20.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:20.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:20.495+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:21.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:21.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:21.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:21.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:21.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:21.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:21.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:21.005+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:21.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:21.480+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:21.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:21.480+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:21.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:21.480+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:21.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:21.481+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:21.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:21.353+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:21.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:21.353+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:21.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:21.353+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:21.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:21.354+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:22.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:22.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:22.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:22.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:22.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:22.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:22.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:22.017+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:22.594 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:22.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:22.595 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:22.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:22.595 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:22.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:22.595 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:22.528+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:22.595 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:22.324+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:22.595 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:22.324+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:22.595 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:22.324+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:22.595 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:22.325+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:23.327 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:02:23.035+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 50467 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:02:23.327 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:23.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:23.327 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:23.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:23.327 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:23.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:23.327 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:23.045+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:23.523+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:23.523+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:23.523+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:23.525+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:23.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:23.321+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:23.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:23.321+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:23.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:23.321+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:23.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:23.322+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:24.371 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:24.000+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:24.372 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:24.000+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:24.372 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:24.000+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:24.372 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:24.001+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:24.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:24.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:24.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:24.498+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:24.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:24.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:24.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:24.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:24.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:24.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:24.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:24.367+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:25.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:25.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:25.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:25.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:25.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:25.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:25.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:25.014+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:25.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:25.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:25.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:25.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:25.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:25.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:25.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:25.349+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:25.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:25.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:25.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:25.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:25.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:25.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:25.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:25.486+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:26.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:26.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:26.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:26.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:26.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:26.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:26.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:26.039+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:26.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:26.339+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:26.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:26.339+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:26.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:26.339+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:26.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:26.339+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:26.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:26.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:26.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:26.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:26.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:26.519+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:27.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:27.046+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:27.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:27.046+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:27.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:27.046+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:27.331 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:27.046+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:27.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:27.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:27.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:27.551+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:27.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:27.324+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:27.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:27.324+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:27.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:27.324+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:27.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:27.325+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:28.299 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:02:28.041+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 50512 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:02:28.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:28.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:28.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:28.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:28.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:28.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:28.300 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:28.055+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:28.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:28.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:28.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:28.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:28.543+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:28.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:28.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:28.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:28.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:28.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:28.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:28.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:28.294+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:29.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:29.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:29.331 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:29.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:29.331 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:29.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:29.331 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:29.052+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:29.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:29.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:29.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:29.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:29.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:29.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:29.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:29.559+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:29.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:29.325+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:29.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:29.325+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:29.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:29.325+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:29.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:29.326+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:30.372 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:30.011+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:30.372 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:30.011+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:30.372 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:30.011+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:30.372 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:30.011+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:30.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:30.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:30.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:30.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:30.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:30.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:30.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:30.513+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:30.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:30.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:30.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:30.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:30.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:30.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:30.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:30.367+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:31.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:31.035+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:31.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:31.035+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:31.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:31.035+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:31.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:31.035+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:31.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:31.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:31.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:31.483+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:31.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:31.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:31.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:31.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:31.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:31.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:31.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:31.331+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:32.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:32.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:32.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:32.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:32.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:32.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:32.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:32.034+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:32.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:32.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:32.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:32.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:32.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:32.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:32.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:32.464+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:32.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:32.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:32.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:32.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:32.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:32.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:32.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:32.349+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:33.398 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:02:33.048+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 50558 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:02:33.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:33.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:33.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:33.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:33.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:33.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:33.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:33.061+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:33.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:33.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:33.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:33.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:33.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:33.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:33.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:33.513+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:33.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:33.392+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:33.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:33.392+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:33.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:33.392+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:33.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:33.392+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:34.383 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:34.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:34.383 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:34.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:34.383 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:34.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:34.383 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:34.111+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:34.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:34.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:34.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:34.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:34.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:34.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:34.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:34.505+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:34.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:34.376+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:34.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:34.376+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:34.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:34.376+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:34.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:34.378+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:35.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:35.068+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:35.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:35.068+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:35.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:35.068+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:35.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:35.068+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:35.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:35.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:35.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:35.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:35.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:35.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:35.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:35.471+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:35.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:35.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:35.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:35.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:35.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:35.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:35.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:35.424+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:36.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:36.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:36.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:36.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:36.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:36.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:36.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:36.042+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:36.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:36.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:36.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:36.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:36.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:36.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:36.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:36.486+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:36.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:36.438+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:36.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:36.438+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:36.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:36.438+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:36.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:36.439+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:37.467 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:37.074+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:37.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:37.074+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:37.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:37.074+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:37.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:37.074+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:37.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:37.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:37.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:37.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:37.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:37.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:37.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:37.530+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:37.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:37.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:37.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:37.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:37.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:37.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:37.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:37.462+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:38.224 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:02:38.054+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 50602 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:02:38.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:38.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:38.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:38.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:38.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:38.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:38.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:38.029+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:38.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:38.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:38.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:38.476+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:38.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:38.510+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:38.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:38.510+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:38.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:38.510+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:38.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:38.510+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:39.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:39.067+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:39.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:39.067+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:39.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:39.067+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:39.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:39.068+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:39.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:39.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:39.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:39.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:39.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:39.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:39.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:39.464+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:39.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:39.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:39.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:39.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:39.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:39.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:39.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:39.441+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:40.408 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:40.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:40.408 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:40.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:40.408 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:40.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:40.408 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:40.111+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:40.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:40.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:40.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:40.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:40.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:40.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:40.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:40.464+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:40.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:40.402+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:40.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:40.402+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:40.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:40.402+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:40.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:40.403+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:41.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:41.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:41.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:41.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:41.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:41.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:41.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:41.159+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:41.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:41.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:41.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:41.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:41.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:41.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:41.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:41.504+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:41.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:41.356+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:41.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:41.356+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:41.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:41.356+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:41.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:41.357+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:42.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:42.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:42.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:42.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:42.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:42.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:42.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:42.315+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:42.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:42.144+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:42.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:42.144+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:42.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:42.144+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:42.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:42.145+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:42.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:42.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:42.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:42.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:42.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:42.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:42.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:42.515+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:43.343 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:02:43.061+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 50648 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:02:43.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:43.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:43.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:43.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:43.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:43.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:43.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:43.171+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:43.651 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:43.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:43.652 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:43.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:43.652 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:43.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:43.652 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:43.522+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:43.652 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:43.337+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:43.652 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:43.337+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:43.652 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:43.337+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:43.652 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:43.338+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:44.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:44.327+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:44.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:44.327+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:44.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:44.327+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:44.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:44.328+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:44.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:44.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:44.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:44.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:44.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:44.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:44.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:44.209+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:44.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:44.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:44.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:44.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:44.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:44.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:44.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:44.497+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:45.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:45.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:45.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:45.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:45.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:45.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:45.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:45.170+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:45.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:45.324+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:45.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:45.324+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:45.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:45.324+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:45.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:45.325+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:45.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:45.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:45.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:45.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:45.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:45.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:45.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:45.528+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:46.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:46.335+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:46.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:46.335+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:46.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:46.335+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:46.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:46.336+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:46.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:46.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:46.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:46.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:46.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:46.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:46.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:46.170+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:46.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:46.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:46.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:46.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:46.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:46.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:46.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:46.562+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:47.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:47.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:47.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:47.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:47.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:47.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:47.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:47.217+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:47.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:47.339+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:47.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:47.339+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:47.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:47.339+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:47.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:47.340+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:47.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:47.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:47.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:47.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:47.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:47.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:47.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:47.608+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:48.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:02:48.067+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 50693 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:02:48.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:48.186+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:48.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:48.186+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:48.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:48.186+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:48.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:48.186+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:48.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:48.301+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:48.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:48.301+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:48.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:48.301+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:48.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:48.302+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:48.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:48.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:48.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:48.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:48.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:48.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:48.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:48.643+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:49.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:49.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:49.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:49.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:49.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:49.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:49.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:49.229+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:49.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:49.302+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:49.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:49.302+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:49.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:49.302+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:49.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:49.303+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:49.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:49.680+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:49.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:49.680+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:49.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:49.680+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:49.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:49.681+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:50.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:50.217+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:50.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:50.217+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:50.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:50.217+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:50.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:50.218+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:50.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:50.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:50.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:50.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:50.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:50.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:50.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:50.345+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:50.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:50.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:50.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:50.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:50.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:50.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:50.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:50.654+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:51.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:51.315+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:51.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:51.315+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:51.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:51.315+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:51.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:51.316+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:51.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:51.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:51.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:51.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:51.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:51.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:51.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:51.174+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:51.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:51.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:51.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:51.608+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:51.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:51.608+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:51.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:51.609+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:52.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:52.334+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:52.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:52.334+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:52.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:52.334+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:52.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:52.336+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:52.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:52.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:52.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:52.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:52.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:52.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:52.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:52.215+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:52.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:52.656+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:52.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:52.656+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:52.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:52.656+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:52.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:52.657+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:53.357 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:02:53.073+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 50739 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:02:53.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:53.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:53.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:53.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:53.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:53.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:53.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:53.261+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:53.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:53.370+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:53.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:53.370+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:53.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:53.370+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:53.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:53.371+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:53.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:53.674+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:53.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:53.674+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:53.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:53.674+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:53.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:53.675+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:54.648 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:54.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:54.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:54.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:54.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:54.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:54.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:54.311+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:54.649 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:54.410+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:54.649 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:54.410+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:54.649 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:54.410+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:54.649 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:54.411+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:54.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:54.642+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:54.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:54.642+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:54.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:54.642+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:54.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:54.643+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:55.686 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:55.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:55.686 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:55.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:55.686 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:55.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:55.686 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:55.419+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:55.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:55.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:55.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:55.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:55.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:55.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:55.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:55.339+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:55.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:55.679+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:55.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:55.679+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:55.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:55.679+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:55.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:55.681+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:56.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:56.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:56.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:56.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:56.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:56.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:56.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:56.325+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:56.659 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:56.385+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:56.659 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:56.385+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:56.659 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:56.385+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:56.660 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:56.386+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:56.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:56.652+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:56.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:56.652+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:56.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:56.652+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:56.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:56.653+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:57.706 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:57.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:57.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:57.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:57.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:57.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:57.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:57.328+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:57.707 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:57.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:57.707 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:57.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:57.707 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:57.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:57.707 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:57.421+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:57.700+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:57.700+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:57.700+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:57.701+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:58.400 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:02:58.080+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 50784 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:02:58.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:58.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:58.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:58.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:58.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:58.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:58.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:58.286+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:58.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:58.394+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:58.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:58.394+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:58.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:58.394+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:58.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:58.395+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:58.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:58.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:58.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:58.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:58.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:58.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:58.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:58.684+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:59.424 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:59.241+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:59.424 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:59.241+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:59.424 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:59.241+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:02:59.424 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:02:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:02:59.241+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:59.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:59.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:59.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:59.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:59.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:59.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:02:59.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:02:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:02:59.419+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:02:59.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:59.709+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:59.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:59.709+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:59.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:59.709+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:02:59.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:02:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:02:59.710+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:00.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:00.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:00.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:00.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:00.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:00.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:00.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:00.436+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:00.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:00.250+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:00.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:00.250+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:00.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:00.250+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:00.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:00.250+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:01.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:00.738+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:01.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:00.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:01.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:00.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:01.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:00.740+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:01.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:01.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:01.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:01.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:01.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:01.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:01.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:01.446+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:01.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:01.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:01.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:01.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:01.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:01.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:01.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:01.246+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:02.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:01.753+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:02.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:01.753+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:02.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:01.753+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:02.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:01.754+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:02.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:02.200+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:02.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:02.200+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:02.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:02.200+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:02.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:02.201+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:02.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:02.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:02.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:02.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:02.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:02.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:02.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:02.712+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:02.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:02.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:02.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:02.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:02.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:02.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:02.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:02.491+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:03.446 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:03:03.085+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 50830 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:03:03.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:03.217+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:03.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:03.217+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:03.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:03.217+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:03.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:03.218+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:03.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:03.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:03.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:03.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:03.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:03.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:03.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:03.493+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:04.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:03.742+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:04.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:03.742+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:04.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:03.742+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:04.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:03.743+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:04.548 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:04.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:04.549 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:04.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:04.549 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:04.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:04.549 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:04.262+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:04.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:04.541+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:04.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:04.541+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:04.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:04.541+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:04.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:04.542+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:04.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:04.741+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:04.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:04.741+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:04.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:04.741+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:04.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:04.742+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:05.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:05.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:05.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:05.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:05.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:05.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:05.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:05.494+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:05.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:05.265+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:05.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:05.265+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:05.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:05.265+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:05.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:05.266+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:05.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:05.752+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:05.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:05.752+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:05.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:05.752+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:05.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:05.753+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:06.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:06.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:06.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:06.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:06.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:06.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:06.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:06.482+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:06.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:06.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:06.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:06.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:06.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:06.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:06.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:06.311+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:07.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:06.801+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:07.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:06.801+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:07.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:06.801+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:07.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:06.802+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:07.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:07.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:07.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:07.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:07.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:07.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:07.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:07.456+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:07.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:07.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:07.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:07.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:07.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:07.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:07.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:07.324+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:08.098 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:07.760+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:08.099 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:07.760+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:08.099 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:07.760+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:08.099 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:07.761+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:08.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:03:08.091+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 50875 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:03:08.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:08.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:08.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:08.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:08.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:08.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:08.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:08.297+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:08.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:08.723+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:08.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:08.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:08.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:08.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:08.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:08.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:08.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:08.473+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:09.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:08.723+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:09.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:08.723+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:09.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:08.723+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:09.709 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:09.256+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:09.709 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:09.256+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:09.709 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:09.256+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:09.709 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:09.257+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:09.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:09.505+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:09.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:09.505+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:09.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:09.505+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:09.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:09.506+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:09.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:09.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:09.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:09.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:09.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:09.701+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:09.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:09.702+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:10.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:10.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:10.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:10.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:10.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:10.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:10.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:10.297+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:10.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:10.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:10.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:10.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:10.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:10.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:10.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:10.678+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:10.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:10.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:10.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:10.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:10.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:10.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:10.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:10.536+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:11.538 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:11.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:11.538 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:11.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:11.538 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:11.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:11.538 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:11.265+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:11.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:11.693+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:11.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:11.693+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:11.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:11.693+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:11.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:11.694+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:11.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:11.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:11.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:11.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:11.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:11.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:11.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:11.532+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:12.547 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:12.243+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:12.548 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:12.243+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:12.548 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:12.243+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:12.548 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:12.244+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:12.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:12.655+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:12.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:12.655+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:12.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:12.655+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:12.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:12.656+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:12.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:12.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:12.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:12.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:12.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:12.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:12.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:12.541+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:13.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:03:13.098+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 50921 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:03:13.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:13.231+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:13.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:13.231+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:13.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:13.231+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:13.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:13.231+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:13.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:13.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:13.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:13.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:13.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:13.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:13.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:13.682+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:13.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:13.510+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:13.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:13.510+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:13.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:13.510+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:13.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:13.512+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:14.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:14.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:14.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:14.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:14.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:14.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:14.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:14.216+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:14.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:14.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:14.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:14.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:14.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:14.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:14.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:14.682+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:14.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:14.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:14.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:14.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:14.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:14.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:14.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:14.512+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:15.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:15.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:15.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:15.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:15.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:15.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:15.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:15.206+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:15.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:15.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:15.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:15.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:15.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:15.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:15.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:15.718+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:15.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:15.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:15.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:15.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:15.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:15.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:15.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:15.487+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:16.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:16.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:16.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:16.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:16.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:16.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:16.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:16.247+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:16.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:16.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:16.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:16.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:16.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:16.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:16.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:16.484+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:17.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:16.721+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:17.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:16.721+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:17.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:16.721+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:17.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:16.722+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:17.504 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:17.231+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:17.504 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:17.231+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:17.504 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:17.231+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:17.504 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:17.232+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:17.758 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:17.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:17.758 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:17.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:17.759 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:17.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:17.759 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:17.498+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:18.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:17.752+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:18.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:17.752+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:18.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:17.752+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:18.112 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:17.753+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:18.463 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:03:18.105+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 50966 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:03:18.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:18.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:18.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:18.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:18.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:18.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:18.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:18.233+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:18.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:18.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:18.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:18.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:18.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:18.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:18.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:18.458+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:19.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:18.736+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:19.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:18.736+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:19.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:18.736+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:19.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:18.737+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:19.706 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:19.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:19.707 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:19.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:19.707 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:19.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:19.707 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:19.421+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:19.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:19.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:19.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:19.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:19.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:19.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:19.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:19.240+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:19.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:19.700+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:19.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:19.700+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:19.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:19.700+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:19.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:19.701+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:20.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:20.417+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:20.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:20.417+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:20.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:20.417+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:20.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:20.419+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:20.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:20.237+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:20.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:20.237+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:20.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:20.237+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:20.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:20.237+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:21.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:20.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:21.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:20.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:21.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:20.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:21.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:20.740+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:21.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:21.399+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:21.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:21.399+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:21.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:21.399+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:21.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:21.400+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:21.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:21.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:21.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:21.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:21.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:21.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:21.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:21.270+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:22.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:21.730+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:22.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:21.730+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:22.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:21.730+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:22.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:21.731+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:22.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:22.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:22.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:22.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:22.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:22.438+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:22.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:22.438+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:22.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:22.274+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:22.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:22.274+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:22.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:22.274+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:22.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:22.275+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:23.117 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:22.767+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:23.118 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:22.767+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:23.118 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:22.767+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:23.118 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:22.768+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:23.443 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:03:23.111+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 51012 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:03:23.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:23.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:23.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:23.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:23.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:23.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:23.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:23.326+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:23.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:23.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:23.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:23.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:23.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:23.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:23.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:23.437+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:24.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:23.779+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:24.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:23.779+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:24.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:23.779+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:24.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:23.780+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:24.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:24.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:24.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:24.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:24.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:24.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:24.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:24.278+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:24.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:24.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:24.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:24.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:24.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:24.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:24.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:24.484+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:25.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:24.815+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:25.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:24.815+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:25.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:24.815+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:25.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:24.816+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:25.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:25.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:25.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:25.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:25.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:25.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:25.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:25.452+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:25.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:25.299+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:25.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:25.299+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:25.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:25.299+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:25.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:25.299+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:26.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:25.801+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:26.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:25.801+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:26.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:25.801+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:26.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:25.802+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:26.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:26.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:26.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:26.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:26.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:26.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:26.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:26.457+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:26.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:26.303+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:26.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:26.303+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:26.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:26.303+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:26.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:26.303+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:27.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:26.832+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:27.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:26.832+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:27.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:26.832+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:27.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:26.833+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:27.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:27.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:27.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:27.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:27.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:27.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:27.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:27.446+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:27.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:27.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:27.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:27.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:27.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:27.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:27.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:27.326+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:28.123 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:27.806+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:28.124 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:27.806+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:28.124 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:27.806+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:28.124 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:27.808+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:28.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:03:28.117+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 51057 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:03:28.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:28.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:28.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:28.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:28.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:28.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:28.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:28.290+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:28.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:28.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:28.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:28.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:28.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:28.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:28.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:28.453+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:29.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:28.834+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:29.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:28.834+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:29.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:28.834+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:29.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:28.835+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:29.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:29.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:29.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:29.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:29.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:29.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:29.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:29.435+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:29.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:29.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:29.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:29.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:29.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:29.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:29.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:29.247+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:30.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:29.806+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:30.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:29.806+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:30.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:29.806+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:30.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:29.807+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:30.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:30.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:30.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:30.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:30.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:30.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:30.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:30.254+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:30.775 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:30.476+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:30.775 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:30.476+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:30.775 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:30.476+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:30.776 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:30.476+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:31.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:30.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:31.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:30.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:31.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:30.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:31.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:30.770+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:31.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:31.443+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:31.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:31.443+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:31.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:31.443+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:31.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:31.444+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:31.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:31.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:31.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:31.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:31.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:31.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:31.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:31.216+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:32.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:31.792+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:32.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:31.792+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:32.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:31.792+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:32.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:31.793+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:32.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:32.435+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:32.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:32.435+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:32.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:32.435+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:32.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:32.436+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:32.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:32.229+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:32.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:32.229+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:32.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:32.229+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:32.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:32.230+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:33.130 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:32.840+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:33.131 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:32.840+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:33.131 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:32.840+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:33.131 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:32.841+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:33.471 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:03:33.124+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 51103 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:03:33.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:33.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:33.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:33.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:33.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:33.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:33.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:33.191+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:33.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:33.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:33.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:33.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:33.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:33.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:33.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:33.466+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:34.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:33.869+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:34.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:33.869+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:34.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:33.869+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:34.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:33.870+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:34.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:34.207+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:34.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:34.207+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:34.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:34.207+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:34.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:34.208+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:34.842 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:34.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:34.842 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:34.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:34.842 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:34.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:34.842 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:34.487+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:35.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:34.836+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:35.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:34.836+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:35.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:34.836+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:35.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:34.836+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:35.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:35.229+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:35.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:35.229+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:35.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:35.229+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:35.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:35.229+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:35.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:35.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:35.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:35.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:35.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:35.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:35.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:35.451+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:36.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:35.870+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:36.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:35.870+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:36.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:35.870+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:36.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:35.870+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:36.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:36.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:36.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:36.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:36.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:36.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:36.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:36.426+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:36.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:36.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:36.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:36.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:36.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:36.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:36.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:36.216+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:37.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:36.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:37.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:36.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:37.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:36.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:37.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:36.875+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:37.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:37.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:37.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:37.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:37.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:37.411+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:37.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:37.412+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:37.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:37.247+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:37.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:37.247+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:37.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:37.247+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:37.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:37.248+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:38.226 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:03:38.130+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 51147 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:03:38.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:37.912+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:38.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:37.912+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:38.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:37.912+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:38.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:37.913+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:38.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:38.451+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:38.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:38.451+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:38.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:38.451+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:38.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:38.452+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:38.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:38.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:38.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:38.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:38.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:38.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:38.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:38.279+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:39.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:38.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:39.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:38.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:39.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:38.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:39.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:38.876+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:39.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:39.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:39.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:39.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:39.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:39.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:39.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:39.424+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:39.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:39.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:39.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:39.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:39.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:39.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:39.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:39.278+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:40.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:39.831+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:40.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:39.831+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:40.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:39.831+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:40.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:39.832+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:40.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:40.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:40.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:40.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:40.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:40.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:40.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:40.308+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:40.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:40.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:40.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:40.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:40.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:40.393+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:40.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:40.395+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:41.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:40.798+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:41.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:40.798+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:41.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:40.798+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:41.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:40.800+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:41.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:41.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:41.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:41.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:41.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:41.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:41.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:41.409+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:41.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:41.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:41.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:41.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:41.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:41.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:41.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:41.355+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:42.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:41.773+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:42.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:41.773+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:42.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:41.773+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:42.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:41.774+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:42.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:42.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:42.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:42.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:42.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:42.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:42.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:42.341+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:42.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:42.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:42.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:42.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:42.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:42.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:42.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:42.404+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:43.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:42.803+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:43.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:42.803+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:43.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:42.803+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:43.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:42.805+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:43.449 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:03:43.136+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 51193 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:03:43.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:43.342+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:43.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:43.342+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:43.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:43.342+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:43.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:43.342+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:43.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:43.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:43.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:43.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:43.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:43.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:43.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:43.444+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:44.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:43.758+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:44.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:43.758+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:44.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:43.758+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:44.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:43.759+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:44.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:44.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:44.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:44.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:44.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:44.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:44.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:44.390+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:44.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:44.410+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:44.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:44.410+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:44.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:44.410+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:44.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:44.411+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:45.092 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:44.745+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:45.092 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:44.745+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:45.092 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:44.745+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:45.092 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:44.746+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:45.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:45.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:45.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:45.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:45.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:45.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:45.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:45.400+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:45.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:45.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:45.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:45.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:45.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:45.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:45.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:45.393+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:45.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:45.709+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:45.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:45.709+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:45.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:45.709+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:45.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:45.710+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:46.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:46.382+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:46.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:46.382+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:46.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:46.382+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:46.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:46.383+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:46.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:46.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:46.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:46.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:46.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:46.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:46.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:46.358+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:47.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:46.723+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:47.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:46.723+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:47.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:46.723+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:47.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:46.724+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:47.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:47.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:47.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:47.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:47.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:47.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:47.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:47.408+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:47.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:47.346+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:47.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:47.346+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:47.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:47.346+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:47.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:47.346+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:48.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:47.734+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:48.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:47.734+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:48.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:47.734+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:48.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:47.734+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:48.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:03:48.143+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 51238 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:03:48.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:48.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:48.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:48.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:48.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:48.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:48.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:48.359+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:48.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:48.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:48.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:48.307+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:48.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:48.761+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:48.902 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:48.761+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:48.902 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:48.761+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:48.902 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:48.762+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:49.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:49.357+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:49.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:49.357+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:49.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:49.357+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:49.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:49.358+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:49.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:49.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:49.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:49.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:49.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:49.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:49.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:49.339+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:50.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:49.753+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:50.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:49.753+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:50.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:49.753+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:50.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:49.754+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:50.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:50.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:50.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:50.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:50.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:50.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:50.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:50.310+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:50.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:50.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:50.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:50.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:50.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:50.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:50.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:50.325+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:51.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:50.724+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:51.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:50.724+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:51.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:50.724+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:51.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:50.725+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:51.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:51.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:51.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:51.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:51.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:51.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:51.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:51.319+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:51.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:51.298+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:51.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:51.298+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:51.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:51.298+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:51.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:51.298+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:51.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:51.734+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:51.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:51.734+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:51.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:51.734+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:51.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:51.735+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:52.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:52.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:52.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:52.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:52.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:52.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:52.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:52.306+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:52.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:52.313+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:52.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:52.313+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:52.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:52.313+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:52.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:52.314+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:53.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:52.777+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:53.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:52.777+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:53.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:52.777+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:53.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:52.777+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:53.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:03:53.150+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 51284 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:03:53.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:53.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:53.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:53.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:53.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:53.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:53.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:53.286+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:53.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:53.257+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:53.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:53.257+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:53.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:53.257+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:53.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:53.259+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:54.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:53.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:54.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:53.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:54.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:53.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:54.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:53.782+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:54.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:54.256+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:54.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:54.256+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:54.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:54.256+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:54.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:54.256+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:54.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:54.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:54.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:54.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:54.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:54.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:54.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:54.223+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:55.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:54.771+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:55.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:54.771+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:55.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:54.771+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:55.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:54.772+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:55.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:55.265+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:55.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:55.265+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:55.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:55.265+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:55.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:55.266+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:55.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:55.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:55.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:55.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:55.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:55.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:55.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:55.214+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:56.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:55.725+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:56.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:55.725+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:56.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:55.725+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:56.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:55.726+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:56.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:56.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:56.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:56.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:56.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:56.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:56.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:56.271+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:56.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:56.185+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:56.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:56.185+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:56.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:56.185+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:56.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:56.185+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:56.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:56.685+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:56.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:56.685+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:56.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:56.685+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:56.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:56.687+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:57.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:57.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:57.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:57.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:57.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:57.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:57.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:57.310+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:57.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:57.164+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:57.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:57.164+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:57.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:57.164+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:57.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:57.164+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:57.661+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:57.661+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:57.661+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:57.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:57.662+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:58.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:03:58.156+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 51324 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:03:58.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:58.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:58.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:58.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:58.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:58.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:58.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:58.189+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:58.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:58.319+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:58.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:58.320+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:58.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:58.320+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:58.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:58.320+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:58.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:58.637+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:58.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:58.637+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:58.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:58.637+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:58.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:58.638+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:59.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:59.168+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:59.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:59.168+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:59.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:59.168+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:03:59.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:03:59.169+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:59.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:59.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:59.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:59.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:59.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:59.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:03:59.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:03:59.349+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:03:59.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:59.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:59.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:59.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:59.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:59.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:03:59.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:03:59.687+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:00.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:00.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:00.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:00.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:00.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:00.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:00.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:00.359+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:00.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:00.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:00.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:00.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:00.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:00.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:00.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:00.203+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:00.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:00.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:00.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:00.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:00.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:00.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:00.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:00.687+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:01.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:01.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:01.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:01.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:01.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:01.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:01.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:01.398+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:01.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:01.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:01.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:01.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:01.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:01.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:01.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:01.215+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:02.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:01.723+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:02.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:01.723+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:02.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:01.723+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:02.201 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:01.724+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:02.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:02.192+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:02.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:02.192+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:02.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:02.192+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:02.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:02.193+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:02.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:02.351+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:02.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:02.351+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:02.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:02.351+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:02.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:02.353+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:02.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:02.689+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:02.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:02.689+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:02.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:02.689+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:02.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:02.689+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:03.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:04:03.162+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 51370 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:04:03.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:03.321+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:03.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:03.321+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:03.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:03.321+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:03.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:03.321+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:03.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:03.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:03.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:03.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:03.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:03.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:03.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:03.190+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:03.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:03.644+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:03.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:03.644+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:03.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:03.644+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:03.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:03.646+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:04.632 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:04.311+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:04.633 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:04.311+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:04.633 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:04.311+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:04.633 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:04.312+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:04.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:04.223+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:04.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:04.223+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:04.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:04.223+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:04.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:04.223+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:04.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:04.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:04.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:04.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:04.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:04.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:04.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:04.626+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:05.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:05.219+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:05.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:05.219+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:05.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:05.219+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:05.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:05.220+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:05.675 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:05.343+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:05.675 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:05.343+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:05.675 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:05.343+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:05.675 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:05.344+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:05.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:05.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:05.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:05.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:05.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:05.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:05.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:05.668+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:06.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:06.382+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:06.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:06.382+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:06.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:06.382+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:06.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:06.383+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:06.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:06.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:06.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:06.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:06.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:06.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:06.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:06.269+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:06.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:06.705+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:06.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:06.705+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:06.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:06.705+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:06.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:06.706+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:07.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:07.356+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:07.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:07.356+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:07.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:07.356+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:07.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:07.357+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:07.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:07.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:07.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:07.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:07.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:07.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:07.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:07.312+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:08.175 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:07.721+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:08.175 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:07.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:08.176 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:07.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:08.176 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:07.723+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:08.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:04:08.167+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 51415 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:04:08.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:08.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:08.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:08.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:08.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:08.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:08.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:08.389+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:08.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:08.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:08.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:08.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:08.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:08.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:08.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:08.285+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:08.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:08.695+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:08.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:08.695+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:08.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:08.695+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:08.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:08.696+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:09.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:09.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:09.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:09.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:09.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:09.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:09.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:09.282+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:09.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:09.408+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:09.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:09.408+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:09.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:09.408+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:09.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:09.409+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:09.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:09.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:09.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:09.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:09.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:09.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:09.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:09.674+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:10.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:10.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:10.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:10.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:10.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:10.398+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:10.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:10.399+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:10.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:10.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:10.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:10.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:10.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:10.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:10.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:10.289+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:10.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:10.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:10.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:10.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:10.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:10.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:10.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:10.633+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:11.621 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:11.261+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:11.622 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:11.261+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:11.622 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:11.261+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:11.622 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:11.262+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:11.622 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:11.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:11.622 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:11.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:11.622 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:11.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:11.622 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:11.438+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:11.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:11.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:11.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:11.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:11.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:11.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:11.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:11.612+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:12.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:12.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:12.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:12.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:12.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:12.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:12.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:12.447+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:12.615 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:12.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:12.615 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:12.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:12.615 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:12.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:12.615 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:12.225+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:12.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:12.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:12.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:12.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:12.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:12.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:12.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:12.607+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:13.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:04:13.175+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 51461 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:04:13.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:13.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:13.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:13.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:13.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:13.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:13.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:13.422+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:13.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:13.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:13.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:13.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:13.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:13.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:13.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:13.209+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:13.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:13.605+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:13.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:13.605+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:13.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:13.605+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:13.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:13.606+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:14.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:14.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:14.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:14.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:14.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:14.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:14.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:14.209+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:14.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:14.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:14.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:14.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:14.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:14.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:14.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:14.457+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:14.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:14.650+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:14.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:14.650+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:14.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:14.650+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:14.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:14.650+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:15.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:15.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:15.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:15.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:15.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:15.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:15.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:15.182+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:15.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:15.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:15.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:15.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:15.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:15.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:15.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:15.470+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:15.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:15.663+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:15.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:15.663+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:15.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:15.663+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:15.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:15.664+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:16.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:16.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:16.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:16.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:16.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:16.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:16.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:16.443+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:16.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:16.224+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:16.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:16.224+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:16.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:16.224+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:16.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:16.224+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:16.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:16.663+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:16.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:16.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:16.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:16.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:16.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:16.664+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:17.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:17.183+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:17.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:17.183+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:17.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:17.183+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:17.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:17.183+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:17.708+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:17.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:17.708+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:17.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:17.708+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:17.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:17.709+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:17.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:17.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:17.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:17.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:17.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:17.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:17.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:17.467+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:18.474 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:04:18.181+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 51505 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:04:18.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:18.183+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:18.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:18.183+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:18.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:18.183+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:18.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:18.184+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:18.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:18.663+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:18.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:18.663+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:18.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:18.663+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:18.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:18.664+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:18.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:18.468+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:18.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:18.468+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:18.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:18.468+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:18.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:18.468+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:19.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:19.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:19.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:19.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:19.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:19.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:19.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:19.150+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:19.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:19.518+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:19.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:19.518+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:19.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:19.518+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:19.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:19.518+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:19.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:19.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:19.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:19.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:19.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:19.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:19.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:19.668+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:20.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:20.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:20.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:20.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:20.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:20.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:20.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:20.125+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:20.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:20.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:20.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:20.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:20.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:20.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:20.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:20.686+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:20.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:20.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:20.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:20.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:20.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:20.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:20.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:20.500+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:21.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:21.093+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:21.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:21.093+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:21.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:21.093+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:21.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:21.094+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:21.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:21.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:21.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:21.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:21.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:21.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:21.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:21.541+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:21.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:21.649+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:21.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:21.649+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:21.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:21.649+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:21.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:21.649+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:22.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:22.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:22.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:22.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:22.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:22.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:22.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:22.105+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:22.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:22.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:22.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:22.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:22.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:22.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:22.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:22.618+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:22.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:22.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:22.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:22.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:22.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:22.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:22.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:22.513+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:23.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:04:23.187+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 51551 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:04:23.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:23.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:23.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:23.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:23.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:23.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:23.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:23.085+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:23.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:23.613+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:23.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:23.613+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:23.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:23.613+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:23.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:23.615+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:23.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:23.517+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:23.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:23.517+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:23.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:23.517+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:23.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:23.518+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:24.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:24.119+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:24.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:24.119+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:24.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:24.119+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:24.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:24.120+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:24.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:24.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:24.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:24.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:24.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:24.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:24.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:24.623+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:24.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:24.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:24.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:24.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:24.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:24.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:24.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:24.523+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:25.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:25.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:25.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:25.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:25.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:25.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:25.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:25.140+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:25.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:25.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:25.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:25.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:25.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:25.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:25.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:25.545+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:25.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:25.621+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:25.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:25.621+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:25.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:25.621+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:25.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:25.623+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:26.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:26.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:26.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:26.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:26.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:26.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:26.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:26.189+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:26.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:26.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:26.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:26.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:26.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:26.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:26.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:26.590+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:26.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:26.630+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:26.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:26.630+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:26.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:26.630+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:26.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:26.631+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:27.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:27.154+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:27.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:27.154+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:27.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:27.154+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:27.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:27.155+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:27.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:27.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:27.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:27.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:27.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:27.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:27.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:27.605+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:27.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:27.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:27.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:27.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:27.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:27.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:27.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:27.629+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:28.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:04:28.194+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 51596 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:04:28.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:28.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:28.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:28.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:28.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:28.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:28.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:28.113+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:28.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:28.670+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:28.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:28.670+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:28.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:28.670+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:28.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:28.671+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:28.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:28.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:28.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:28.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:28.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:28.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:28.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:28.645+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:29.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:29.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:29.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:29.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:29.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:29.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:29.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:29.118+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:29.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:29.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:29.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:29.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:29.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:29.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:29.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:29.698+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:29.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:29.612+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:29.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:29.612+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:29.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:29.612+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:29.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:29.613+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:30.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:30.093+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:30.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:30.093+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:30.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:30.093+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:30.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:30.093+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:30.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:30.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:30.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:30.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:30.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:30.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:30.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:30.698+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:30.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:30.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:30.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:30.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:30.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:30.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:30.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:30.587+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:31.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:31.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:31.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:31.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:31.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:31.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:31.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:31.132+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:31.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:31.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:31.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:31.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:31.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:31.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:31.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:31.697+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:31.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:31.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:31.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:31.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:31.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:31.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:31.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:31.605+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:32.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:32.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:32.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:32.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:32.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:32.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:32.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:32.170+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:32.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:32.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:32.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:32.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:32.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:32.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:32.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:32.677+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:32.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:32.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:32.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:32.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:32.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:32.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:32.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:32.581+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:33.477 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:04:33.200+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 51642 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:04:33.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:33.153+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:33.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:33.153+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:33.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:33.153+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:33.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:33.153+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:33.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:33.671+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:33.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:33.671+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:33.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:33.671+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:33.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:33.671+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:33.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:33.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:33.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:33.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:33.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:33.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:33.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:33.580+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:34.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:34.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:34.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:34.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:34.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:34.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:34.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:34.150+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:34.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:34.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:34.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:34.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:34.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:34.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:34.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:34.644+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:34.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:34.588+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:34.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:34.588+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:34.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:34.588+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:34.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:34.589+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:35.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:35.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:35.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:35.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:35.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:35.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:35.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:35.146+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:35.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:35.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:35.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:35.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:35.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:35.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:35.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:35.648+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:35.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:35.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:35.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:35.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:35.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:35.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:35.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:35.565+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:36.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:36.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:36.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:36.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:36.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:36.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:36.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:36.175+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:36.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:36.563+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:36.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:36.563+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:36.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:36.563+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:36.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:36.564+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:36.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:36.680+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:36.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:36.680+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:36.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:36.680+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:36.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:36.681+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:37.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:37.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:37.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:37.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:37.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:37.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:37.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:37.182+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:37.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:37.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:37.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:37.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:37.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:37.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:37.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:37.665+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:37.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:37.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:37.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:37.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:37.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:37.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:37.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:37.586+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:38.477 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:04:38.207+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 51686 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:04:38.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:38.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:38.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:38.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:38.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:38.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:38.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:38.199+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:38.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:38.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:38.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:38.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:38.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:38.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:38.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:38.576+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:38.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:38.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:38.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:38.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:38.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:38.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:38.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:38.660+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:39.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:39.247+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:39.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:39.247+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:39.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:39.247+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:39.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:39.247+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:39.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:39.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:39.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:39.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:39.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:39.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:39.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:39.611+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:39.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:39.656+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:39.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:39.656+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:39.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:39.656+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:39.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:39.657+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:40.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:40.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:40.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:40.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:40.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:40.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:40.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:40.285+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:40.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:40.679+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:40.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:40.679+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:40.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:40.679+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:40.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:40.681+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:40.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:40.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:40.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:40.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:40.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:40.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:40.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:40.619+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:41.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:41.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:41.592 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:41.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:41.592 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:41.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:41.592 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:41.269+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:41.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:41.660+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:41.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:41.660+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:41.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:41.660+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:41.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:41.661+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:41.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:41.584+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:41.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:41.584+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:41.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:41.584+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:41.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:41.585+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:42.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:42.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:42.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:42.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:42.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:42.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:42.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:42.281+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:42.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:42.702+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:42.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:42.702+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:42.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:42.702+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:42.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:42.703+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:42.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:42.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:42.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:42.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:42.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:42.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:42.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:42.630+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:43.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:04:43.213+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 51732 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:04:43.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:43.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:43.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:43.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:43.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:43.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:43.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:43.277+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:43.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:43.742+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:43.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:43.742+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:43.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:43.742+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:43.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:43.743+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:43.908 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:43.650+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:43.909 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:43.650+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:43.909 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:43.650+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:43.909 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:43.651+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:44.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:44.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:44.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:44.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:44.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:44.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:44.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:44.298+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:44.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:44.716+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:44.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:44.716+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:44.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:44.716+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:44.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:44.717+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:44.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:44.670+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:44.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:44.670+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:44.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:44.670+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:44.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:44.672+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:45.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:45.316+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:45.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:45.316+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:45.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:45.316+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:45.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:45.317+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:45.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:45.747+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:45.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:45.747+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:45.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:45.747+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:45.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:45.748+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:45.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:45.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:45.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:45.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:45.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:45.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:45.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:45.643+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:46.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:46.293+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:46.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:46.293+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:46.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:46.293+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:46.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:46.294+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:46.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:46.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:46.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:46.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:46.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:46.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:46.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:46.692+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:46.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:46.699+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:46.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:46.699+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:46.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:46.699+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:46.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:46.700+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:47.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:47.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:47.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:47.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:47.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:47.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:47.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:47.270+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:47.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:47.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:47.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:47.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:47.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:47.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:47.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:47.689+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:47.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:47.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:47.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:47.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:47.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:47.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:47.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:47.684+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:48.477 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:04:48.219+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 51777 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:04:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:48.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:48.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:48.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:48.302+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:48.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:48.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:48.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:48.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:48.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:48.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:48.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:48.700+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:48.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:48.699+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:48.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:48.699+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:48.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:48.699+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:48.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:48.700+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:49.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:49.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:49.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:49.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:49.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:49.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:49.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:49.351+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:49.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:49.652+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:49.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:49.652+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:49.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:49.652+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:49.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:49.654+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:49.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:49.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:49.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:49.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:49.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:49.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:49.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:49.729+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:50.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:50.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:50.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:50.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:50.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:50.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:50.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:50.387+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:50.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:50.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:50.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:50.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:50.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:50.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:50.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:50.729+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:50.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:50.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:50.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:50.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:50.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:50.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:50.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:50.683+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:51.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:51.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:51.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:51.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:51.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:51.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:51.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:51.431+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:51.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:51.725+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:51.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:51.725+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:51.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:51.725+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:51.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:51.726+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:51.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:51.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:51.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:51.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:51.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:51.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:51.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:51.693+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:52.689 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:52.427+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:52.689 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:52.427+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:52.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:52.427+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:52.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:52.427+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:52.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:52.656+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:52.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:52.656+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:52.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:52.656+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:52.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:52.657+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:52.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:52.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:52.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:52.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:52.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:52.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:52.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:52.684+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:53.469 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:04:53.226+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 51824 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:04:53.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:53.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:53.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:53.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:53.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:53.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:53.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:53.679+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:53.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:53.645+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:53.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:53.645+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:53.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:53.645+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:53.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:53.646+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:53.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:53.462+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:53.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:53.462+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:53.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:53.462+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:53.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:53.462+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:54.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:54.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:54.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:54.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:54.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:54.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:54.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:54.614+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:54.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:54.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:54.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:54.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:54.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:54.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:54.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:54.650+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:54.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:54.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:54.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:54.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:54.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:54.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:54.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:54.495+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:55.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:55.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:55.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:55.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:55.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:55.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:55.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:55.660+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:55.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:55.602+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:55.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:55.602+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:55.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:55.602+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:55.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:55.604+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:55.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:55.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:55.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:55.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:55.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:55.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:55.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:55.504+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:56.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:56.556+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:56.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:56.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:56.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:56.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:56.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:56.557+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:56.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:56.665+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:56.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:56.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:56.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:56.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:56.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:56.666+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:56.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:56.471+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:56.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:56.471+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:56.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:56.471+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:56.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:56.471+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:57.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:57.602+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:57.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:57.602+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:57.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:57.602+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:57.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:57.603+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:57.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:57.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:57.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:57.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:57.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:57.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:57.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:57.677+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:57.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:57.450+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:57.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:57.450+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:57.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:57.450+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:57.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:57.450+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:58.645 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:04:58.232+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 51868 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:04:58.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:58.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:58.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:58.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:58.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:58.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:58.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:58.478+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:58.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:58.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:58.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:58.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:58.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:58.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:04:58.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:58.708+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:58.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:58.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:58.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:58.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:58.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:58.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:58.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:58.640+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:59.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:59.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:59.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:59.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:59.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:59.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:04:59.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:04:59.620+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:04:59.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:59.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:59.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:59.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:59.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:59.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:04:59.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:04:59.459+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:00.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:59.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:00.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:59.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:00.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:59.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:00.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:04:59.745+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:00.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:00.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:00.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:00.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:00.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:00.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:00.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:00.599+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:00.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:00.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:00.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:00.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:00.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:00.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:00.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:00.718+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:00.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:00.503+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:00.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:00.503+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:00.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:00.503+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:00.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:00.504+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:01.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:01.490+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:01.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:01.490+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:01.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:01.490+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:01.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:01.490+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:01.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:01.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:01.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:01.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:01.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:01.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:01.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:01.584+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:01.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:01.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:01.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:01.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:01.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:01.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:01.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:01.737+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:02.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:02.566+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:02.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:02.566+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:02.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:02.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:02.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:02.567+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:02.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:02.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:02.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:02.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:02.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:02.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:02.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:02.489+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:03.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:02.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:03.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:02.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:03.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:02.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:03.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:02.765+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:03.508 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:05:03.239+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 51914 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:05:03.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:03.718+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:03.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:03.718+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:03.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:03.718+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:03.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:03.719+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:03.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:03.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:03.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:03.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:03.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:03.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:03.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:03.500+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:03.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:03.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:03.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:03.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:03.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:03.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:03.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:03.527+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:04.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:04.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:04.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:04.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:04.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:04.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:04.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:04.479+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:04.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:04.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:04.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:04.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:04.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:04.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:04.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:04.686+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:04.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:04.543+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:04.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:04.543+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:04.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:04.543+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:04.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:04.544+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:05.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:05.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:05.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:05.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:05.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:05.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:05.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:05.458+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:05.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:05.534+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:05.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:05.534+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:05.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:05.534+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:05.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:05.534+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:05.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:05.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:05.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:05.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:05.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:05.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:05.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:05.692+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:06.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:06.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:06.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:06.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:06.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:06.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:06.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:06.463+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:06.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:06.517+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:06.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:06.517+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:06.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:06.517+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:06.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:06.517+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:06.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:06.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:06.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:06.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:06.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:06.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:06.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:06.687+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:07.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:07.506+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:07.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:07.506+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:07.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:07.506+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:07.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:07.507+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:07.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:07.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:07.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:07.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:07.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:07.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:07.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:07.425+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:07.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:07.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:07.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:07.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:07.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:07.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:07.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:07.669+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:08.530 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:05:08.245+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 51959 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:05:08.531 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:08.391+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:08.531 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:08.392+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:08.531 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:08.392+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:08.531 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:08.392+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:08.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:08.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:08.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:08.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:08.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:08.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:08.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:08.700+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:08.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:08.522+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:08.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:08.522+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:08.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:08.522+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:08.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:08.523+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:09.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:09.412+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:09.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:09.412+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:09.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:09.412+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:09.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:09.414+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:09.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:09.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:09.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:09.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:09.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:09.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:09.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:09.518+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:09.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:09.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:09.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:09.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:09.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:09.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:09.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:09.654+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:10.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:10.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:10.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:10.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:10.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:10.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:10.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:10.383+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:10.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:10.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:10.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:10.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:10.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:10.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:10.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:10.530+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:10.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:10.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:10.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:10.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:10.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:10.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:10.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:10.616+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:11.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:11.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:11.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:11.414+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:11.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:11.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:11.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:11.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:11.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:11.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:11.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:11.616+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:11.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:11.576+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:11.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:11.576+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:11.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:11.576+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:11.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:11.577+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:12.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:12.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:12.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:12.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:12.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:12.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:12.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:12.398+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:12.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:12.602+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:12.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:12.602+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:12.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:12.602+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:12.672 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:12.603+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:12.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:12.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:12.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:12.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:12.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:12.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:12.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:12.664+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:13.623 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:05:13.253+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 52005 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:05:13.624 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:13.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:13.624 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:13.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:13.624 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:13.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:13.624 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:13.436+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:13.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:13.616+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:13.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:13.616+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:13.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:13.616+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:13.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:13.617+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:13.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:13.687+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:13.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:13.687+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:13.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:13.687+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:13.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:13.687+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:14.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:14.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:14.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:14.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:14.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:14.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:14.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:14.420+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:14.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:14.644+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:14.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:14.644+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:14.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:14.644+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:14.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:14.645+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:15.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:14.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:15.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:14.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:15.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:14.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:15.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:14.729+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:15.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:15.668+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:15.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:15.668+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:15.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:15.668+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:15.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:15.669+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:15.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:15.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:15.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:15.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:15.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:15.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:15.465+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:16.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:15.730+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:16.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:15.730+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:16.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:15.730+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:16.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:15.731+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:16.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:16.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:16.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:16.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:16.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:16.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:16.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:16.444+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:16.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:16.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:16.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:16.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:16.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:16.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:16.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:16.770+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:16.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:16.686+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:16.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:16.686+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:16.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:16.686+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:16.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:16.687+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:17.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:17.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:17.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:17.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:17.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:17.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:17.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:17.469+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:18.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:17.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:18.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:17.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:18.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:17.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:18.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:17.818+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:18.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:17.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:18.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:17.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:18.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:17.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:18.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:17.724+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:18.727 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:05:18.260+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 52049 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:05:18.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:18.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:18.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:18.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:18.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:18.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:18.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:18.432+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:19.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:18.861+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:19.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:18.861+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:19.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:18.861+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:19.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:18.861+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:19.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:18.720+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:19.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:18.720+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:19.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:18.720+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:19.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:18.721+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:19.855 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:19.715+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:19.855 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:19.715+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:19.855 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:19.715+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:19.856 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:19.715+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:19.856 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:19.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:19.856 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:19.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:19.856 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:19.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:19.856 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:19.472+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:20.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:19.848+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:20.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:19.848+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:20.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:19.848+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:20.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:19.849+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:20.772 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:20.684+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:20.772 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:20.684+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:20.772 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:20.684+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:20.772 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:20.685+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:20.773 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:20.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:20.773 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:20.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:20.773 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:20.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:20.773 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:20.484+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:21.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:20.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:21.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:20.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:21.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:20.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:21.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:20.899+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:21.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:21.659+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:21.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:21.659+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:21.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:21.659+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:21.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:21.660+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:21.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:21.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:21.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:21.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:21.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:21.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:21.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:21.456+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:22.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:21.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:22.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:21.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:22.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:21.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:22.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:21.930+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:22.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:22.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:22.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:22.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:22.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:22.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:22.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:22.478+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:22.910 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:22.614+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:22.910 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:22.614+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:22.910 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:22.614+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:22.910 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:22.614+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:23.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:22.902+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:23.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:22.902+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:23.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:22.902+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:23.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:22.902+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:23.532 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:05:23.266+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 52095 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:05:23.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:23.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:23.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:23.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:23.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:23.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:23.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:23.526+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:23.845 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:23.601+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:23.845 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:23.601+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:23.845 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:23.601+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:23.845 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:23.602+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:24.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:23.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:24.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:23.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:24.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:23.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:24.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:23.887+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:24.751 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:24.607+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:24.751 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:24.607+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:24.751 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:24.607+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:24.751 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:24.607+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:24.751 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:24.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:24.752 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:24.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:24.752 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:24.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:24.752 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:24.491+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:25.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:24.890+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:25.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:24.890+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:25.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:24.890+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:25.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:24.890+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:25.894 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:25.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:25.894 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:25.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:25.894 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:25.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:25.894 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:25.537+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:25.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:25.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:25.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:25.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:25.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:25.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:25.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:25.580+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:26.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:25.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:26.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:25.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:26.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:25.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:26.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:25.887+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:26.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:26.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:26.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:26.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:26.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:26.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:26.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:26.544+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:26.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:26.533+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:26.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:26.533+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:26.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:26.533+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:26.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:26.534+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:27.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:26.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:27.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:26.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:27.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:26.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:27.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:26.892+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:27.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:27.581+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:27.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:27.581+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:27.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:27.581+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:27.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:27.582+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:27.939 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:27.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:27.939 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:27.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:27.939 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:27.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:27.939 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:27.517+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:28.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:27.931+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:28.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:27.931+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:28.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:27.931+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:28.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:27.932+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:28.556 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:05:28.272+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 52140 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:05:28.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:28.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:28.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:28.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:28.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:28.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:28.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:28.550+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:28.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:28.628+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:28.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:28.628+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:28.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:28.628+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:28.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:28.629+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:29.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:28.971+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:29.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:28.971+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:29.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:28.971+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:29.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:28.972+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:29.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:29.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:29.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:29.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:29.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:29.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:29.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:29.547+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:29.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:29.646+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:29.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:29.646+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:29.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:29.646+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:29.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:29.647+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:30.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:29.988+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:30.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:29.988+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:30.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:29.988+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:30.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:29.988+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:30.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:30.560+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:30.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:30.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:30.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:30.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:30.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:30.562+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:30.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:30.632+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:30.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:30.632+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:30.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:30.632+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:30.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:30.633+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:31.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:30.955+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:31.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:30.955+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:31.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:30.955+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:31.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:30.956+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:31.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:31.613+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:31.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:31.613+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:31.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:31.613+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:31.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:31.614+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:31.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:31.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:31.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:31.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:31.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:31.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:31.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:31.522+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:32.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:31.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:32.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:31.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:32.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:31.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:32.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:31.926+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:32.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:32.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:32.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:32.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:32.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:32.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:32.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:32.501+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:32.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:32.633+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:32.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:32.633+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:32.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:32.633+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:32.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:32.634+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:33.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:32.879+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:33.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:32.879+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:33.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:32.879+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:33.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:32.880+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:33.552 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:05:33.279+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 52186 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:05:33.867 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:33.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:33.867 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:33.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:33.867 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:33.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:33.867 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:33.546+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:33.867 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:33.638+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:33.867 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:33.638+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:33.868 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:33.638+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:33.868 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:33.639+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:34.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:33.859+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:34.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:33.859+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:34.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:33.859+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:34.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:33.861+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:34.914 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:34.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:34.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:34.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:34.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:34.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:34.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:34.596+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:34.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:34.611+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:34.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:34.611+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:34.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:34.611+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:34.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:34.612+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:35.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:34.907+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:35.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:34.907+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:35.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:34.907+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:35.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:34.908+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:35.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:35.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:35.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:35.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:35.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:35.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:35.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:35.618+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:35.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:35.652+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:35.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:35.652+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:35.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:35.652+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:35.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:35.652+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:36.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:35.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:36.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:35.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:36.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:35.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:36.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:35.938+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:36.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:36.600+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:36.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:36.600+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:36.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:36.600+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:36.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:36.601+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:36.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:36.687+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:36.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:36.687+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:36.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:36.687+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:36.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:36.687+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:37.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:36.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:37.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:36.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:37.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:36.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:37.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:36.980+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:37.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:37.626+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:37.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:37.626+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:37.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:37.626+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:37.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:37.626+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:37.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:37.659+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:37.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:37.659+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:37.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:37.659+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:37.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:37.659+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:38.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:38.003+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:38.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:38.003+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:38.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:38.003+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:38.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:38.004+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:38.592 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:05:38.285+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 52230 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:05:38.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:38.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:38.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:38.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:38.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:38.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:38.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:38.586+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:38.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:38.620+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:38.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:38.620+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:38.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:38.620+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:38.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:38.620+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:39.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:38.969+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:39.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:38.969+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:39.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:38.969+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:39.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:38.970+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:39.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:39.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:39.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:39.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:39.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:39.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:39.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:39.620+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:39.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:39.640+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:39.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:39.640+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:39.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:39.640+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:39.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:39.640+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:40.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:39.973+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:40.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:39.973+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:40.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:39.973+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:40.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:39.974+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:40.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:40.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:40.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:40.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:40.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:40.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:40.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:40.668+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:40.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:40.612+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:40.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:40.612+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:40.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:40.612+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:40.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:40.612+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:41.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:40.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:41.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:40.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:41.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:40.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:41.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:40.978+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:41.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:41.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:41.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:41.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:41.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:41.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:41.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:41.629+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:41.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:41.654+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:41.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:41.654+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:41.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:41.654+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:41.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:41.655+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:42.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:41.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:42.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:41.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:42.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:41.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:42.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:41.990+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:42.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:42.625+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:42.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:42.625+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:42.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:42.625+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:42.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:42.625+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:42.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:42.653+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:42.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:42.653+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:42.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:42.653+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:42.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:42.654+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:43.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:42.983+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:43.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:42.983+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:43.300 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:42.983+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:43.300 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:42.984+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:43.613 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:05:43.292+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 52276 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:05:43.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:43.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:43.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:43.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:43.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:43.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:43.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:43.643+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:43.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:43.605+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:43.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:43.605+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:43.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:43.605+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:43.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:43.606+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:44.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:43.943+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:44.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:43.943+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:44.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:43.943+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:44.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:43.944+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:44.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:44.637+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:44.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:44.637+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:44.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:44.637+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:44.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:44.638+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:44.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:44.619+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:44.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:44.619+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:44.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:44.619+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:44.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:44.619+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:45.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:44.988+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:45.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:44.988+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:45.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:44.988+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:45.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:44.989+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:45.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:45.652+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:45.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:45.652+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:45.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:45.652+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:45.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:45.653+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:45.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:45.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:45.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:45.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:45.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:45.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:45.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:45.641+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:46.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:45.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:46.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:45.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:46.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:45.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:46.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:45.991+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:46.689 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:46.606+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:46.689 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:46.606+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:46.689 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:46.606+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:46.689 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:46.606+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:46.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:46.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:46.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:46.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:46.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:46.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:46.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:46.682+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:47.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:46.960+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:47.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:46.960+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:47.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:46.960+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:47.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:46.961+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:47.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:47.605+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:47.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:47.605+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:47.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:47.605+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:47.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:47.606+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:47.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:47.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:47.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:47.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:47.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:47.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:47.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:47.660+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:48.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:47.963+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:48.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:47.963+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:48.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:47.963+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:48.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:47.964+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:48.605 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:05:48.299+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 52321 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:05:48.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:48.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:48.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:48.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:48.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:48.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:48.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:48.698+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:48.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:48.598+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:48.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:48.598+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:48.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:48.598+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:48.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:48.598+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:49.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:48.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:49.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:48.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:49.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:48.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:49.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:48.974+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:49.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:49.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:49.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:49.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:49.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:49.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:49.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:49.550+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:49.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:49.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:49.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:49.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:49.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:49.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:49.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:49.684+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:50.212 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:49.936+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:50.212 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:49.936+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:50.212 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:49.936+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:50.212 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:49.938+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:50.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:50.710+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:50.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:50.710+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:50.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:50.710+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:50.941 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:50.712+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:50.941 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:50.573+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:50.941 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:50.573+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:50.941 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:50.573+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:50.941 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:50.574+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:51.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:50.933+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:51.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:50.933+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:51.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:50.933+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:51.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:50.934+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:51.761 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:51.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:51.761 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:51.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:51.761 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:51.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:51.761 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:51.576+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:52.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:51.753+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:52.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:51.753+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:52.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:51.753+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:52.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:51.755+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:52.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:51.905+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:52.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:51.905+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:52.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:51.905+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:52.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:51.906+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:52.874 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:52.783+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:52.874 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:52.783+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:52.874 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:52.783+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:52.874 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:52.785+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:52.875 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:52.564+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:52.875 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:52.564+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:52.875 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:52.564+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:52.875 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:52.564+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:53.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:52.866+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:53.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:52.866+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:53.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:52.866+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:53.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:52.868+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:53.727 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:05:53.306+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 52367 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:05:53.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:53.536+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:53.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:53.536+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:53.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:53.536+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:53.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:53.536+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:54.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:53.815+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:54.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:53.815+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:54.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:53.815+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:54.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:53.816+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:54.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:53.902+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:54.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:53.902+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:54.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:53.902+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:54.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:53.903+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:54.867 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:54.495+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:54.867 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:54.495+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:54.867 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:54.495+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:54.867 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:54.495+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:55.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:54.859+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:55.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:54.859+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:55.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:54.859+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:55.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:54.861+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:55.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:54.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:55.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:54.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:55.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:54.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:55.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:54.941+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:55.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:55.454+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:55.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:55.454+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:55.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:55.454+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:55.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:55.454+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:56.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:55.886+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:56.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:55.886+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:56.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:55.886+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:56.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:55.887+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:56.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:55.955+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:56.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:55.955+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:56.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:55.955+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:56.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:55.955+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:56.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:56.468+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:56.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:56.468+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:56.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:56.468+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:56.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:56.469+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:57.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:56.891+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:57.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:56.891+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:57.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:56.891+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:57.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:56.892+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:57.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:56.955+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:57.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:56.955+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:57.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:56.955+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:57.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:56.956+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:57.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:57.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:57.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:57.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:57.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:57.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:57.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:57.444+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:58.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:57.896+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:58.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:57.896+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:58.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:57.896+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:58.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:57.897+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:58.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:57.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:58.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:57.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:58.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:57.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:58.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:57.926+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:58.727 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:05:58.312+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 52413 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:05:58.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:58.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:58.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:58.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:58.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:58.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:58.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:58.403+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:59.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:58.897+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:59.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:58.897+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:59.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:58.897+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:05:59.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:58.898+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:59.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:58.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:59.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:58.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:59.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:58.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:05:59.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:58.883+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:05:59.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:59.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:59.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:59.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:59.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:59.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:05:59.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:05:59.387+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:00.212 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:59.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:00.212 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:59.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:00.212 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:59.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:00.212 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:05:59.921+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:00.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:59.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:00.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:59.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:00.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:59.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:00.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:05:59.854+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:00.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:00.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:00.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:00.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:00.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:00.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:00.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:00.387+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:00.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:00.880+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:00.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:00.880+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:00.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:00.880+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:00.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:00.881+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:00.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:00.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:00.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:00.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:00.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:00.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:00.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:00.819+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:01.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:01.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:01.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:01.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:01.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:01.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:01.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:01.424+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:02.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:01.916+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:02.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:01.916+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:02.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:01.916+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:02.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:01.917+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:02.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:01.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:02.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:01.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:02.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:01.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:02.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:01.850+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:02.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:02.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:02.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:02.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:02.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:02.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:02.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:02.411+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:03.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:02.907+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:03.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:02.907+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:03.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:02.907+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:03.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:02.908+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:03.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:02.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:03.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:02.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:03.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:02.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:03.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:02.825+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:03.727 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:06:03.318+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 52459 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:06:03.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:03.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:03.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:03.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:03.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:03.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:03.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:03.411+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:04.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:03.858+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:04.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:03.858+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:04.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:03.858+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:04.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:03.859+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:04.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:03.846+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:04.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:03.846+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:04.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:03.846+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:04.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:03.847+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:04.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:04.377+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:04.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:04.377+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:04.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:04.377+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:04.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:04.377+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:05.106 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:04.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:05.106 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:04.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:05.106 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:04.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:05.106 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:04.819+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:05.106 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:04.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:05.106 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:04.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:05.106 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:04.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:05.106 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:04.830+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:05.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:05.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:05.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:05.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:05.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:05.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:05.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:05.335+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:06.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:05.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:06.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:05.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:06.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:05.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:06.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:05.777+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:06.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:05.870+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:06.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:05.870+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:06.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:05.870+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:06.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:05.871+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:06.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:06.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:06.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:06.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:06.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:06.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:06.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:06.348+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:07.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:06.768+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:07.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:06.768+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:07.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:06.768+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:07.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:06.769+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:07.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:06.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:07.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:06.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:07.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:06.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:07.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:06.903+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:07.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:07.303+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:07.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:07.303+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:07.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:07.303+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:07.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:07.304+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:08.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:07.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:08.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:07.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:08.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:07.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:08.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:07.792+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:08.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:07.878+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:08.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:07.878+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:08.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:07.878+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:08.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:07.879+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:08.727 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:06:08.324+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 52504 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:06:08.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:08.288+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:08.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:08.288+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:08.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:08.288+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:08.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:08.288+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:09.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:08.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:09.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:08.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:09.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:08.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:09.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:08.921+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:09.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:08.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:09.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:08.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:09.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:08.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:09.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:08.792+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:09.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:09.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:09.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:09.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:09.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:09.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:09.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:09.337+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:10.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:09.917+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:10.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:09.917+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:10.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:09.917+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:10.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:09.918+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:10.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:09.784+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:10.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:09.784+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:10.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:09.784+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:10.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:09.785+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:10.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:10.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:10.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:10.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:10.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:10.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:10.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:10.383+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:11.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:10.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:11.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:10.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:11.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:10.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:11.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:10.887+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:11.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:10.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:11.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:10.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:11.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:10.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:11.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:10.813+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:11.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:11.429+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:11.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:11.429+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:11.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:11.429+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:11.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:11.429+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:12.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:11.936+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:12.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:11.936+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:12.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:11.936+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:12.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:11.937+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:12.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:11.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:12.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:11.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:12.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:11.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:12.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:11.767+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:12.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:12.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:12.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:12.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:12.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:12.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:12.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:12.445+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:12.997 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:12.957+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:12.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:12.957+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:12.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:12.957+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:12.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:12.957+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:12.998 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:12.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:12.998 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:12.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:12.998 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:12.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:12.998 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:12.720+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:13.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:06:13.331+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 52550 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:06:13.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:13.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:13.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:13.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:13.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:13.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:13.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:13.419+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:14.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:13.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:14.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:13.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:14.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:13.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:14.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:13.997+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:14.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:13.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:14.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:13.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:14.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:13.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:14.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:13.765+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:14.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:14.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:14.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:14.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:14.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:14.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:14.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:14.445+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:15.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:15.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:15.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:15.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:15.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:15.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:15.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:15.013+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:15.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:14.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:15.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:14.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:15.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:14.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:15.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:14.767+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:15.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:15.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:15.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:15.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:15.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:15.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:15.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:15.459+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:15.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:15.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:15.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:15.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:15.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:15.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:15.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:15.718+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:16.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:15.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:16.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:15.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:16.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:15.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:16.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:15.991+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:16.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:16.484+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:16.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:16.484+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:16.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:16.484+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:16.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:16.485+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:17.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:16.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:17.034 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:16.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:17.034 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:16.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:17.034 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:16.738+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:17.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:17.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:17.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:17.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:17.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:17.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:17.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:17.027+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:17.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:17.490+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:17.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:17.490+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:17.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:17.490+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:17.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:17.490+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:17.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:17.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:17.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:17.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:17.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:17.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:17.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:17.720+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:18.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:18.043+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:18.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:18.043+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:18.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:18.043+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:18.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:18.043+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:18.701 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:06:18.338+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 52594 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:06:18.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:18.517+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:18.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:18.517+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:18.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:18.517+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:18.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:18.517+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:18.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:18.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:18.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:18.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:18.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:18.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:18.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:18.694+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:19.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:19.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:19.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:19.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:19.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:19.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:19.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:19.091+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:19.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:19.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:19.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:19.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:19.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:19.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:19.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:19.711+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:19.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:19.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:19.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:19.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:19.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:19.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:19.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:19.499+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:20.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:20.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:20.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:20.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:20.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:20.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:20.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:20.089+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:20.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:20.450+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:20.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:20.450+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:20.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:20.450+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:20.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:20.451+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:20.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:20.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:20.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:20.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:20.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:20.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:20.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:20.681+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:21.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:21.100+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:21.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:21.100+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:21.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:21.100+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:21.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:21.100+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:21.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:21.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:21.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:21.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:21.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:21.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:21.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:21.436+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:21.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:21.709+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:21.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:21.709+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:21.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:21.709+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:21.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:21.710+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:22.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:22.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:22.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:22.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:22.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:22.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:22.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:22.129+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:22.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:22.681+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:22.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:22.681+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:22.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:22.681+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:22.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:22.682+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:22.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:22.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:22.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:22.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:22.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:22.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:22.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:22.451+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:23.427 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:06:23.345+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 52640 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:06:23.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:23.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:23.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:23.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:23.428 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:23.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:23.428 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:23.117+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:23.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:23.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:23.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:23.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:23.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:23.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:23.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:23.419+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:23.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:23.687+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:23.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:23.687+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:23.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:23.687+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:23.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:23.688+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:24.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:24.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:24.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:24.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:24.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:24.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:24.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:24.142+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:24.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:24.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:24.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:24.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:24.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:24.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:24.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:24.386+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:25.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:24.727+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:25.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:24.727+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:25.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:24.727+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:25.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:24.728+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:25.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:25.104+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:25.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:25.104+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:25.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:25.104+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:25.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:25.105+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:25.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:25.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:25.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:25.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:25.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:25.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:25.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:25.376+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:26.077 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:25.727+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:26.077 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:25.727+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:26.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:25.727+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:26.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:25.728+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:26.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:26.069+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:26.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:26.070+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:26.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:26.070+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:26.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:26.070+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:26.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:26.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:26.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:26.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:26.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:26.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:26.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:26.371+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:27.097 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:26.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:27.097 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:26.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:27.097 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:26.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:27.097 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:26.753+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:27.376 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:27.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:27.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:27.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:27.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:27.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:27.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:27.090+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:27.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:27.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:27.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:27.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:27.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:27.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:27.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:27.369+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:28.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:27.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:28.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:27.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:28.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:27.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:28.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:27.720+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:28.381 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:06:28.350+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 52685 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:06:28.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:28.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:28.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:28.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:28.382 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:28.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:28.382 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:28.124+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:28.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:28.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:28.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:28.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:28.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:28.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:28.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:28.408+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:28.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:28.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:28.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:28.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:28.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:28.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:28.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:28.707+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:29.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:29.079+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:29.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:29.079+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:29.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:29.079+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:29.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:29.080+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:29.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:29.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:29.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:29.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:29.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:29.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:29.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:29.395+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:29.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:29.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:29.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:29.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:29.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:29.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:29.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:29.667+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:30.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:30.058+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:30.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:30.058+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:30.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:30.058+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:30.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:30.060+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:30.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:30.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:30.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:30.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:30.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:30.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:30.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:30.379+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:30.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:30.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:30.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:30.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:30.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:30.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:30.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:30.693+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:31.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:31.094+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:31.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:31.094+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:31.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:31.094+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:31.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:31.095+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:31.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:31.401+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:31.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:31.401+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:31.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:31.401+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:31.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:31.402+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:31.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:31.714+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:31.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:31.714+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:31.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:31.714+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:31.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:31.715+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:32.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:32.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:32.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:32.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:32.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:32.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:32.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:32.114+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:32.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:32.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:32.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:32.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:32.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:32.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:32.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:32.396+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:33.073 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:32.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:33.074 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:32.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:33.074 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:32.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:33.074 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:32.727+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:33.366 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:33.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:33.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:33.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:33.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:33.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:33.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:33.067+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:33.727 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:06:33.358+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 52731 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:06:33.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:33.435+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:33.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:33.435+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:33.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:33.435+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:33.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:33.435+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:34.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:33.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:34.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:33.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:34.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:33.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:34.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:33.721+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:34.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:34.048+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:34.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:34.048+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:34.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:34.048+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:34.426 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:34.049+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:34.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:34.417+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:34.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:34.417+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:34.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:34.417+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:34.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:34.418+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:35.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:34.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:35.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:34.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:35.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:34.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:35.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:34.730+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:35.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:35.037+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:35.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:35.037+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:35.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:35.037+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:35.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:35.038+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:35.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:35.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:35.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:35.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:35.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:35.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:35.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:35.408+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:36.046 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:35.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:36.047 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:35.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:36.047 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:35.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:36.047 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:35.732+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:36.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:36.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:36.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:36.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:36.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:36.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:36.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:36.040+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:36.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:36.403+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:36.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:36.403+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:36.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:36.403+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:36.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:36.404+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:37.030 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:36.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:37.030 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:36.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:37.030 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:36.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:37.030 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:36.725+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:37.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:37.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:37.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:37.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:37.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:37.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:37.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:37.023+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:37.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:37.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:37.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:37.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:37.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:37.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:37.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:37.394+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:38.001 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:37.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:38.001 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:37.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:38.001 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:37.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:38.001 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:37.739+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:38.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:37.993+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:38.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:37.993+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:38.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:37.993+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:38.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:37.994+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:38.727 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:06:38.365+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 52776 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:06:38.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:38.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:38.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:38.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:38.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:38.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:38.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:38.367+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:39.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:38.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:39.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:38.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:39.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:38.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:39.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:38.765+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:39.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:38.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:39.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:38.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:39.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:38.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:39.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:38.948+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:39.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:39.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:39.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:39.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:39.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:39.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:39.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:39.334+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:40.216 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:39.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:40.216 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:39.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:40.216 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:39.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:40.216 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:39.977+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:40.216 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:39.809+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:40.217 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:39.809+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:40.217 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:39.809+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:40.217 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:39.809+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:40.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:40.316+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:40.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:40.317+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:40.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:40.317+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:40.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:40.317+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:40.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:40.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:40.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:40.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:40.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:40.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:40.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:40.830+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:41.285 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:40.993+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:41.286 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:40.993+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:41.286 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:40.993+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:41.286 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:40.994+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:41.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:41.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:41.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:41.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:41.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:41.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:41.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:41.278+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:42.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:41.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:42.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:41.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:42.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:41.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:42.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:41.786+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:42.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:41.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:42.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:41.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:42.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:41.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:42.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:41.965+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:42.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:42.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:42.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:42.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:42.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:42.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:42.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:42.240+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:43.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:42.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:43.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:42.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:43.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:42.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:43.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:42.802+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:43.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:42.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:43.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:42.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:43.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:42.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:43.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:42.939+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:43.498 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:06:43.371+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 52822 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:06:43.499 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:43.231+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:43.499 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:43.231+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:43.499 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:43.231+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:43.499 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:43.231+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:44.201 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:43.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:44.201 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:43.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:44.201 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:43.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:44.201 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:43.982+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:44.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:43.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:44.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:43.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:44.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:43.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:44.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:43.792+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:44.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:44.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:44.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:44.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:44.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:44.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:44.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:44.193+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:45.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:44.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:45.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:44.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:45.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:44.964+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:45.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:44.965+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:45.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:44.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:45.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:44.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:45.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:44.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:45.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:44.783+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:45.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:45.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:45.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:45.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:45.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:45.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:45.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:45.191+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:46.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:45.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:46.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:45.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:46.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:45.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:46.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:45.757+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:46.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:45.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:46.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:45.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:46.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:45.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:46.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:45.922+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:46.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:46.230+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:46.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:46.230+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:46.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:46.230+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:46.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:46.230+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:47.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:46.953+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:47.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:46.953+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:47.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:46.953+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:47.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:46.954+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:47.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:46.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:47.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:46.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:47.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:46.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:47.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:46.766+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:47.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:47.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:47.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:47.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:47.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:47.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:47.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:47.226+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:48.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:47.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:48.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:47.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:48.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:47.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:48.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:47.721+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:48.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:47.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:48.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:47.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:48.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:47.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:48.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:47.955+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:48.507 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:06:48.378+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 52867 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:06:48.508 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:48.227+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:48.508 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:48.227+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:48.508 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:48.227+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:48.508 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:48.227+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:48.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:48.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:48.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:48.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:48.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:48.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:48.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:48.741+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:49.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:49.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:49.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:49.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:49.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:49.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:49.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:49.202+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:49.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:48.975+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:49.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:48.975+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:49.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:48.975+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:49.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:48.975+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:49.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T03:06:49.608+0000 7ff0e4ed9700 0 monclient(hunting): authenticate timed out after 300 2024-04-27T03:06:49.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:49.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:49.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:49.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:49.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:49.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:49.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:49.739+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:50.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:49.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:50.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:49.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:50.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:49.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:50.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:49.975+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:50.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:50.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:50.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:50.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:50.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:50.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:50.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:50.162+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:51.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:50.993+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:51.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:50.993+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:51.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:50.993+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:51.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:50.994+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:51.212 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:50.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:51.212 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:50.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:51.212 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:50.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:51.212 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:50.777+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:51.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:51.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:51.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:51.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:51.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:51.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:51.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:51.203+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:52.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:51.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:52.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:51.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:52.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:51.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:52.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:51.757+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:52.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:52.023+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:52.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:52.023+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:52.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:52.023+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:52.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:52.025+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:52.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:52.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:52.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:52.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:52.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:52.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:52.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:52.163+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:53.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:52.749+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:53.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:52.749+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:53.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:52.749+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:53.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:52.750+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:53.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:53.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:53.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:53.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:53.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:53.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:53.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:53.062+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:53.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:53.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:53.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:53.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:53.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:53.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:53.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:53.155+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:53.720 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:06:53.384+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 52913 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:06:53.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:53.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:53.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:53.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:53.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:53.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:53.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:53.713+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:54.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:54.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:54.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:54.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:54.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:54.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:54.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:54.026+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:54.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:54.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:54.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:54.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:54.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:54.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:54.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:54.106+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:55.025 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:54.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:55.025 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:54.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:55.025 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:54.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:55.025 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:54.754+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:55.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:55.017+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:55.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:55.017+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:55.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:55.017+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:55.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:55.018+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:55.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:55.062+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:55.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:55.062+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:55.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:55.062+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:55.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:55.062+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:55.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:55.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:55.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:55.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:55.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:55.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:55.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:55.735+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:56.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:56.048+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:56.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:56.048+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:56.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:56.048+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:56.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:56.050+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:56.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:56.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:56.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:56.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:56.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:56.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:56.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:56.021+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:56.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:56.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:56.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:56.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:56.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:56.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:56.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:56.712+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:57.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:57.030+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:57.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:57.030+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:57.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:57.030+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:57.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:57.030+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:57.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:56.982+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:57.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:56.982+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:57.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:56.982+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:57.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:56.982+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:57.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:57.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:57.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:57.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:57.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:57.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:57.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:57.690+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:58.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:57.962+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:58.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:57.962+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:58.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:57.962+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:58.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:57.962+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:58.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:58.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:58.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:58.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:58.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:58.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:58.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:58.016+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:58.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:06:58.391+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 52957 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:06:58.999 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:58.730+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:59.000 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:58.730+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:59.000 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:58.730+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:06:59.000 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:58.731+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:59.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:59.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:59.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:59.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:59.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:59.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:06:59.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:06:59.061+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:06:59.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:58.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:59.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:58.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:59.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:58.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:06:59.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:58.992+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:00.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:59.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:00.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:59.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:00.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:59.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:00.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:06:59.993+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:00.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:59.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:00.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:59.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:00.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:59.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:00.037 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:06:59.747+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:00.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:00.028+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:00.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:00.028+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:00.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:00.028+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:00.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:00.029+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:00.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:00.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:00.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:00.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:00.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:00.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:00.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:00.788+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:01.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:01.059+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:01.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:01.059+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:01.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:01.059+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:01.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:01.060+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:01.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:01.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:01.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:01.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:01.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:01.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:01.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:01.011+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:02.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:01.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:02.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:01.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:02.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:01.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:02.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:01.829+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:02.111 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:01.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:02.111 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:01.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:02.111 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:01.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:02.111 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:01.998+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:02.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:02.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:02.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:02.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:02.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:02.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:02.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:02.102+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:03.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:02.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:03.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:02.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:03.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:02.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:03.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:02.962+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:03.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:02.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:03.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:02.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:03.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:02.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:03.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:02.863+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:03.400 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:03.126+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:03.400 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:03.126+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:03.400 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:03.126+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:03.400 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:03.128+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:03.727 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:07:03.396+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53003 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:07:04.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:03.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:04.115 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:03.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:04.115 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:03.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:04.115 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:03.824+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:04.115 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:03.976+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:04.115 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:03.976+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:04.115 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:03.976+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:04.115 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:03.977+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:04.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:04.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:04.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:04.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:04.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:04.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:04.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:04.106+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:05.161 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:04.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:05.161 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:04.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:05.161 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:04.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:05.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:04.791+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:05.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:04.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:05.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:04.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:05.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:04.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:05.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:04.954+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:05.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:05.152+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:05.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:05.152+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:05.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:05.152+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:05.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:05.154+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:05.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:05.828+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:05.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:05.828+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:05.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:05.828+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:05.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:05.829+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:06.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:06.166+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:06.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:06.166+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:06.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:06.166+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:06.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:06.166+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:06.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:05.949+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:06.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:05.949+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:06.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:05.949+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:06.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:05.950+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:07.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:06.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:07.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:06.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:07.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:06.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:07.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:06.944+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:07.133 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:06.832+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:07.133 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:06.832+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:07.133 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:06.832+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:07.133 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:06.832+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:07.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:07.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:07.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:07.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:07.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:07.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:07.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:07.124+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:08.175 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:07.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:08.175 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:07.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:08.175 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:07.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:08.175 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:07.818+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:08.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:07.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:08.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:07.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:08.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:07.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:08.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:07.963+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:08.430 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:07:08.402+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53048 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:07:08.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:08.166+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:08.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:08.166+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:08.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:08.166+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:08.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:08.167+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:09.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:08.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:09.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:08.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:09.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:08.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:09.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:08.823+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:09.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:08.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:09.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:08.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:09.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:08.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:09.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:08.955+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:09.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:09.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:09.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:09.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:09.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:09.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:09.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:09.212+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:10.172 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:09.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:10.172 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:09.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:10.172 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:09.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:10.172 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:09.807+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:10.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:09.990+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:10.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:09.990+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:10.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:09.990+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:10.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:09.991+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:10.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:10.163+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:10.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:10.163+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:10.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:10.163+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:10.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:10.164+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:11.019 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:10.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:11.019 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:10.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:11.019 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:10.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:11.020 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:10.776+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:11.365 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:11.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:11.365 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:11.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:11.365 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:11.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:11.365 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:11.176+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:11.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:11.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:11.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:11.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:11.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:11.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:11.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:11.011+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:12.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:11.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:12.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:11.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:12.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:11.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:12.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:11.748+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:12.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:12.198+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:12.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:12.198+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:12.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:12.198+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:12.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:12.198+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:12.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:12.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:12.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:12.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:12.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:12.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:12.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:12.056+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:13.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:12.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:13.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:12.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:13.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:12.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:13.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:12.719+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:13.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:13.247+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:13.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:13.247+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:13.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:13.247+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:13.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:13.248+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:13.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:13.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:13.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:13.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:13.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:13.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:13.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:13.050+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:13.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:07:13.409+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53094 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:07:14.067 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:13.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:14.067 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:13.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:14.067 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:13.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:14.067 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:13.729+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:14.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:14.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:14.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:14.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:14.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:14.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:14.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:14.059+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:14.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:14.285+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:14.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:14.285+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:14.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:14.285+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:14.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:14.286+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:14.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:14.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:14.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:14.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:14.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:14.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:14.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:14.706+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:15.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:15.040+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:15.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:15.040+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:15.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:15.040+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:15.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:15.041+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:15.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:15.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:15.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:15.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:15.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:15.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:15.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:15.253+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:16.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:15.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:16.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:15.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:16.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:15.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:16.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:15.749+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:16.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:16.249+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:16.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:16.249+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:16.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:16.249+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:16.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:16.250+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:16.376 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:16.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:16.376 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:16.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:16.376 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:16.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:16.376 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:16.072+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:17.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:16.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:17.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:16.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:17.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:16.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:17.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:16.777+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:17.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:17.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:17.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:17.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:17.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:17.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:17.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:17.106+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:17.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:17.270+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:17.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:17.270+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:17.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:17.270+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:17.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:17.271+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:18.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:17.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:18.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:17.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:18.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:17.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:18.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:17.805+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:18.424 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:18.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:18.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:18.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:18.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:18.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:18.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:18.307+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:18.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:18.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:18.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:18.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:18.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:18.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:18.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:18.083+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:18.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:07:18.416+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53138 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:07:19.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:18.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:19.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:18.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:19.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:18.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:19.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:18.830+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:19.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:19.076+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:19.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:19.076+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:19.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:19.076+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:19.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:19.077+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:19.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:19.350+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:19.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:19.350+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:19.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:19.350+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:19.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:19.350+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:20.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:19.831+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:20.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:19.831+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:20.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:19.831+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:20.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:19.832+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:20.385 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:20.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:20.385 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:20.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:20.385 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:20.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:20.385 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:20.104+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:20.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:20.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:20.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:20.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:20.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:20.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:20.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:20.377+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:21.094 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:20.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:21.094 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:20.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:21.094 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:20.853+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:21.094 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:20.854+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:21.409 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:21.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:21.409 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:21.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:21.409 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:21.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:21.409 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:21.086+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:21.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:21.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:21.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:21.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:21.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:21.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:21.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:21.402+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:22.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:21.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:22.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:21.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:22.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:21.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:22.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:21.885+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:22.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:22.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:22.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:22.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:22.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:22.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:22.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:22.090+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:22.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:22.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:22.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:22.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:22.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:22.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:22.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:22.405+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:22.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:22.903+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:22.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:22.903+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:22.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:22.903+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:22.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:22.903+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:23.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:23.134+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:23.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:23.134+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:23.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:23.134+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:23.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:23.135+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:23.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:07:23.423+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53184 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:07:23.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:23.392+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:23.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:23.392+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:23.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:23.392+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:23.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:23.393+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:24.155 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:23.893+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:24.155 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:23.893+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:24.155 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:23.893+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:24.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:23.894+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:24.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:24.147+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:24.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:24.147+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:24.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:24.147+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:24.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:24.147+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:24.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:24.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:24.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:24.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:24.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:24.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:24.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:24.387+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:25.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:24.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:25.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:24.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:25.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:24.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:25.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:24.855+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:25.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:25.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:25.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:25.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:25.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:25.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:25.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:25.108+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:25.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:25.411+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:25.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:25.411+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:25.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:25.411+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:25.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:25.411+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:26.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:26.078+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:26.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:26.078+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:26.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:26.078+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:26.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:26.079+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:26.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:25.864+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:26.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:25.864+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:26.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:25.864+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:26.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:25.865+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:26.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:26.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:26.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:26.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:26.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:26.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:26.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:26.441+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:26.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:26.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:26.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:26.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:26.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:26.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:26.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:26.818+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:27.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:27.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:27.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:27.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:27.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:27.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:27.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:27.041+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:27.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:27.488+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:27.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:27.488+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:27.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:27.488+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:27.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:27.489+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:28.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:27.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:28.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:27.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:28.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:27.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:28.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:27.777+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:28.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:28.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:28.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:28.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:28.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:28.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:28.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:28.013+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:28.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:07:28.429+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53229 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:07:28.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:28.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:28.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:28.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:28.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:28.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:28.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:28.467+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:29.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:28.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:29.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:28.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:29.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:28.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:29.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:28.794+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:29.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:29.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:29.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:29.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:29.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:29.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:29.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:29.062+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:29.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:29.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:29.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:29.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:29.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:29.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:29.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:29.438+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:30.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:29.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:30.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:29.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:30.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:29.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:30.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:29.792+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:30.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:30.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:30.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:30.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:30.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:30.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:30.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:30.046+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:30.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:30.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:30.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:30.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:30.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:30.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:30.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:30.443+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:31.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:30.996+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:31.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:30.996+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:31.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:30.996+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:31.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:30.997+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:31.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:30.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:31.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:30.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:31.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:30.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:31.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:30.842+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:31.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:31.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:31.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:31.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:31.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:31.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:31.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:31.419+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:32.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:31.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:32.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:31.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:32.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:31.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:32.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:31.953+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:32.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:31.863+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:32.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:31.863+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:32.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:31.863+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:32.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:31.864+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:32.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:32.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:32.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:32.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:32.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:32.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:32.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:32.456+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:33.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:32.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:33.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:32.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:33.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:32.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:33.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:32.861+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:33.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:32.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:33.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:32.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:33.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:32.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:33.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:33.000+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:33.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:07:33.436+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53275 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:07:33.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:33.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:33.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:33.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:33.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:33.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:33.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:33.425+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:34.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:33.982+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:34.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:33.982+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:34.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:33.982+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:34.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:33.982+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:34.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:33.837+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:34.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:33.837+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:34.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:33.837+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:34.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:33.837+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:34.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:34.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:34.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:34.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:34.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:34.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:34.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:34.398+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:35.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:34.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:35.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:34.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:35.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:34.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:35.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:34.863+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:35.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:34.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:35.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:34.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:35.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:34.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:35.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:34.944+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:35.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:35.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:35.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:35.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:35.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:35.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:35.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:35.369+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:36.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:35.883+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:36.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:35.883+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:36.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:35.883+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:36.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:35.884+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:36.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:35.964+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:36.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:35.964+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:36.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:35.964+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:36.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:35.965+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:36.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:36.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:36.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:36.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:36.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:36.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:36.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:36.384+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:37.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:36.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:37.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:36.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:37.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:36.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:37.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:36.868+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:37.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:36.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:37.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:36.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:37.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:36.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:37.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:36.948+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:37.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:37.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:37.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:37.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:37.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:37.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:37.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:37.406+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:38.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:37.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:38.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:37.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:38.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:37.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:38.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:37.885+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:38.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:37.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:38.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:37.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:38.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:37.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:38.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:37.929+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:38.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:07:38.442+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53319 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:07:38.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:38.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:38.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:38.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:38.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:38.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:38.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:38.434+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:39.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:38.914+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:39.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:38.914+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:39.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:38.914+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:39.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:38.916+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:39.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:38.977+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:39.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:38.977+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:39.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:38.977+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:39.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:38.978+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:39.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:39.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:39.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:39.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:39.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:39.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:39.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:39.465+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:40.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:39.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:40.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:39.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:40.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:39.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:40.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:39.912+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:40.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:40.003+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:40.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:40.003+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:40.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:40.003+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:40.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:40.003+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:40.965 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:40.499+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:40.965 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:40.499+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:40.965 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:40.499+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:40.965 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:40.500+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:41.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:40.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:41.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:40.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:41.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:40.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:41.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:40.957+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:41.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:41.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:41.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:41.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:41.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:41.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:41.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:41.010+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:41.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:41.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:41.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:41.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:41.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:41.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:41.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:41.491+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:42.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:41.946+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:42.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:41.946+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:42.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:41.946+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:42.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:41.947+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:42.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:42.039+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:42.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:42.039+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:42.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:42.039+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:42.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:42.039+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:42.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:42.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:42.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:42.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:42.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:42.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:42.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:42.535+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:43.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:43.057+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:43.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:43.057+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:43.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:43.057+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:43.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:43.057+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:43.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:42.937+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:43.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:42.937+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:43.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:42.937+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:43.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:42.938+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:43.729 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:07:43.448+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53365 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:07:43.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:43.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:43.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:43.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:43.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:43.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:43.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:43.537+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:44.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:43.944+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:44.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:43.944+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:44.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:43.944+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:44.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:43.944+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:44.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:44.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:44.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:44.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:44.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:44.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:44.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:44.030+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:44.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:44.576+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:44.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:44.576+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:44.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:44.576+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:44.907 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:44.578+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:45.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:44.983+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:45.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:44.983+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:45.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:44.983+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:45.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:44.984+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:45.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:44.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:45.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:44.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:45.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:44.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:45.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:44.898+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:45.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:45.562+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:45.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:45.562+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:45.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:45.562+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:45.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:45.564+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:46.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:45.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:46.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:45.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:46.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:45.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:46.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:45.870+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:46.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:46.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:46.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:46.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:46.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:46.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:46.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:46.028+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:46.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:46.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:46.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:46.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:46.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:46.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:46.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:46.546+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:47.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:46.871+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:47.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:46.871+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:47.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:46.871+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:47.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:46.872+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:47.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:47.018+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:47.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:47.018+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:47.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:47.018+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:47.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:47.019+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:47.870 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:47.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:47.871 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:47.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:47.871 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:47.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:47.871 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:47.576+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:48.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:47.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:48.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:47.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:48.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:47.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:48.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:47.863+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:48.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:48.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:48.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:48.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:48.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:48.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:48.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:48.024+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:48.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:07:48.456+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53410 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:07:48.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:48.605+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:48.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:48.605+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:48.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:48.605+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:48.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:48.606+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:49.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:48.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:49.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:48.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:49.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:48.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:49.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:48.978+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:49.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:48.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:49.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:48.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:49.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:48.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:49.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:48.912+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:49.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:49.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:49.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:49.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:49.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:49.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:49.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:49.625+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:50.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:49.914+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:50.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:49.914+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:50.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:49.914+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:50.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:49.915+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:50.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:49.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:50.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:49.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:50.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:49.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:50.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:49.960+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:50.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:50.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:50.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:50.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:50.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:50.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:50.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:50.639+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:50.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:50.872+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:50.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:50.872+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:50.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:50.872+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:50.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:50.873+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:51.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:50.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:51.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:50.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:51.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:50.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:51.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:50.937+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:51.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:51.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:51.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:51.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:51.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:51.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:51.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:51.655+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:51.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:51.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:51.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:51.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:51.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:51.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:51.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:51.840+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:52.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:51.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:52.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:51.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:52.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:51.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:52.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:51.937+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:52.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:52.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:52.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:52.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:52.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:52.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:52.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:52.619+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:52.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:52.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:52.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:52.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:52.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:52.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:52.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:52.795+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:53.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:52.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:53.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:52.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:53.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:52.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:53.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:52.972+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:53.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:07:53.462+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53452 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:07:53.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:53.596+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:53.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:53.596+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:53.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:53.596+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:53.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:53.597+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:54.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:53.778+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:54.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:53.778+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:54.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:53.778+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:54.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:53.779+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:54.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:53.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:54.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:53.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:54.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:53.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:54.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:53.935+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:54.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:54.621+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:54.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:54.621+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:54.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:54.621+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:54.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:54.622+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:54.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:54.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:54.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:54.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:54.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:54.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:54.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:54.814+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:55.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:54.907+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:55.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:54.907+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:55.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:54.907+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:55.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:54.907+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:55.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:55.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:55.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:55.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:55.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:55.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:55.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:55.827+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:55.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:55.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:55.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:55.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:55.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:55.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:55.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:55.580+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:56.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:55.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:56.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:55.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:56.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:55.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:56.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:55.918+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:56.863 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:56.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:56.863 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:56.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:56.863 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:56.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:56.863 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:56.577+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:57.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:56.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:57.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:56.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:57.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:56.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:57.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:56.901+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:57.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:56.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:57.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:56.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:57.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:56.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:57.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:56.855+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:57.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:57.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:57.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:57.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:57.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:57.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:57.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:57.546+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:58.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:57.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:58.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:57.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:58.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:57.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:58.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:57.873+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:58.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:57.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:58.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:57.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:58.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:57.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:58.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:57.842+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:58.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:07:58.469+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53496 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:07:58.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:58.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:58.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:58.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:58.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:58.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:58.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:58.525+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:59.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:58.854+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:59.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:58.854+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:59.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:58.854+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:07:59.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:58.855+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:59.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:58.878+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:59.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:58.878+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:59.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:58.878+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:07:59.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:58.879+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:07:59.828 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:59.502+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:59.828 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:59.502+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:59.828 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:59.502+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:07:59.828 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:07:59.503+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:00.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:59.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:00.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:59.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:00.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:59.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:00.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:07:59.862+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:00.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:59.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:00.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:59.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:00.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:59.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:00.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:07:59.820+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:00.786 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:00.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:00.786 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:00.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:00.786 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:00.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:00.786 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:00.546+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:01.188 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:00.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:01.188 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:00.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:01.188 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:00.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:01.188 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:00.778+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:01.188 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:00.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:01.189 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:00.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:01.189 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:00.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:01.189 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:00.883+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:01.763 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:01.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:01.763 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:01.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:01.763 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:01.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:01.763 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:01.497+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:02.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:01.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:02.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:01.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:02.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:01.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:02.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:01.834+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:02.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:01.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:02.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:01.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:02.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:01.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:02.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:01.755+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:02.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:02.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:02.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:02.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:02.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:02.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:02.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:02.472+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:02.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:02.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:02.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:02.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:02.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:02.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:02.820 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:02.712+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:03.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:02.810+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:03.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:02.810+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:03.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:02.810+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:03.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:02.811+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:03.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:08:03.475+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53542 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:08:03.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:03.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:03.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:03.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:03.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:03.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:03.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:03.448+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:04.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:03.725+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:04.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:03.725+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:04.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:03.725+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:04.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:03.726+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:04.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:03.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:04.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:03.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:04.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:03.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:04.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:03.783+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:04.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:04.695+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:04.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:04.695+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:04.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:04.695+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:04.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:04.695+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:04.791 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:04.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:04.791 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:04.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:04.791 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:04.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:04.791 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:04.488+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:05.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:04.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:05.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:04.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:05.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:04.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:05.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:04.778+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:05.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:05.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:05.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:05.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:05.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:05.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:05.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:05.711+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:05.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:05.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:05.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:05.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:05.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:05.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:05.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:05.527+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:05.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:05.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:05.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:05.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:05.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:05.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:05.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:05.736+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:06.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:06.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:06.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:06.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:06.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:06.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:06.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:06.707+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:06.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:06.732+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:06.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:06.732+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:06.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:06.732+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:06.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:06.732+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:06.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:06.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:06.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:06.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:06.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:06.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:06.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:06.573+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:07.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:07.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:07.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:07.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:07.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:07.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:07.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:07.708+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:07.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:07.768+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:07.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:07.768+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:07.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:07.768+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:07.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:07.768+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:07.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:07.616+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:07.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:07.616+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:07.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:07.616+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:07.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:07.617+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:08.795 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:08:08.481+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53587 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:08:08.795 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:08.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:08.795 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:08.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:08.795 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:08.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:08.796 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:08.599+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:08.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:08.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:08.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:08.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:08.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:08.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:08.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:08.711+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:09.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:08.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:09.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:08.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:09.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:08.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:09.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:08.786+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:09.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:09.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:09.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:09.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:09.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:09.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:09.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:09.759+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:09.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:09.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:09.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:09.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:09.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:09.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:09.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:09.813+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:09.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:09.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:09.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:09.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:09.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:09.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:09.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:09.587+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:10.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:10.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:10.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:10.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:10.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:10.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:10.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:10.635+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:10.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:10.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:10.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:10.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:10.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:10.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:10.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:10.863+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:10.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:10.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:10.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:10.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:10.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:10.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:10.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:10.772+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:11.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:11.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:11.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:11.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:11.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:11.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:11.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:11.621+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:11.909 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:11.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:11.909 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:11.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:11.909 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:11.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:11.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:11.792+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:12.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:11.899+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:12.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:11.899+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:12.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:11.899+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:12.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:11.900+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:12.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:12.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:12.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:12.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:12.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:12.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:12.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:12.630+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:12.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:12.783+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:12.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:12.783+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:12.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:12.783+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:12.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:12.784+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:12.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:12.858+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:12.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:12.858+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:12.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:12.858+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:12.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:12.859+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:13.766 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:08:13.487+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53633 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:08:13.766 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:13.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:13.766 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:13.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:13.766 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:13.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:13.766 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:13.592+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:14.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:13.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:14.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:13.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:14.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:13.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:14.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:13.872+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:14.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:13.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:14.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:13.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:14.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:13.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:14.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:13.796+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:14.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:14.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:14.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:14.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:14.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:14.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:14.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:14.812+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:14.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:14.592+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:14.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:14.592+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:14.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:14.592+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:14.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:14.593+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:15.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:14.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:15.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:14.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:15.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:14.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:15.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:14.890+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:15.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:15.570+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:15.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:15.570+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:15.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:15.570+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:15.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:15.570+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:15.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:15.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:15.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:15.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:15.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:15.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:15.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:15.771+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:16.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:15.876+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:16.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:15.876+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:16.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:15.876+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:16.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:15.876+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:16.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:16.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:16.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:16.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:16.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:16.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:16.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:16.759+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:16.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:16.603+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:16.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:16.603+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:16.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:16.603+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:16.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:16.604+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:16.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:16.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:16.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:16.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:16.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:16.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:16.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:16.848+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:17.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:17.636+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:17.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:17.636+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:17.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:17.636+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:17.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:17.636+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:17.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:17.767+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:17.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:17.767+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:17.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:17.767+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:17.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:17.768+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:17.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:17.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:17.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:17.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:17.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:17.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:17.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:17.884+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:18.783 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:08:18.495+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53677 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:08:18.783 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:18.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:18.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:18.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:18.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:18.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:18.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:18.634+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:19.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:18.774+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:19.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:18.774+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:19.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:18.774+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:19.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:18.775+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:19.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:18.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:19.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:18.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:19.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:18.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:19.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:18.894+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:19.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:19.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:19.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:19.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:19.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:19.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:19.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:19.639+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:19.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:19.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:19.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:19.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:19.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:19.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:19.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:19.824+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:19.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:19.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:19.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:19.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:19.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:19.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:19.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:19.885+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:20.981 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:20.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:20.981 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:20.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:20.981 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:20.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:20.981 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:20.663+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:20.982 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:20.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:20.982 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:20.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:20.982 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:20.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:20.982 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:20.786+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:20.983 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:20.872+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:20.983 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:20.872+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:20.983 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:20.872+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:20.983 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:20.873+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:21.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:21.778+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:21.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:21.778+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:21.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:21.778+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:21.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:21.780+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:21.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:21.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:21.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:21.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:21.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:21.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:21.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:21.853+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:21.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:21.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:21.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:21.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:21.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:21.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:21.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:21.647+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:22.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:22.765+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:22.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:22.765+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:22.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:22.765+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:22.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:22.766+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:22.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:22.840+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:22.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:22.840+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:22.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:22.840+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:22.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:22.841+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:22.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:22.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:22.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:22.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:22.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:22.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:22.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:22.677+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:23.767 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:08:23.501+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53723 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:08:23.767 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:23.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:23.767 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:23.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:23.767 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:23.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:23.767 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:23.665+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:24.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:23.783+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:24.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:23.783+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:24.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:23.783+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:24.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:23.784+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:24.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:23.811+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:24.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:23.811+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:24.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:23.811+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:24.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:23.812+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:24.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:24.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:24.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:24.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:24.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:24.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:24.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:24.685+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:24.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:24.807+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:24.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:24.807+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:24.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:24.807+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:24.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:24.808+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:24.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:24.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:24.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:24.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:24.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:24.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:24.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:24.822+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:25.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:25.689+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:25.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:25.689+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:25.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:25.689+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:25.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:25.690+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:26.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:25.826+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:26.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:25.826+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:26.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:25.826+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:26.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:25.827+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:26.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:25.838+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:26.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:25.838+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:26.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:25.838+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:26.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:25.839+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:27.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:26.846+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:27.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:26.846+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:27.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:26.846+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:27.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:26.847+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:27.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:26.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:27.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:26.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:27.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:26.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:27.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:26.823+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:27.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:26.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:27.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:26.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:27.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:26.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:27.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:26.732+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:28.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:27.729+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:28.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:27.729+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:28.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:27.729+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:28.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:27.730+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:28.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:27.816+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:28.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:27.816+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:28.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:27.816+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:28.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:27.817+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:28.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:27.782+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:28.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:27.782+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:28.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:27.782+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:28.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:27.783+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:28.772 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:08:28.508+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53768 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:08:28.772 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:28.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:28.772 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:28.757+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:28.772 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:28.757+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:28.772 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:28.757+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:29.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:28.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:29.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:28.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:29.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:28.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:29.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:28.823+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:29.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:28.763+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:29.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:28.763+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:29.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:28.763+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:29.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:28.764+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:30.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:29.728+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:30.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:29.728+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:30.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:29.728+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:30.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:29.728+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:30.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:29.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:30.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:29.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:30.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:29.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:30.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:29.795+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:30.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:29.804+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:30.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:29.804+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:30.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:29.804+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:30.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:29.805+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:30.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:30.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:30.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:30.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:30.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:30.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:30.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:30.792+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:30.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:30.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:30.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:30.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:30.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:30.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:30.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:30.711+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:30.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:30.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:30.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:30.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:30.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:30.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:30.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:30.786+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:31.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:31.803+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:31.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:31.803+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:31.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:31.803+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:31.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:31.803+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:31.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:31.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:31.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:31.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:31.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:31.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:31.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:31.801+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:31.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:31.675+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:31.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:31.675+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:31.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:31.675+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:31.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:31.675+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:32.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:32.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:32.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:32.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:32.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:32.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:32.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:32.825+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:32.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:32.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:32.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:32.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:32.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:32.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:32.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:32.764+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:32.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:32.631+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:32.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:32.631+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:32.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:32.631+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:32.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:32.632+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:33.785 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:08:33.515+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53814 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:08:33.785 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:33.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:33.785 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:33.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:33.786 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:33.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:33.786 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:33.762+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:33.786 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:33.639+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:33.786 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:33.639+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:33.786 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:33.639+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:33.786 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:33.639+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:34.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:33.776+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:34.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:33.776+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:34.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:33.776+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:34.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:33.778+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:34.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:34.671+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:34.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:34.671+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:34.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:34.671+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:34.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:34.672+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:34.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:34.755+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:34.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:34.755+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:34.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:34.755+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:34.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:34.756+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:34.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:34.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:34.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:34.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:34.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:34.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:34.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:34.716+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:35.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:35.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:35.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:35.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:35.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:35.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:35.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:35.693+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:35.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:35.778+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:35.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:35.778+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:35.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:35.778+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:35.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:35.778+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:35.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:35.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:35.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:35.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:35.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:35.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:35.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:35.695+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:36.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:36.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:36.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:36.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:36.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:36.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:36.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:36.825+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:36.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:36.681+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:36.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:36.681+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:36.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:36.681+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:36.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:36.682+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:36.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:36.737+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:36.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:36.737+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:36.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:36.737+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:36.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:36.738+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:37.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:37.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:37.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:37.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:37.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:37.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:37.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:37.644+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:37.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:37.740+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:37.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:37.740+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:37.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:37.740+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:37.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:37.741+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:37.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:37.808+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:37.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:37.808+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:37.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:37.808+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:37.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:37.809+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:38.834 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:08:38.521+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53858 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:08:38.835 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:38.755+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:38.835 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:38.755+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:38.835 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:38.755+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:38.835 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:38.756+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:38.835 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:38.639+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:38.835 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:38.639+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:38.835 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:38.639+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:38.835 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:38.640+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:39.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:38.826+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:39.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:38.826+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:39.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:38.826+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:39.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:38.826+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:39.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:39.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:39.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:39.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:39.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:39.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:39.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:39.838+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:39.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:39.726+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:39.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:39.726+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:39.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:39.726+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:39.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:39.727+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:39.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:39.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:39.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:39.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:39.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:39.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:39.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:39.635+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:40.846 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:40.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:40.846 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:40.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:40.846 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:40.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:40.846 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:40.587+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:40.847 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:40.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:40.847 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:40.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:40.847 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:40.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:40.847 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:40.712+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:41.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:40.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:41.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:40.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:41.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:40.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:41.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:40.839+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:41.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:41.616+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:41.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:41.616+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:41.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:41.616+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:41.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:41.618+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:41.850 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:41.694+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:41.850 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:41.694+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:41.850 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:41.694+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:41.850 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:41.695+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:42.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:41.840+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:42.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:41.840+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:42.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:41.840+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:42.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:41.841+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:42.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:42.822+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:42.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:42.822+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:42.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:42.822+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:42.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:42.822+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:42.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:42.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:42.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:42.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:42.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:42.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:42.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:42.578+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:42.981 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:42.707+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:42.981 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:42.707+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:42.981 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:42.707+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:42.981 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:42.707+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:43.816 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:08:43.528+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53904 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:08:43.816 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:43.717+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:43.816 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:43.717+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:43.816 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:43.717+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:43.816 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:43.717+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:43.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:43.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:43.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:43.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:43.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:43.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:43.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:43.563+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:44.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:43.807+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:44.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:43.807+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:44.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:43.807+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:44.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:43.808+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:44.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:44.702+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:44.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:44.702+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:44.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:44.702+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:44.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:44.702+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:44.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:44.767+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:44.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:44.767+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:44.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:44.767+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:44.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:44.768+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:44.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:44.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:44.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:44.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:44.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:44.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:44.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:44.581+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:45.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:45.795+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:45.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:45.795+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:45.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:45.795+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:45.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:45.796+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:45.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:45.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:45.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:45.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:45.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:45.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:45.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:45.560+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:45.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:45.731+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:45.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:45.731+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:45.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:45.731+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:45.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:45.732+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:46.633 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:46.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:46.633 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:46.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:46.633 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:46.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:46.634 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:46.556+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:46.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:46.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:46.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:46.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:46.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:46.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:46.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:46.771+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:46.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:46.758+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:46.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:46.758+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:46.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:46.758+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:46.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:46.758+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:47.816 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:47.516+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:47.816 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:47.516+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:47.816 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:47.516+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:47.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:47.517+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:48.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:47.807+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:48.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:47.807+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:48.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:47.807+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:48.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:47.809+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:48.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:47.807+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:48.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:47.807+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:48.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:47.807+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:48.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:47.808+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:48.840 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:08:48.535+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53949 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:08:48.840 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:48.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:48.840 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:48.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:48.840 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:48.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:48.840 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:48.486+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:49.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:48.831+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:49.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:48.831+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:49.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:48.831+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:49.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:48.832+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:49.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:48.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:49.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:48.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:49.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:48.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:49.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:48.844+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:49.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:49.459+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:49.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:49.459+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:49.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:49.459+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:49.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:49.460+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:50.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:49.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:50.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:49.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:50.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:49.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:50.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:49.814+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:50.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:49.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:50.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:49.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:50.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:49.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:50.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:49.825+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:50.785 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:50.481+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:50.785 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:50.481+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:50.785 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:50.481+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:50.785 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:50.482+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:51.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:50.776+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:51.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:50.776+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:51.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:50.776+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:51.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:50.777+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:51.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:50.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:51.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:50.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:51.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:50.812+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:51.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:50.813+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:51.760 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:51.750+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:51.761 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:51.750+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:51.761 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:51.750+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:51.761 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:51.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:51.761 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:51.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:51.761 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:51.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:51.761 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:51.502+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:52.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:51.752+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:52.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:51.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:52.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:51.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:52.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:51.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:52.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:51.817+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:52.769 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:52.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:52.769 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:52.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:52.769 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:52.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:52.769 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:52.509+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:52.760+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:53.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:52.760+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:53.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:52.760+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:53.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:52.761+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:53.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:52.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:53.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:52.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:53.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:52.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:53.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:52.824+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:53.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:52.876+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:52.877773+0000) 2024-04-27T03:08:53.750 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:08:53.542+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 53995 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:08:53.750 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:53.540+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:53.541247+0000) 2024-04-27T03:08:53.751 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:53.539+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:53.540529+0000) 2024-04-27T03:08:53.751 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:53.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:53.751 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:53.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:53.751 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:53.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:53.751 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:53.490+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:53.751 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:53.540+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:53.541431+0000) 2024-04-27T03:08:54.003 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:53.745+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:53.746067+0000) 2024-04-27T03:08:54.003 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:53.776+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:54.004 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:53.776+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:54.004 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:53.776+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:54.004 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:53.777+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:54.004 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:53.877+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:53.877957+0000) 2024-04-27T03:08:54.004 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:53.741+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:53.742638+0000) 2024-04-27T03:08:54.004 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:53.790+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:54.004 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:53.790+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:54.004 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:53.790+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:54.004 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:53.791+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:54.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:54.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:54.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:54.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:54.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:54.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:54.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:54.511+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:55.003 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:54.750+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:54.750837+0000) 2024-04-27T03:08:55.003 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:54.744+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:54.745499+0000) 2024-04-27T03:08:55.004 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:54.814+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:55.004 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:54.814+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:55.004 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:54.814+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:55.004 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:54.815+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:55.004 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:54.749+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:55.004 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:54.749+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:55.004 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:54.749+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:55.004 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:54.750+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:55.004 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:54.879+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:54.880249+0000) 2024-04-27T03:08:55.813 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:55.746+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:55.747626+0000) 2024-04-27T03:08:55.814 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:55.743+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:55.814 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:55.743+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:55.814 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:55.743+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:55.814 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:55.743+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:55.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:55.539+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:55.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:55.539+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:55.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:55.539+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:55.815 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:55.540+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:55.815 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:55.752+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:55.752932+0000) 2024-04-27T03:08:56.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:55.881+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:55.882531+0000) 2024-04-27T03:08:56.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:55.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:56.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:55.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:56.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:55.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:56.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:55.851+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:56.707 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:56.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:56.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:56.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:56.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:56.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:56.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:56.579+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:56.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:56.753+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:56.754244+0000) 2024-04-27T03:08:56.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:56.699+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:56.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:56.699+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:56.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:56.699+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:56.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:56.699+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:56.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:56.883+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:56.883991+0000) 2024-04-27T03:08:56.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:56.748+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:56.748890+0000) 2024-04-27T03:08:56.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:56.858+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:56.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:56.858+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:56.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:56.858+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:56.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:56.858+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:57.891 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:57.682+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:57.892 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:57.682+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:57.892 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:57.682+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:57.892 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:57.683+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:57.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:57.748+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:57.749080+0000) 2024-04-27T03:08:57.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:57.820+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:57.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:57.820+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:57.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:57.820+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:57.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:57.821+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:57.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:57.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:57.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:57.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:57.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:57.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:57.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:57.587+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:57.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:57.753+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:57.754459+0000) 2024-04-27T03:08:58.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:57.883+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:57.884147+0000) 2024-04-27T03:08:58.814 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:08:58.549+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54039 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:08:58.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:58.548+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:58.549007+0000) 2024-04-27T03:08:58.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:58.748+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:58.749324+0000) 2024-04-27T03:08:58.815 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:58.548+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:58.548949+0000) 2024-04-27T03:08:58.815 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:58.702+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:58.815 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:58.702+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:58.815 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:58.702+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:58.816 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:58.702+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:58.816 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:58.547+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:58.548450+0000) 2024-04-27T03:08:58.816 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:58.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:58.816 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:58.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:58.816 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:58.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:58.816 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:58.575+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:58.816 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:58.753+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:58.754726+0000) 2024-04-27T03:08:59.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:58.883+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:58.884320+0000) 2024-04-27T03:08:59.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:58.806+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:59.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:58.806+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:59.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:58.806+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:08:59.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:58.807+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:59.813 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:59.750+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:59.751427+0000) 2024-04-27T03:08:59.813 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:59.699+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:59.813 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:59.699+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:59.813 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:59.699+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:08:59.813 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:59.699+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:59.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:59.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:59.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:59.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:59.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:59.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:08:59.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:59.551+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:08:59.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:08:59.756+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:59.757010+0000) 2024-04-27T03:09:00.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:08:59.886+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:08:59.887319+0000) 2024-04-27T03:09:00.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:59.804+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:00.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:59.804+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:00.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:59.804+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:00.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:08:59.805+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:00.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:00.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:00.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:00.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:00.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:00.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:00.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:00.693+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:00.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:00.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:00.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:00.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:00.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:00.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:00.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:00.587+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:00.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:00.758+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:00.759179+0000) 2024-04-27T03:09:00.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:00.753+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:00.754309+0000) 2024-04-27T03:09:00.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:00.765+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:00.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:00.765+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:00.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:00.765+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:00.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:00.765+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:01.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:00.890+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:00.891357+0000) 2024-04-27T03:09:01.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:01.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:01.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:01.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:01.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:01.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:01.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:01.567+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:01.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:01.760+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:01.760824+0000) 2024-04-27T03:09:01.903 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:01.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:01.903 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:01.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:01.904 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:01.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:01.904 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:01.709+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:01.904 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:01.892+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:01.892845+0000) 2024-04-27T03:09:01.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:01.721+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:01.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:01.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:01.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:01.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:01.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:01.723+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:01.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:01.755+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:01.756172+0000) 2024-04-27T03:09:02.901 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:02.683+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:02.901 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:02.683+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:02.901 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:02.683+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:02.901 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:02.683+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:02.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:02.692+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:02.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:02.692+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:02.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:02.692+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:02.902 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:02.694+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:02.902 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:02.755+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:02.756353+0000) 2024-04-27T03:09:02.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:02.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:02.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:02.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:02.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:02.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:02.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:02.616+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:02.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:02.760+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:02.760995+0000) 2024-04-27T03:09:03.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:02.892+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:02.893015+0000) 2024-04-27T03:09:03.901 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:09:03.556+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54085 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:09:03.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:03.555+0000 7f3a20159700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:03.555880+0000) 2024-04-27T03:09:03.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:03.655+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:03.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:03.655+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:03.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:03.655+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:03.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:03.656+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:03.902 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:03.755+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:03.756527+0000) 2024-04-27T03:09:03.902 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:03.554+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:03.555106+0000) 2024-04-27T03:09:03.902 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:03.701+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:03.902 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:03.701+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:03.902 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:03.701+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:03.902 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:03.701+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:03.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:03.554+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:03.555226+0000) 2024-04-27T03:09:03.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:03.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:03.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:03.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:03.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:03.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:03.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:03.600+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:03.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:03.760+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:03.761240+0000) 2024-04-27T03:09:04.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:03.892+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:03.893211+0000) 2024-04-27T03:09:04.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:04.636+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:04.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:04.636+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:04.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:04.636+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:04.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:04.638+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:04.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:04.757+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:04.758793+0000) 2024-04-27T03:09:04.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:04.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:04.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:04.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:04.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:04.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:04.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:04.599+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:04.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:04.764+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:04.765524+0000) 2024-04-27T03:09:04.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:04.728+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:04.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:04.728+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:04.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:04.728+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:04.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:04.728+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:05.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:04.895+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:04.896275+0000) 2024-04-27T03:09:05.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:05.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:05.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:05.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:05.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:05.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:05.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:05.676+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:05.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:05.760+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:05.760981+0000) 2024-04-27T03:09:05.909 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:05.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:05.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:05.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:05.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:05.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:05.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:05.605+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:05.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:05.767+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:05.768312+0000) 2024-04-27T03:09:05.910 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:05.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:05.910 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:05.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:05.910 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:05.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:05.910 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:05.751+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:06.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:05.900+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:05.901073+0000) 2024-04-27T03:09:06.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:06.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:06.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:06.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:06.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:06.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:06.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:06.572+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:06.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:06.768+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:06.769684+0000) 2024-04-27T03:09:06.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:06.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:06.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:06.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:06.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:06.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:06.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:06.719+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:06.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:06.761+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:06.762290+0000) 2024-04-27T03:09:06.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:06.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:06.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:06.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:06.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:06.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:06.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:06.761+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:07.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:06.903+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:06.904599+0000) 2024-04-27T03:09:07.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:07.763+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:07.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:07.763+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:07.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:07.763+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:07.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:07.764+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:07.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:07.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:07.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:07.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:07.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:07.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:07.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:07.676+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:07.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:07.760+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:07.762468+0000) 2024-04-27T03:09:07.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:07.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:07.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:07.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:07.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:07.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:07.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:07.598+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:07.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:07.768+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:07.769911+0000) 2024-04-27T03:09:08.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:07.903+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:07.904794+0000) 2024-04-27T03:09:08.912 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:09:08.561+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54130 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:09:08.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:08.560+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:08.562447+0000) 2024-04-27T03:09:08.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:08.677+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:08.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:08.677+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:08.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:08.677+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:08.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:08.678+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:08.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:08.760+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:08.762701+0000) 2024-04-27T03:09:08.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:08.559+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:08.561512+0000) 2024-04-27T03:09:08.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:08.717+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:08.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:08.717+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:08.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:08.717+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:08.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:08.717+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:08.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:08.560+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:08.562580+0000) 2024-04-27T03:09:08.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:08.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:08.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:08.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:08.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:08.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:08.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:08.630+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:08.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:08.768+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:08.770143+0000) 2024-04-27T03:09:09.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:08.903+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:08.904988+0000) 2024-04-27T03:09:09.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:09.743+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:09.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:09.743+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:09.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:09.743+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:09.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:09.743+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:09.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:09.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:09.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:09.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:09.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:09.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:09.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:09.659+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:09.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:09.762+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:09.764757+0000) 2024-04-27T03:09:09.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:09.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:09.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:09.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:09.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:09.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:09.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:09.623+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:09.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:09.771+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:09.773157+0000) 2024-04-27T03:09:10.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:09.906+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:09.908330+0000) 2024-04-27T03:09:10.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:10.757+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:10.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:10.757+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:10.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:10.757+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:10.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:10.757+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:10.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:10.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:10.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:10.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:10.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:10.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:10.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:10.622+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:10.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:10.765+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:10.767519+0000) 2024-04-27T03:09:10.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:10.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:10.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:10.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:10.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:10.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:10.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:10.665+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:10.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:10.773+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:10.775795+0000) 2024-04-27T03:09:11.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:10.910+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:10.912654+0000) 2024-04-27T03:09:11.886 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:11.616+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:11.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:11.616+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:11.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:11.616+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:11.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:11.618+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:11.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:11.775+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:11.777669+0000) 2024-04-27T03:09:11.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:11.649+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:11.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:11.649+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:11.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:11.649+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:11.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:11.650+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:11.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:11.768+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:11.770034+0000) 2024-04-27T03:09:11.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:11.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:11.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:11.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:11.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:11.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:11.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:11.777+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:12.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:11.913+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:11.915136+0000) 2024-04-27T03:09:12.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:12.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:12.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:12.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:12.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:12.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:12.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:12.678+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:12.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:12.768+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:12.770239+0000) 2024-04-27T03:09:12.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:12.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:12.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:12.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:12.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:12.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:12.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:12.662+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:12.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:12.776+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:12.777845+0000) 2024-04-27T03:09:12.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:12.745+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:12.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:12.745+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:12.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:12.745+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:12.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:12.746+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:12.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:12.913+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:12.915304+0000) 2024-04-27T03:09:13.916 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:09:13.568+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54176 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:09:13.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:13.565+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:13.567542+0000) 2024-04-27T03:09:13.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:13.700+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:13.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:13.700+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:13.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:13.700+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:13.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:13.701+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:13.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:13.567+0000 7f3a20159700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:13.569478+0000) 2024-04-27T03:09:13.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:13.641+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:13.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:13.641+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:13.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:13.641+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:13.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:13.641+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:13.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:13.768+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:13.770436+0000) 2024-04-27T03:09:13.918 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:13.566+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:13.568190+0000) 2024-04-27T03:09:13.918 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:13.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:13.918 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:13.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:13.918 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:13.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:13.918 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:13.664+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:13.918 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:13.776+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:13.778037+0000) 2024-04-27T03:09:14.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:13.913+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:13.915494+0000) 2024-04-27T03:09:14.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:14.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:14.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:14.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:14.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:14.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:14.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:14.675+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:14.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:14.770+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:14.772597+0000) 2024-04-27T03:09:14.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:14.667+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:14.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:14.667+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:14.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:14.667+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:14.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:14.667+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:14.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:14.915+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:14.917599+0000) 2024-04-27T03:09:14.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:14.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:14.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:14.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:14.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:14.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:14.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:14.710+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:14.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:14.780+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:14.782315+0000) 2024-04-27T03:09:15.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:15.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:15.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:15.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:15.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:15.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:15.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:15.754+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:15.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:15.782+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:15.784475+0000) 2024-04-27T03:09:15.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:15.715+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:15.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:15.715+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:15.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:15.715+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:15.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:15.715+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:15.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:15.772+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:15.774775+0000) 2024-04-27T03:09:15.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:15.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:15.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:15.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:15.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:15.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:15.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:15.669+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:15.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:15.917+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:15.919702+0000) 2024-04-27T03:09:16.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:16.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:16.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:16.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:16.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:16.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:16.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:16.728+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:16.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:16.785+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:16.787564+0000) 2024-04-27T03:09:16.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:16.753+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:16.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:16.753+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:16.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:16.753+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:16.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:16.753+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:16.981 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:16.776+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:16.777839+0000) 2024-04-27T03:09:16.981 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:16.697+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:16.981 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:16.697+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:16.981 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:16.697+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:16.982 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:16.698+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:16.982 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:16.920+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:16.922649+0000) 2024-04-27T03:09:18.006 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:17.776+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:17.778018+0000) 2024-04-27T03:09:18.006 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:17.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:18.006 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:17.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:18.006 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:17.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:18.006 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:17.802+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:18.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:17.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:18.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:17.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:18.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:17.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:18.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:17.750+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:18.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:17.785+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:17.787749+0000) 2024-04-27T03:09:18.007 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:17.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:18.007 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:17.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:18.007 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:17.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:18.007 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:17.736+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:18.007 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:17.921+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:17.922896+0000) 2024-04-27T03:09:18.917 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:09:18.574+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54220 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:09:18.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:18.573+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:18.575279+0000) 2024-04-27T03:09:18.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:18.755+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:18.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:18.755+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:18.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:18.755+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:18.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:18.756+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:18.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:18.573+0000 7f3a20159700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:18.575056+0000) 2024-04-27T03:09:18.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:18.776+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:18.778128+0000) 2024-04-27T03:09:18.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:18.818+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:18.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:18.818+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:18.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:18.818+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:18.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:18.819+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:18.918 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:18.572+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:18.574676+0000) 2024-04-27T03:09:18.918 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:18.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:18.918 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:18.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:18.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:18.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:18.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:18.782+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:18.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:18.786+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:18.787882+0000) 2024-04-27T03:09:19.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:18.921+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:18.923067+0000) 2024-04-27T03:09:19.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:19.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:19.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:19.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:19.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:19.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:19.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:19.742+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:19.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:19.790+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:19.792155+0000) 2024-04-27T03:09:19.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:19.778+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:19.780237+0000) 2024-04-27T03:09:19.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:19.804+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:19.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:19.804+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:19.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:19.804+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:19.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:19.805+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:19.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:19.719+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:19.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:19.719+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:19.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:19.719+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:19.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:19.719+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:19.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:19.923+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:19.925747+0000) 2024-04-27T03:09:21.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:20.722+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:21.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:20.722+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:21.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:20.722+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:21.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:20.723+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:21.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:20.928+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:20.928993+0000) 2024-04-27T03:09:21.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:20.781+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:20.782477+0000) 2024-04-27T03:09:21.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:20.825+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:21.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:20.825+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:21.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:20.825+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:21.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:20.826+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:21.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:20.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:21.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:20.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:21.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:20.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:21.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:20.735+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:21.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:20.793+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:20.794556+0000) 2024-04-27T03:09:22.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:21.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:22.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:21.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:22.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:21.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:22.007 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:21.725+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:22.007 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:21.930+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:21.930843+0000) 2024-04-27T03:09:22.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:21.783+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:21.783935+0000) 2024-04-27T03:09:22.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:21.784+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:22.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:21.784+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:22.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:21.784+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:22.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:21.785+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:22.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:21.727+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:22.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:21.727+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:22.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:21.727+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:22.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:21.728+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:22.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:21.795+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:21.796185+0000) 2024-04-27T03:09:22.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:22.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:22.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:22.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:22.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:22.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:22.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:22.754+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:22.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:22.795+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:22.796413+0000) 2024-04-27T03:09:22.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:22.783+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:22.784117+0000) 2024-04-27T03:09:22.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:22.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:22.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:22.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:22.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:22.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:22.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:22.831+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:22.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:22.714+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:22.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:22.714+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:22.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:22.714+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:22.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:22.714+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:22.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:22.930+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:22.931170+0000) 2024-04-27T03:09:23.917 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:09:23.582+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54266 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:09:23.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:23.581+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:23.582619+0000) 2024-04-27T03:09:23.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:23.745+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:23.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:23.745+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:23.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:23.745+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:23.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:23.745+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:23.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:23.581+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:23.582101+0000) 2024-04-27T03:09:23.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:23.781+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:23.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:23.781+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:23.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:23.781+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:23.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:23.782+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:23.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:23.783+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:23.784386+0000) 2024-04-27T03:09:23.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:23.580+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:23.581465+0000) 2024-04-27T03:09:23.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:23.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:23.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:23.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:23.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:23.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:23.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:23.792+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:23.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:23.795+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:23.796635+0000) 2024-04-27T03:09:24.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:23.930+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:23.931413+0000) 2024-04-27T03:09:25.006 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:24.737+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:25.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:24.737+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:25.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:24.737+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:25.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:24.738+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:25.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:24.785+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:24.786466+0000) 2024-04-27T03:09:25.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:24.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:25.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:24.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:25.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:24.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:25.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:24.744+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:25.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:24.800+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:24.801675+0000) 2024-04-27T03:09:25.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:24.769+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:25.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:24.769+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:25.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:24.769+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:25.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:24.770+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:25.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:24.932+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:24.933735+0000) 2024-04-27T03:09:26.006 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:25.730+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:26.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:25.730+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:26.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:25.730+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:26.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:25.730+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:26.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:25.787+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:25.788534+0000) 2024-04-27T03:09:26.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:25.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:26.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:25.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:26.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:25.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:26.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:25.722+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:26.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:25.804+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:25.804913+0000) 2024-04-27T03:09:26.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:25.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:26.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:25.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:26.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:25.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:26.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:25.762+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:26.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:25.935+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:25.935865+0000) 2024-04-27T03:09:27.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:26.733+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:27.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:26.733+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:27.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:26.733+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:27.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:26.734+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:27.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:26.789+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:26.790163+0000) 2024-04-27T03:09:27.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:26.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:27.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:26.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:27.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:26.736+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:27.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:26.736+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:27.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:26.805+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:26.806585+0000) 2024-04-27T03:09:27.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:26.748+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:27.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:26.748+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:27.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:26.748+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:27.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:26.748+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:27.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:26.936+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:26.937260+0000) 2024-04-27T03:09:27.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:27.720+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:27.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:27.720+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:27.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:27.720+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:27.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:27.720+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:27.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:27.936+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:27.937424+0000) 2024-04-27T03:09:27.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:27.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:27.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:27.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:27.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:27.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:27.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:27.719+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:27.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:27.789+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:27.790406+0000) 2024-04-27T03:09:27.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:27.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:27.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:27.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:27.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:27.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:27.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:27.778+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:27.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:27.805+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:27.806752+0000) 2024-04-27T03:09:28.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:09:28.588+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54311 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:09:28.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:28.587+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:28.588140+0000) 2024-04-27T03:09:28.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:28.743+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:28.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:28.743+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:28.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:28.743+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:28.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:28.743+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:28.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:28.588+0000 7f3a20159700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:28.588916+0000) 2024-04-27T03:09:28.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:28.765+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:28.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:28.765+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:28.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:28.765+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:28.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:28.766+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:28.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:28.789+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:28.790585+0000) 2024-04-27T03:09:28.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:28.587+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:28.587931+0000) 2024-04-27T03:09:28.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:28.806+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:28.806940+0000) 2024-04-27T03:09:28.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:28.810+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:28.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:28.810+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:28.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:28.810+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:28.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:28.811+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:29.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:28.936+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:28.937597+0000) 2024-04-27T03:09:30.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:29.784+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:30.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:29.784+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:30.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:29.784+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:30.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:29.785+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:30.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:29.792+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:29.792818+0000) 2024-04-27T03:09:30.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:29.809+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:29.810081+0000) 2024-04-27T03:09:30.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:29.852+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:30.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:29.852+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:30.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:29.852+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:30.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:29.853+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:30.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:29.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:30.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:29.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:30.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:29.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:30.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:29.754+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:30.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:29.938+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:29.939770+0000) 2024-04-27T03:09:31.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:30.756+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:31.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:30.756+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:31.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:30.756+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:31.008 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:30.758+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:31.008 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:30.795+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:30.796021+0000) 2024-04-27T03:09:31.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:30.736+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:31.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:30.736+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:31.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:30.736+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:31.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:30.736+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:31.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:30.941+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:30.942009+0000) 2024-04-27T03:09:31.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:30.812+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:30.813194+0000) 2024-04-27T03:09:31.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:30.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:31.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:30.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:31.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:30.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:31.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:30.868+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:32.007 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:31.728+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:32.007 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:31.728+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:32.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:31.728+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:32.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:31.729+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:32.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:31.942+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:31.943490+0000) 2024-04-27T03:09:32.008 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:31.797+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:31.798622+0000) 2024-04-27T03:09:32.008 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:31.798+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:32.008 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:31.798+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:32.008 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:31.798+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:32.008 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:31.799+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:32.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:31.814+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:31.814839+0000) 2024-04-27T03:09:32.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:31.879+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:32.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:31.879+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:32.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:31.879+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:32.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:31.879+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:32.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:32.814+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:32.815020+0000) 2024-04-27T03:09:32.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:32.904+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:32.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:32.904+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:32.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:32.904+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:32.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:32.905+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:32.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:32.797+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:32.798740+0000) 2024-04-27T03:09:32.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:32.828+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:32.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:32.828+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:32.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:32.828+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:32.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:32.829+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:32.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:32.770+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:32.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:32.770+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:32.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:32.770+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:32.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:32.771+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:32.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:32.942+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:32.943675+0000) 2024-04-27T03:09:33.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:09:33.596+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54357 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:09:33.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:33.594+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:33.594830+0000) 2024-04-27T03:09:33.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:33.814+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:33.815176+0000) 2024-04-27T03:09:33.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:33.593+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:33.594647+0000) 2024-04-27T03:09:33.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:33.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:33.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:33.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:33.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:33.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:33.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:33.798+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:33.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:33.594+0000 7f3a20159700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:33.595605+0000) 2024-04-27T03:09:33.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:33.798+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:33.798923+0000) 2024-04-27T03:09:33.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:33.811+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:33.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:33.811+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:33.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:33.811+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:33.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:33.812+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:34.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:33.943+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:33.943853+0000) 2024-04-27T03:09:34.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:33.946+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:34.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:33.946+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:34.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:33.946+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:34.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:33.947+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:35.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:34.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:35.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:34.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:35.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:34.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:35.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:34.801+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:35.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:34.802+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:34.803140+0000) 2024-04-27T03:09:35.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:34.816+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:34.817322+0000) 2024-04-27T03:09:35.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:34.918+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:35.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:34.918+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:35.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:34.918+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:35.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:34.919+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:35.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:34.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:35.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:34.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:35.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:34.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:35.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:34.799+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:35.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:34.946+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:34.947005+0000) 2024-04-27T03:09:36.008 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:35.804+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:35.805282+0000) 2024-04-27T03:09:36.008 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:35.846+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:36.008 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:35.846+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:36.008 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:35.846+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:36.008 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:35.846+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:36.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:35.820+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:35.821627+0000) 2024-04-27T03:09:36.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:35.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:36.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:35.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:36.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:35.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:36.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:35.926+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:36.009 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:35.786+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:36.009 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:35.786+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:36.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:35.786+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:36.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:35.786+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:36.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:35.948+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:35.949636+0000) 2024-04-27T03:09:37.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:36.822+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:36.823157+0000) 2024-04-27T03:09:37.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:36.937+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:37.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:36.937+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:37.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:36.937+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:37.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:36.939+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:37.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:36.805+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:36.806733+0000) 2024-04-27T03:09:37.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:36.807+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:37.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:36.807+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:37.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:36.807+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:37.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:36.808+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:37.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:36.768+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:37.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:36.768+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:37.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:36.768+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:37.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:36.768+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:37.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:36.950+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:36.951542+0000) 2024-04-27T03:09:38.008 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:37.775+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:38.009 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:37.775+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:38.009 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:37.775+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:38.009 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:37.776+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:38.009 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:37.806+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:37.806948+0000) 2024-04-27T03:09:38.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:37.732+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:38.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:37.732+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:38.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:37.732+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:38.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:37.732+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:38.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:37.950+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:37.951715+0000) 2024-04-27T03:09:38.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:37.822+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:37.823326+0000) 2024-04-27T03:09:38.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:37.921+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:38.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:37.921+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:38.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:37.921+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:38.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:37.922+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:38.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:09:38.602+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54401 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:09:38.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:38.599+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:38.600664+0000) 2024-04-27T03:09:38.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:38.822+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:38.823490+0000) 2024-04-27T03:09:38.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:38.600+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:38.601703+0000) 2024-04-27T03:09:38.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:38.749+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:38.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:38.749+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:38.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:38.749+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:38.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:38.749+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:38.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:38.602+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:38.603504+0000) 2024-04-27T03:09:38.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:38.788+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:38.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:38.788+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:38.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:38.788+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:38.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:38.789+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:38.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:38.806+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:38.807104+0000) 2024-04-27T03:09:39.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:38.951+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:38.951840+0000) 2024-04-27T03:09:39.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:38.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:39.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:38.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:39.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:38.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:39.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:38.925+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:40.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:39.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:40.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:39.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:40.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:39.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:40.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:39.788+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:40.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:39.953+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:39.954596+0000) 2024-04-27T03:09:40.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:39.794+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:40.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:39.794+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:40.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:39.794+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:40.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:39.795+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:40.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:39.811+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:39.811840+0000) 2024-04-27T03:09:40.077 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:39.827+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:39.828010+0000) 2024-04-27T03:09:40.077 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:39.897+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:40.077 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:39.897+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:40.077 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:39.897+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:40.077 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:39.898+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:41.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:40.830+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:40.831129+0000) 2024-04-27T03:09:41.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:40.904+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:41.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:40.904+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:41.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:40.904+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:41.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:40.906+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:41.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:40.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:41.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:40.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:41.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:40.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:41.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:40.769+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:41.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:40.814+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:40.815052+0000) 2024-04-27T03:09:41.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:40.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:41.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:40.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:41.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:40.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:41.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:40.835+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:41.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:40.956+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:40.957354+0000) 2024-04-27T03:09:42.008 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:41.723+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:42.009 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:41.723+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:42.009 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:41.723+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:42.009 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:41.725+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:42.009 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:41.815+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:41.816409+0000) 2024-04-27T03:09:42.009 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:41.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:42.009 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:41.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:42.009 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:41.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:42.009 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:41.819+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:42.009 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:41.957+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:41.958733+0000) 2024-04-27T03:09:42.010 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:41.831+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:41.832636+0000) 2024-04-27T03:09:42.010 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:41.863+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:42.010 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:41.863+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:42.010 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:41.863+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:42.010 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:41.863+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:42.983 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:42.721+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:42.983 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:42.721+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:42.983 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:42.721+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:42.983 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:42.722+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:42.983 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:42.815+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:42.816598+0000) 2024-04-27T03:09:42.983 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:42.775+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:42.983 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:42.775+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:42.984 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:42.775+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:42.984 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:42.775+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:42.984 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:42.958+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:42.958917+0000) 2024-04-27T03:09:42.984 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:42.832+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:42.832827+0000) 2024-04-27T03:09:42.984 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:42.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:42.984 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:42.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:42.984 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:42.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:05.913510+0000 (oldest deadline 2024-04-27T02:59:25.913510+0000) 2024-04-27T03:09:42.984 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:42.851+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:43.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:09:43.609+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54442 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:09:43.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:43.607+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:43.608073+0000) 2024-04-27T03:09:43.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:43.815+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:43.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:43.832+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:43.832991+0000) 2024-04-27T03:09:43.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:43.606+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:43.607422+0000) 2024-04-27T03:09:43.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:43.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:43.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:43.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:43.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:43.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:43.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:43.676+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:43.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:43.815+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:43.816770+0000) 2024-04-27T03:09:43.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:43.607+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:43.608555+0000) 2024-04-27T03:09:43.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:43.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:43.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:43.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:43.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:43.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:12.747561+0000 (oldest deadline 2024-04-27T02:59:32.747561+0000) 2024-04-27T03:09:43.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:43.822+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:44.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:43.958+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:43.959073+0000) 2024-04-27T03:09:44.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:44.826+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:44.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:44.836+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:44.837334+0000) 2024-04-27T03:09:44.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:44.859+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:44.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:44.657+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:44.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:44.657+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:44.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:44.657+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:44.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:44.658+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:44.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:44.818+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:44.818993+0000) 2024-04-27T03:09:45.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:44.960+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:44.961353+0000) 2024-04-27T03:09:45.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:45.804+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:45.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:45.831+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:45.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:45.838+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:45.839405+0000) 2024-04-27T03:09:45.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:45.623+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:45.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:45.623+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:45.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:45.623+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:45.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:45.624+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:45.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:45.820+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:45.821102+0000) 2024-04-27T03:09:46.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:45.962+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:45.963403+0000) 2024-04-27T03:09:46.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:46.804+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:46.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:46.788+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:46.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:46.840+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:46.840918+0000) 2024-04-27T03:09:46.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:46.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:46.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:46.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:46.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:46.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:46.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:46.621+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:46.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:46.821+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:46.822441+0000) 2024-04-27T03:09:47.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:46.965+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:46.966485+0000) 2024-04-27T03:09:47.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:47.782+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:47.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:47.807+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:47.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:47.840+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:47.841070+0000) 2024-04-27T03:09:47.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:47.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:47.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:47.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:47.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:47.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:47.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:47.628+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:47.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:47.821+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:47.822632+0000) 2024-04-27T03:09:48.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:47.965+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:47.966640+0000) 2024-04-27T03:09:48.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:09:48.616+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54480 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:09:48.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:48.614+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:48.614841+0000) 2024-04-27T03:09:48.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:48.789+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:48.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:48.615+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:48.615935+0000) 2024-04-27T03:09:48.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:48.840+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:48.841236+0000) 2024-04-27T03:09:48.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:48.857+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:48.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:48.613+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:48.614597+0000) 2024-04-27T03:09:48.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:48.648+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:48.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:48.648+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:48.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:48.648+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:48.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:48.649+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:48.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:48.822+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:48.822820+0000) 2024-04-27T03:09:49.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:48.966+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:48.966814+0000) 2024-04-27T03:09:49.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:49.835+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:49.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:49.842+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:49.843441+0000) 2024-04-27T03:09:49.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:49.775+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:49.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:49.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:49.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:49.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:49.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:49.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T02:59:18.910471+0000 (oldest deadline 2024-04-27T02:59:38.910471+0000) 2024-04-27T03:09:49.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:49.678+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:49.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:49.824+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:49.824845+0000) 2024-04-27T03:09:50.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:49.968+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:49.969042+0000) 2024-04-27T03:09:50.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:50.846+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:50.847762+0000) 2024-04-27T03:09:50.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:50.885+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:50.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:50.737+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:50.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:50.678+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:50.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:50.826+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:50.826899+0000) 2024-04-27T03:09:51.326 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:50.972+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:50.973124+0000) 2024-04-27T03:09:51.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:51.743+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:51.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:51.703+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:51.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:51.828+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:51.828819+0000) 2024-04-27T03:09:51.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:51.842+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:51.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:51.849+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:51.850542+0000) 2024-04-27T03:09:52.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:51.975+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:51.976383+0000) 2024-04-27T03:09:52.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:52.759+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:52.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:52.688+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:52.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:52.828+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:52.828998+0000) 2024-04-27T03:09:52.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:52.829+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:52.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:52.849+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:52.850743+0000) 2024-04-27T03:09:53.326 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:52.975+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:52.976632+0000) 2024-04-27T03:09:53.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:09:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:09:53.623+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54520 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:09:53.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:53.621+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:53.622082+0000) 2024-04-27T03:09:53.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:53.812+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:53.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:53.850+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:53.850927+0000) 2024-04-27T03:09:53.922 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:53.621+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:53.622081+0000) 2024-04-27T03:09:53.922 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:53.730+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:53.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:53.622+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:53.623611+0000) 2024-04-27T03:09:53.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:53.706+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:53.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:53.828+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:53.829206+0000) 2024-04-27T03:09:54.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:53.976+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:53.976824+0000) 2024-04-27T03:09:54.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:54.735+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:54.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:54.677+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:54.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:54.831+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:54.831860+0000) 2024-04-27T03:09:54.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:54.814+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:54.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:54.854+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:54.855248+0000) 2024-04-27T03:09:55.329 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:54.980+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:54.981018+0000) 2024-04-27T03:09:55.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:55.785+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:55.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:55.857+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:55.857899+0000) 2024-04-27T03:09:55.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:55.743+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:55.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:55.708+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:55.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:55.834+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:55.835038+0000) 2024-04-27T03:09:56.329 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:55.982+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:55.983125+0000) 2024-04-27T03:09:56.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:56.702+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:57.003 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:56.836+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:56.836979+0000) 2024-04-27T03:09:57.003 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:56.741+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:57.004 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:56.858+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:56.859790+0000) 2024-04-27T03:09:57.004 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:56.694+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:57.254 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:56.983+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:56.984608+0000) 2024-04-27T03:09:57.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:57.672+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:57.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:57.752+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:57.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:57.836+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:57.837182+0000) 2024-04-27T03:09:57.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:57.726+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:57.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:57.859+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:57.859987+0000) 2024-04-27T03:09:58.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:57.983+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:57.984783+0000) 2024-04-27T03:09:58.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:09:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:09:58.630+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54558 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:09:58.922 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:58.628+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:58.629092+0000) 2024-04-27T03:09:58.922 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:58.647+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:58.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:58.629+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:58.630389+0000) 2024-04-27T03:09:58.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:58.795+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:58.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:58.836+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:58.837447+0000) 2024-04-27T03:09:58.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:58.628+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:58.629557+0000) 2024-04-27T03:09:58.923 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:58.761+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:58.923 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:58.859+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:58.860165+0000) 2024-04-27T03:09:59.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:58.984+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:58.984981+0000) 2024-04-27T03:09:59.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:59.642+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:59.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:59.830+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:59.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:09:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:09:59.841+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:59.842026+0000) 2024-04-27T03:09:59.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:59.783+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:09:59.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:09:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:09:59.861+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:59.862241+0000) 2024-04-27T03:10:00.239 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:09:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:09:59.988+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:09:59.989009+0000) 2024-04-27T03:10:00.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:00.692+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:00.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:00.845+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:00.846763+0000) 2024-04-27T03:10:00.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:00.856+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:00.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:00.786+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:00.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:00.863+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:00.864519+0000) 2024-04-27T03:10:01.332 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:00.990+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:00.991708+0000) 2024-04-27T03:10:02.001 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:01.729+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:02.002 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:01.848+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:01.849530+0000) 2024-04-27T03:10:02.002 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:01.904+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:02.002 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:01.794+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:02.002 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:01.865+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:01.866130+0000) 2024-04-27T03:10:02.332 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:01.992+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:01.993305+0000) 2024-04-27T03:10:02.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:02.712+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:02.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:02.848+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:02.849756+0000) 2024-04-27T03:10:02.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:02.895+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:02.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:02.800+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:02.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:02.865+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:02.866447+0000) 2024-04-27T03:10:03.332 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:02.992+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:02.993535+0000) 2024-04-27T03:10:03.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:10:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:10:03.638+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54598 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:10:03.922 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:03.636+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:03.636868+0000) 2024-04-27T03:10:03.922 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:03.682+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:03.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:03.636+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:03.637550+0000) 2024-04-27T03:10:03.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:03.847+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:03.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:03.849+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:03.849968+0000) 2024-04-27T03:10:03.923 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:03.636+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:03.637710+0000) 2024-04-27T03:10:03.923 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:03.816+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:03.923 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:03.865+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:03.866648+0000) 2024-04-27T03:10:04.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:03.992+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:03.993741+0000) 2024-04-27T03:10:04.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:04.701+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:04.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:04.851+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:04.852062+0000) 2024-04-27T03:10:04.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:04.889+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:04.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:04.819+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:04.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:04.868+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:04.868799+0000) 2024-04-27T03:10:05.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:04.997+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:04.998785+0000) 2024-04-27T03:10:05.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:05.773+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:05.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:05.870+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:05.870951+0000) 2024-04-27T03:10:05.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:05.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:05.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:05.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:05.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:05.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:05.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:05.694+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:05.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:05.855+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:05.855844+0000) 2024-04-27T03:10:05.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:05.857+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:06.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:06.003+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:06.003825+0000) 2024-04-27T03:10:06.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:06.703+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:06.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:06.703+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:06.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:06.703+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:06.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:06.704+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:06.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:06.850+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:06.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:06.856+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:06.857284+0000) 2024-04-27T03:10:06.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:06.784+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:06.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:06.784+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:06.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:06.784+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:06.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:06.784+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:06.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:06.871+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:06.872362+0000) 2024-04-27T03:10:07.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:07.006+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:07.007175+0000) 2024-04-27T03:10:07.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:07.827+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:07.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:07.856+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:07.857462+0000) 2024-04-27T03:10:07.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:07.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:07.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:07.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:07.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:07.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:07.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:07.665+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:07.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:07.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:07.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:07.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:07.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:07.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:07.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:07.773+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:07.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:07.871+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:07.872544+0000) 2024-04-27T03:10:08.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:08.006+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:08.007403+0000) 2024-04-27T03:10:08.923 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:10:08.644+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54637 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:10:08.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:08.644+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:08.644894+0000) 2024-04-27T03:10:08.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:08.835+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:08.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:08.856+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:08.857650+0000) 2024-04-27T03:10:08.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:08.635+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:08.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:08.635+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:08.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:08.635+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:08.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:08.636+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:08.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:08.641+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:08.642743+0000) 2024-04-27T03:10:08.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:08.644+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:08.644886+0000) 2024-04-27T03:10:08.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:08.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:08.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:08.808+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:08.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:08.808+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:08.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:08.808+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:08.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:08.871+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:08.872732+0000) 2024-04-27T03:10:09.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:09.006+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:09.007648+0000) 2024-04-27T03:10:09.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:09.790+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:09.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:09.860+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:09.860881+0000) 2024-04-27T03:10:09.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:09.650+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:09.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:09.650+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:09.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:09.650+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:09.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:09.651+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:09.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:09.824+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:09.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:09.824+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:09.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:09.824+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:09.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:09.825+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:09.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:09.876+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:09.877009+0000) 2024-04-27T03:10:10.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:10.009+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:10.010787+0000) 2024-04-27T03:10:10.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:10.817+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:10.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:10.862+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:10.862980+0000) 2024-04-27T03:10:10.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:10.872+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:10.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:10.872+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:10.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:10.872+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:10.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:10.874+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:10.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:10.878+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:10.879672+0000) 2024-04-27T03:10:10.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:10.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:10.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:10.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:10.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:10.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:10.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:10.670+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:11.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:11.012+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:11.013514+0000) 2024-04-27T03:10:11.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:11.788+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:11.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:11.863+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:11.864428+0000) 2024-04-27T03:10:11.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:11.676+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:11.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:11.676+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:11.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:11.676+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:11.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:11.677+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:11.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:11.880+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:11.881706+0000) 2024-04-27T03:10:11.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:11.920+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:11.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:11.920+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:11.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:11.920+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:11.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:11.922+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:12.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:12.014+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:12.015099+0000) 2024-04-27T03:10:12.872 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:12.765+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:12.873 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:12.700+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:12.873 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:12.700+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:12.873 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:12.700+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:12.873 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:12.700+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:13.123 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:12.863+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:12.864619+0000) 2024-04-27T03:10:13.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:13.014+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:13.015282+0000) 2024-04-27T03:10:13.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:12.881+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:12.881821+0000) 2024-04-27T03:10:13.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:12.946+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:13.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:12.946+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:13.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:12.946+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:13.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:12.948+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:13.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:10:13.650+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54676 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:10:13.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:13.649+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:13.651070+0000) 2024-04-27T03:10:13.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:13.691+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:13.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:13.691+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:13.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:13.691+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:13.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:13.692+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:13.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:13.649+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:13.651299+0000) 2024-04-27T03:10:13.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:13.812+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:13.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:13.812+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:13.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:13.812+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:13.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:13.813+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:13.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:13.863+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:13.864818+0000) 2024-04-27T03:10:13.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:13.648+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:13.650314+0000) 2024-04-27T03:10:13.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:13.880+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:13.882005+0000) 2024-04-27T03:10:14.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:14.013+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:14.015463+0000) 2024-04-27T03:10:14.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:13.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:14.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:13.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:14.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:13.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:14.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:13.909+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:15.012 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:14.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:15.012 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:14.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:15.012 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:14.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:15.012 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:14.884+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:14.886644+0000) 2024-04-27T03:10:15.013 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:14.885+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:15.013 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:14.778+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:15.013 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:14.778+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:15.013 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:14.778+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:15.013 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:14.780+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:15.013 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:14.868+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:14.869821+0000) 2024-04-27T03:10:15.013 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:14.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:15.013 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:14.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:15.014 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:14.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:15.014 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:14.733+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:15.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:15.015+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:15.017629+0000) 2024-04-27T03:10:15.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:15.699+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:15.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:15.699+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:15.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:15.699+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:15.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:15.699+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:15.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:15.753+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:15.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:15.753+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:15.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:15.753+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:15.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:15.755+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:15.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:15.871+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:15.873002+0000) 2024-04-27T03:10:15.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:15.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:15.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:15.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:15.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:15.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:15.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:15.851+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:15.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:15.887+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:15.889445+0000) 2024-04-27T03:10:16.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:16.018+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:16.019943+0000) 2024-04-27T03:10:16.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:16.889+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:16.891577+0000) 2024-04-27T03:10:16.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:16.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:16.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:16.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:16.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:16.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:16.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:16.890+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:16.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:16.653+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:16.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:16.653+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:16.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:16.653+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:16.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:16.653+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:16.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:16.787+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:16.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:16.787+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:16.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:16.787+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:16.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:16.788+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:16.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:16.872+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:16.874560+0000) 2024-04-27T03:10:17.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:17.019+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:17.021410+0000) 2024-04-27T03:10:17.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:17.689+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:17.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:17.689+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:17.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:17.689+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:17.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:17.690+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:17.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:17.805+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:17.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:17.805+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:17.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:17.805+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:17.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:17.806+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:17.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:17.872+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:17.874739+0000) 2024-04-27T03:10:17.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:17.889+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:17.891687+0000) 2024-04-27T03:10:17.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:17.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:17.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:17.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:17.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:17.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:17.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:17.893+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:18.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:18.019+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:18.021583+0000) 2024-04-27T03:10:18.924 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:10:18.657+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54715 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:10:18.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:18.654+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:18.656058+0000) 2024-04-27T03:10:18.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:18.708+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:18.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:18.708+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:18.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:18.708+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:18.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:18.709+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:18.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:18.654+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:18.656663+0000) 2024-04-27T03:10:18.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:18.832+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:18.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:18.832+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:18.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:18.832+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:18.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:18.832+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:18.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:18.873+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:18.874925+0000) 2024-04-27T03:10:18.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:18.655+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:18.657462+0000) 2024-04-27T03:10:18.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:18.855+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:18.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:18.855+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:18.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:18.855+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:18.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:18.856+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:18.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:18.890+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:18.891860+0000) 2024-04-27T03:10:19.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:19.019+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:19.021773+0000) 2024-04-27T03:10:19.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:19.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:19.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:19.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:19.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:19.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:19.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:19.669+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:19.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:19.864+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:19.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:19.864+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:19.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:19.864+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:19.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:19.865+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:19.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:19.875+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:19.877115+0000) 2024-04-27T03:10:19.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:19.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:19.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:19.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:19.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:19.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:19.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:19.891+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:19.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:19.894+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:19.896122+0000) 2024-04-27T03:10:20.347 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:20.022+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:20.023942+0000) 2024-04-27T03:10:20.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:20.898+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:20.900432+0000) 2024-04-27T03:10:20.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:20.879+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:20.881329+0000) 2024-04-27T03:10:20.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:20.898+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:20.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:20.898+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:20.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:20.898+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:20.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:20.898+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:20.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:20.666+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:20.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:20.666+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:20.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:20.666+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:20.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:20.667+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:21.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:21.026+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:21.028141+0000) 2024-04-27T03:10:21.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:20.920+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:21.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:20.920+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:21.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:20.920+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:21.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:20.921+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:21.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:21.713+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:21.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:21.713+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:21.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:21.714+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:21.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:21.714+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:21.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:21.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:21.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:21.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:21.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:21.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:21.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:21.892+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:21.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:21.900+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:21.902537+0000) 2024-04-27T03:10:21.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:21.853+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:21.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:21.853+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:21.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:21.853+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:21.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:21.854+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:21.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:21.880+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:21.882791+0000) 2024-04-27T03:10:22.347 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:22.029+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:22.031610+0000) 2024-04-27T03:10:22.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:22.684+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:22.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:22.684+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:22.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:22.684+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:22.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:22.684+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:22.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:22.817+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:22.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:22.817+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:22.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:22.817+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:22.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:22.818+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:22.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:22.881+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:22.882970+0000) 2024-04-27T03:10:22.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:22.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:22.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:22.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:22.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:22.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:22.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:22.852+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:22.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:22.900+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:22.902783+0000) 2024-04-27T03:10:23.347 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:23.030+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:23.031842+0000) 2024-04-27T03:10:23.924 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:10:23.664+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54754 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:10:23.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:23.662+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:23.664316+0000) 2024-04-27T03:10:23.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:23.663+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:23.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:23.663+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:23.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:23.663+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:23.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:23.664+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:23.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:23.662+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:23.664156+0000) 2024-04-27T03:10:23.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:23.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:23.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:23.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:23.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:23.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:23.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:23.787+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:23.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:23.881+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:23.883143+0000) 2024-04-27T03:10:23.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:23.662+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:23.664028+0000) 2024-04-27T03:10:23.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:23.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:23.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:23.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:23.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:23.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:23.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:23.818+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:23.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:23.901+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:23.902982+0000) 2024-04-27T03:10:24.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:24.030+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:24.031998+0000) 2024-04-27T03:10:24.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:24.670+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:24.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:24.670+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:24.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:24.670+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:24.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:24.670+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:24.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:24.781+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:24.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:24.781+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:24.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:24.781+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:24.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:24.782+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:24.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:24.886+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:24.887893+0000) 2024-04-27T03:10:24.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:24.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:24.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:24.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:24.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:24.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:24.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:24.807+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:24.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:24.903+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:24.905226+0000) 2024-04-27T03:10:25.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:25.032+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:25.034286+0000) 2024-04-27T03:10:25.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:25.792+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:25.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:25.792+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:25.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:25.792+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:25.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:25.793+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:25.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:25.890+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:25.892175+0000) 2024-04-27T03:10:25.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:25.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:25.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:25.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:25.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:25.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:25.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:25.845+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:25.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:25.908+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:25.910090+0000) 2024-04-27T03:10:25.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:25.675+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:25.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:25.675+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:25.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:25.675+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:25.981 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:25.675+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:26.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:26.035+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:26.036532+0000) 2024-04-27T03:10:26.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:26.717+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:26.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:26.717+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:26.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:26.717+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:26.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:26.717+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:26.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:26.778+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:26.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:26.778+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:26.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:26.778+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:26.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:26.779+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:26.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:26.893+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:26.894214+0000) 2024-04-27T03:10:26.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:26.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:26.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:26.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:26.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:26.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:26.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:26.805+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:26.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:26.912+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:26.913006+0000) 2024-04-27T03:10:27.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:27.037+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:27.038346+0000) 2024-04-27T03:10:27.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:27.808+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:27.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:27.808+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:27.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:27.808+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:27.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:27.809+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:27.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:27.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:27.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:27.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:27.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:27.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:27.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:27.826+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:27.904 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:27.725+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:27.904 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:27.725+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:27.904 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:27.725+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:27.904 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:27.726+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:28.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:28.037+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:28.038675+0000) 2024-04-27T03:10:28.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:27.893+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:27.894549+0000) 2024-04-27T03:10:28.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:27.912+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:27.913203+0000) 2024-04-27T03:10:28.979 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:10:28.672+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54793 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:10:28.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:28.672+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:28.673560+0000) 2024-04-27T03:10:28.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:28.845+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:28.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:28.845+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:28.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:28.845+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:28.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:28.846+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:28.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:28.894+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:28.894835+0000) 2024-04-27T03:10:28.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:28.670+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:28.670959+0000) 2024-04-27T03:10:28.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:28.687+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:28.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:28.687+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:28.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:28.687+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:28.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:28.687+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:28.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:28.670+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:28.671384+0000) 2024-04-27T03:10:28.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:28.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:28.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:28.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:28.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:28.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:28.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:28.830+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:28.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:28.912+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:28.913377+0000) 2024-04-27T03:10:29.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:29.038+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:29.038893+0000) 2024-04-27T03:10:30.014 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:29.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:30.014 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:29.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:30.014 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:29.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:30.014 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:29.839+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:30.014 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:29.914+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:29.915667+0000) 2024-04-27T03:10:30.014 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:29.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:30.015 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:29.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:30.015 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:29.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:30.015 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:29.734+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:30.015 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:29.891+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:30.015 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:29.891+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:30.015 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:29.891+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:30.015 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:29.892+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:30.015 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:29.897+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:29.898012+0000) 2024-04-27T03:10:30.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:30.040+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:30.041572+0000) 2024-04-27T03:10:31.014 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:30.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:31.014 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:30.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:31.014 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:30.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:31.014 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:30.724+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:31.015 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:30.847+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:31.015 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:30.847+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:31.015 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:30.847+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:31.015 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:30.848+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:31.015 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:30.917+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:30.917812+0000) 2024-04-27T03:10:31.015 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:30.848+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:31.015 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:30.848+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:31.015 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:30.848+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:31.015 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:30.848+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:31.016 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:30.899+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:30.900255+0000) 2024-04-27T03:10:31.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:31.045+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:31.046267+0000) 2024-04-27T03:10:32.058 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:31.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:32.058 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:31.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:32.058 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:31.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:32.058 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:31.888+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:32.058 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:31.900+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:31.901731+0000) 2024-04-27T03:10:32.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:31.803+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:32.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:31.803+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:32.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:31.803+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:32.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:31.804+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:32.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:31.918+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:31.919270+0000) 2024-04-27T03:10:32.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:31.765+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:32.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:31.765+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:32.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:31.765+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:32.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:31.765+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:32.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:32.048+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:32.049552+0000) 2024-04-27T03:10:32.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:32.808+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:32.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:32.808+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:32.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:32.808+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:32.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:32.809+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:32.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:32.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:32.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:32.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:32.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:32.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:32.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:32.807+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:32.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:32.877+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:32.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:32.877+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:32.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:32.877+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:32.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:32.878+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:32.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:32.901+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:32.901950+0000) 2024-04-27T03:10:33.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:33.049+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:33.049985+0000) 2024-04-27T03:10:33.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:32.918+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:32.919434+0000) 2024-04-27T03:10:33.979 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:10:33.679+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54831 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:10:33.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:33.676+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:33.677202+0000) 2024-04-27T03:10:33.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:33.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:33.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:33.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:33.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:33.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:33.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:33.784+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:33.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:33.677+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:33.678581+0000) 2024-04-27T03:10:33.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:33.901+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:33.902112+0000) 2024-04-27T03:10:33.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:33.919+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:33.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:33.919+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:33.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:33.919+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:33.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:33.920+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:33.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:33.678+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:33.679541+0000) 2024-04-27T03:10:33.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:33.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:33.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:33.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:33.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:33.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:33.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:33.822+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:33.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:33.918+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:33.919684+0000) 2024-04-27T03:10:34.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:34.049+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:34.050167+0000) 2024-04-27T03:10:35.061 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:34.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:35.061 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:34.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:35.062 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:34.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:35.062 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:34.785+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:35.062 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:34.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:35.062 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:34.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:35.062 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:34.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:35.062 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:34.782+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:35.062 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:34.922+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:34.922940+0000) 2024-04-27T03:10:35.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:34.903+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:34.904754+0000) 2024-04-27T03:10:35.063 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:34.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:35.063 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:34.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:35.063 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:34.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:35.063 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:34.932+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:35.356 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:35.052+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:35.053205+0000) 2024-04-27T03:10:36.014 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:35.907+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:35.908027+0000) 2024-04-27T03:10:36.015 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:35.982+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:36.015 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:35.982+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:36.015 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:35.982+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:36.015 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:35.982+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:36.015 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:35.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:36.015 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:35.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:36.015 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:35.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:36.015 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:35.750+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:36.015 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:35.924+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:35.925736+0000) 2024-04-27T03:10:36.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:35.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:36.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:35.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:36.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:35.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:36.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:35.788+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:36.356 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:36.055+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:36.056103+0000) 2024-04-27T03:10:37.015 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:36.909+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:36.910099+0000) 2024-04-27T03:10:37.015 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:36.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:37.015 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:36.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:37.015 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:36.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:37.015 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:36.799+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:37.015 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:36.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:37.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:36.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:37.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:36.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:37.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:36.741+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:37.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:36.927+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:36.928697+0000) 2024-04-27T03:10:37.356 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:37.057+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:37.057972+0000) 2024-04-27T03:10:37.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:37.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:37.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:37.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:37.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:37.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:37.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:37.007+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:37.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:37.909+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:37.910283+0000) 2024-04-27T03:10:37.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:37.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:37.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:37.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:37.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:37.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:37.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:37.822+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:37.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:37.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:37.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:37.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:37.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:37.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:37.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:37.771+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:37.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:37.928+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:37.928900+0000) 2024-04-27T03:10:38.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:38.057+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:38.058125+0000) 2024-04-27T03:10:38.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:38.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:38.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:38.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:38.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:38.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:38.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:38.005+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:38.979 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:10:38.685+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54863 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:10:38.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:38.684+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:38.685614+0000) 2024-04-27T03:10:38.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:38.909+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:38.910457+0000) 2024-04-27T03:10:38.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:38.683+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:38.684690+0000) 2024-04-27T03:10:38.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:38.765+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:38.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:38.765+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:38.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:38.765+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:38.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:38.766+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:38.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:38.928+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:38.929076+0000) 2024-04-27T03:10:38.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:38.685+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:38.685997+0000) 2024-04-27T03:10:38.981 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:38.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:38.981 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:38.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:38.981 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:38.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:38.981 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:38.814+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:39.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:39.057+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:39.058282+0000) 2024-04-27T03:10:39.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:39.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:39.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:39.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:39.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:39.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:39.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:39.003+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:40.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:39.911+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:39.912554+0000) 2024-04-27T03:10:40.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:40.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:40.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:40.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:40.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:40.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:40.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:40.015+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:40.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:39.825+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:40.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:39.825+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:40.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:39.825+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:40.072 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:39.825+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:40.073 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:39.789+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:40.073 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:39.789+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:40.073 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:39.789+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:40.073 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:39.789+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:40.073 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:39.930+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:39.931290+0000) 2024-04-27T03:10:40.361 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:40.062+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:40.063291+0000) 2024-04-27T03:10:41.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:40.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:41.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:40.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:41.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:40.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:41.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:40.785+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:41.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:40.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:41.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:40.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:41.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:40.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:41.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:40.757+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:41.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:40.933+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:40.934405+0000) 2024-04-27T03:10:41.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:40.914+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:40.914841+0000) 2024-04-27T03:10:41.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:40.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:41.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:40.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:41.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:40.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:41.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:40.987+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:41.361 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:41.065+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:41.066412+0000) 2024-04-27T03:10:42.015 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:41.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:42.015 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:41.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:42.015 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:41.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:42.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:41.785+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:42.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:41.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:42.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:41.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:42.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:41.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:42.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:41.726+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:42.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:41.937+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:41.937841+0000) 2024-04-27T03:10:42.016 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:41.916+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:41.916846+0000) 2024-04-27T03:10:42.016 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:41.944+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:42.017 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:41.944+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:42.017 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:41.944+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:42.017 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:41.945+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:42.361 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:42.068+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:42.068848+0000) 2024-04-27T03:10:42.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:42.916+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:42.917043+0000) 2024-04-27T03:10:42.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:42.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:42.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:42.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:42.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:42.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:42.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:42.828+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:42.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:42.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:42.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:42.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:42.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:42.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:42.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:42.718+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:42.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:42.937+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:42.938008+0000) 2024-04-27T03:10:43.335 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:43.068+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:43.069071+0000) 2024-04-27T03:10:43.335 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:42.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:43.335 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:42.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:43.335 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:42.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:43.335 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:42.993+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:43.946 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:10:43.693+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54902 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:10:43.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:43.691+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:43.692026+0000) 2024-04-27T03:10:43.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:43.916+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:43.917289+0000) 2024-04-27T03:10:43.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:43.690+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:43.691567+0000) 2024-04-27T03:10:43.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:43.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:43.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:43.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:43.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:43.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:43.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:43.819+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:43.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:43.690+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:43.691166+0000) 2024-04-27T03:10:43.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:43.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:43.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:43.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:43.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:43.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:43.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:43.694+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:43.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:43.937+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:43.938194+0000) 2024-04-27T03:10:44.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:44.068+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:44.069313+0000) 2024-04-27T03:10:44.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:44.027+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:44.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:44.027+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:44.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:44.027+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:44.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:44.027+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:44.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:44.919+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:44.920232+0000) 2024-04-27T03:10:44.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:44.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:44.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:44.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:44.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:44.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:44.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:44.864+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:44.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:44.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:44.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:44.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:44.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:44.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:44.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:44.692+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:44.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:44.940+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:44.941490+0000) 2024-04-27T03:10:45.364 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:45.073+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:45.073994+0000) 2024-04-27T03:10:45.364 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:44.990+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:45.364 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:44.990+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:45.364 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:44.990+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:45.364 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:44.991+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:45.927 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:45.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:45.927 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:45.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:45.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:45.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:45.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:45.866+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:45.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:45.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:45.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:45.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:45.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:45.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:45.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:45.658+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:46.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:45.942+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:45.943669+0000) 2024-04-27T03:10:46.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:46.075+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:46.076172+0000) 2024-04-27T03:10:46.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:45.921+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:45.922434+0000) 2024-04-27T03:10:46.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:45.942+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:46.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:45.942+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:46.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:45.942+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:46.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:45.943+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:46.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:46.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:46.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:46.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:46.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:46.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:46.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:46.904+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:46.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:46.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:46.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:46.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:46.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:46.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:46.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:46.662+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:46.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:46.831+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:46.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:46.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:46.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:46.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:46.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:46.832+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:47.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:46.924+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:46.925347+0000) 2024-04-27T03:10:47.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:47.077+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:47.078465+0000) 2024-04-27T03:10:47.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:46.944+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:46.945360+0000) 2024-04-27T03:10:47.927 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:47.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:47.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:47.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:47.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:47.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:47.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:47.785+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:47.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:47.656+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:47.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:47.656+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:47.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:47.656+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:47.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:47.657+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:48.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:48.077+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:48.078656+0000) 2024-04-27T03:10:48.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:47.944+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:47.945542+0000) 2024-04-27T03:10:48.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:47.924+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:47.925517+0000) 2024-04-27T03:10:48.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:47.952+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:48.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:47.952+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:48.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:47.952+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:48.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:47.953+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:48.954 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:10:48.699+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54941 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:10:48.954 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:48.698+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:48.699323+0000) 2024-04-27T03:10:48.954 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:48.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:48.954 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:48.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:48.954 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:48.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:48.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:48.757+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:48.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:48.697+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:48.698146+0000) 2024-04-27T03:10:48.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:48.915+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:48.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:48.915+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:48.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:48.915+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:48.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:48.916+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:48.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:48.924+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:48.925614+0000) 2024-04-27T03:10:48.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:48.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:48.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:48.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:48.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:48.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:48.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:48.678+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:48.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:48.698+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:48.699408+0000) 2024-04-27T03:10:48.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:48.944+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:48.945739+0000) 2024-04-27T03:10:49.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:49.078+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:49.078848+0000) 2024-04-27T03:10:49.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:49.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:49.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:49.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:49.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:49.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:49.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:49.656+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:49.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:49.721+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:49.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:49.721+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:49.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:49.721+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:49.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:49.721+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:50.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:50.080+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:50.080896+0000) 2024-04-27T03:10:50.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:49.949+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:49.949840+0000) 2024-04-27T03:10:50.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:49.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:50.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:49.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:50.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:49.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:50.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:49.923+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:50.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:49.927+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:49.928407+0000) 2024-04-27T03:10:50.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:50.727+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:50.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:50.727+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:50.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:50.727+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:50.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:50.727+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:50.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:50.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:50.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:50.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:50.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:50.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:50.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:50.625+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:51.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:50.951+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:50.952539+0000) 2024-04-27T03:10:51.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:51.082+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:51.083000+0000) 2024-04-27T03:10:51.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:50.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:51.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:50.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:51.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:50.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:51.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:50.904+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:51.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:50.932+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:50.933059+0000) 2024-04-27T03:10:51.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:51.872+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:51.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:51.872+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:51.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:51.872+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:51.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:51.873+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:51.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:51.752+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:51.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:51.752+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:51.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:51.752+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:51.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:51.753+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:51.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:51.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:51.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:51.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:51.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:51.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:51.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:51.632+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:52.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:51.953+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:51.954047+0000) 2024-04-27T03:10:52.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:51.933+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:51.934524+0000) 2024-04-27T03:10:52.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:52.083+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:52.084480+0000) 2024-04-27T03:10:52.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:52.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:52.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:52.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:52.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:52.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:52.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:52.584+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:52.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:52.953+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:52.954281+0000) 2024-04-27T03:10:52.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:52.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:52.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:52.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:52.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:52.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:52.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:52.711+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:52.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:52.859+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:52.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:52.859+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:52.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:52.859+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:52.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:52.860+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:52.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:52.933+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:52.934774+0000) 2024-04-27T03:10:53.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:53.083+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:53.084682+0000) 2024-04-27T03:10:53.928 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:10:53.706+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 54980 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:10:53.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:53.696+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:53.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:53.696+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:53.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:53.696+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:53.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:53.697+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:53.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:53.705+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:53.706001+0000) 2024-04-27T03:10:53.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:53.705+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:53.706032+0000) 2024-04-27T03:10:53.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:53.811+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:53.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:53.811+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:53.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:53.811+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:53.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:53.812+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:53.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:53.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:53.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:53.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:53.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:53.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:53.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:53.612+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:53.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:53.706+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:53.707073+0000) 2024-04-27T03:10:54.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:54.084+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:54.084848+0000) 2024-04-27T03:10:54.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:53.934+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:53.934948+0000) 2024-04-27T03:10:54.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:53.953+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:53.954517+0000) 2024-04-27T03:10:54.706 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:54.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:54.706 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:54.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:54.706 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:54.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:54.706 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:54.568+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:54.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:54.697+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:54.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:54.697+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:54.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:54.697+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:54.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:54.697+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:54.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:54.807+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:54.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:54.807+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:54.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:54.807+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:54.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:54.807+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:54.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:54.937+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:54.938060+0000) 2024-04-27T03:10:55.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:54.956+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:54.957490+0000) 2024-04-27T03:10:55.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:55.088+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:55.089796+0000) 2024-04-27T03:10:55.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:55.789+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:55.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:55.789+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:55.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:55.789+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:55.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:55.789+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:55.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:55.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:55.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:55.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:55.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:55.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:55.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:55.616+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:55.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:55.649+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:55.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:55.649+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:55.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:55.649+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:55.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:55.650+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:56.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:55.939+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:55.940240+0000) 2024-04-27T03:10:56.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:55.959+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:55.960227+0000) 2024-04-27T03:10:56.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:56.091+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:56.092020+0000) 2024-04-27T03:10:56.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:56.767+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:56.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:56.767+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:56.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:56.767+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:56.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:56.767+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:56.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:56.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:56.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:56.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:56.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:56.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:56.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:56.578+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:56.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:56.675+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:56.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:56.675+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:56.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:56.675+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:56.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:56.675+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:57.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:56.941+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:56.942344+0000) 2024-04-27T03:10:57.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:56.961+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:56.961901+0000) 2024-04-27T03:10:57.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:57.092+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:57.093559+0000) 2024-04-27T03:10:57.656 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:57.566+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:57.656 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:57.566+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:57.656 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:57.566+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:57.657 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:57.567+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:57.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:57.723+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:57.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:57.723+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:57.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:57.723+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:57.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:57.724+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:57.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:57.647+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:57.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:57.647+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:57.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:57.647+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:57.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:57.647+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:58.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:57.941+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:57.942524+0000) 2024-04-27T03:10:58.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:57.961+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:57.962070+0000) 2024-04-27T03:10:58.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:58.092+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:58.093749+0000) 2024-04-27T03:10:58.929 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:10:58.714+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55019 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:10:58.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:58.653+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:58.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:58.653+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:58.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:58.653+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:58.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:58.654+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:58.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:58.711+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:58.712129+0000) 2024-04-27T03:10:58.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:58.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:58.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:58.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:58.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:58.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:58.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:58.616+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:58.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:58.710+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:58.710953+0000) 2024-04-27T03:10:58.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:58.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:58.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:58.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:58.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:58.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:58.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:58.699+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:58.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:58.711+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:58.712312+0000) 2024-04-27T03:10:59.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:58.961+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:58.962236+0000) 2024-04-27T03:10:59.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:58.941+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:58.942706+0000) 2024-04-27T03:10:59.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:59.093+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:59.093941+0000) 2024-04-27T03:10:59.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:59.728+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:59.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:59.728+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:59.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:59.728+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:10:59.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:59.729+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:59.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:59.662+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:59.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:59.662+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:59.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:59.662+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:10:59.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:10:59.662+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:10:59.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:59.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:59.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:59.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:59.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:59.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:10:59.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:59.619+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:00.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:00.097+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:00.098581+0000) 2024-04-27T03:11:00.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:10:59.946+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:59.947395+0000) 2024-04-27T03:11:00.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:10:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:10:59.966+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:10:59.967213+0000) 2024-04-27T03:11:00.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:00.637+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:00.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:00.637+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:00.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:00.637+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:00.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:00.637+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:00.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:00.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:00.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:00.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:00.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:00.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:00.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:00.740+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:00.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:00.641+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:00.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:00.641+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:00.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:00.641+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:00.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:00.642+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:01.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:00.948+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:00.949670+0000) 2024-04-27T03:11:01.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:00.969+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:00.969820+0000) 2024-04-27T03:11:01.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:01.100+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:01.101062+0000) 2024-04-27T03:11:01.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:01.767+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:01.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:01.767+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:01.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:01.767+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:01.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:01.768+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:01.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:01.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:01.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:01.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:01.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:01.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:01.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:01.630+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:01.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:01.618+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:01.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:01.618+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:01.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:01.618+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:01.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:01.618+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:02.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:01.950+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:01.951347+0000) 2024-04-27T03:11:02.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:01.972+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:01.972890+0000) 2024-04-27T03:11:02.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:02.101+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:02.102668+0000) 2024-04-27T03:11:02.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:02.721+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:02.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:02.721+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:02.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:02.721+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:02.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:02.722+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:02.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:02.621+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:02.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:02.621+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:02.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:02.621+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:02.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:02.622+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:02.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:02.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:02.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:02.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:02.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:02.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:02.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:02.622+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:03.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:02.950+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:02.951532+0000) 2024-04-27T03:11:03.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:02.972+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:02.973058+0000) 2024-04-27T03:11:03.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:03.102+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:03.102832+0000) 2024-04-27T03:11:03.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:03.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:03.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:03.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:03.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:03.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:03.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:03.596+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:03.929 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:11:03.720+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55058 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:11:03.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:03.719+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:03.719918+0000) 2024-04-27T03:11:03.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:03.718+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:03.719046+0000) 2024-04-27T03:11:03.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:03.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:03.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:03.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:03.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:03.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:03.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:03.771+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:03.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:03.665+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:03.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:03.665+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:03.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:03.665+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:03.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:03.665+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:03.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:03.718+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:03.719724+0000) 2024-04-27T03:11:04.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:03.950+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:03.951717+0000) 2024-04-27T03:11:04.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:03.972+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:03.973251+0000) 2024-04-27T03:11:04.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:04.102+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:04.103093+0000) 2024-04-27T03:11:04.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:04.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:04.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:04.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:04.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:04.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:04.675 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:04.572+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:04.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:04.665+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:04.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:04.665+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:04.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:04.665+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:04.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:04.665+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:04.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:04.816+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:04.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:04.816+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:04.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:04.816+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:04.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:04.818+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:05.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:05.104+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:05.105732+0000) 2024-04-27T03:11:05.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:04.953+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:04.954427+0000) 2024-04-27T03:11:05.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:04.974+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:04.975531+0000) 2024-04-27T03:11:05.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:05.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:05.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:05.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:05.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:05.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:05.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:05.560+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:05.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:05.782+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:05.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:05.782+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:05.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:05.782+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:05.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:05.784+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:05.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:05.621+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:05.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:05.621+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:05.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:05.621+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:05.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:05.622+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:06.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:05.956+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:05.957310+0000) 2024-04-27T03:11:06.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:05.979+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:05.979853+0000) 2024-04-27T03:11:06.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:06.107+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:06.108575+0000) 2024-04-27T03:11:06.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:06.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:06.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:06.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:06.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:06.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:06.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:06.557+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:06.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:06.578+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:06.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:06.578+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:06.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:06.578+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:06.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:06.578+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:07.085 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:06.981+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:06.981908+0000) 2024-04-27T03:11:07.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:06.820+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:07.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:06.820+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:07.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:06.820+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:07.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:06.821+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:07.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:06.958+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:06.959232+0000) 2024-04-27T03:11:07.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:07.109+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:07.110715+0000) 2024-04-27T03:11:07.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:07.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:07.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:07.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:07.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:07.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:07.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:07.565+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:07.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:07.615+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:07.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:07.615+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:07.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:07.615+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:07.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:07.615+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:08.085 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:07.981+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:07.982138+0000) 2024-04-27T03:11:08.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:07.776+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:08.086 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:07.776+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:08.086 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:07.776+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:08.086 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:07.777+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:08.086 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:07.958+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:07.959413+0000) 2024-04-27T03:11:08.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:08.110+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:08.110839+0000) 2024-04-27T03:11:08.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:08.573+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:08.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:08.573+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:08.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:08.573+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:08.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:08.574+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:08.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:08.561+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:08.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:08.561+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:08.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:08.561+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:08.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:08.562+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:08.991 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:11:08.727+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55097 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:11:08.991 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:08.726+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:08.727679+0000) 2024-04-27T03:11:08.992 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:08.725+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:08.726236+0000) 2024-04-27T03:11:08.992 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:08.725+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:08.726509+0000) 2024-04-27T03:11:08.992 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:08.756+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:08.992 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:08.756+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:08.992 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:08.756+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:08.993 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:08.757+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:08.993 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:08.958+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:08.959580+0000) 2024-04-27T03:11:09.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:09.110+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:09.111034+0000) 2024-04-27T03:11:09.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:08.981+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:08.982307+0000) 2024-04-27T03:11:09.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:09.594+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:09.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:09.594+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:09.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:09.594+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:09.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:09.594+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:09.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:09.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:09.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:09.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:09.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:09.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:09.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:09.578+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:10.018 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:09.984+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:09.985054+0000) 2024-04-27T03:11:10.019 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:09.736+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:10.019 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:09.736+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:10.019 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:09.736+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:10.019 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:09.737+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:10.019 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:09.963+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:09.963800+0000) 2024-04-27T03:11:10.380 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:10.113+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:10.114116+0000) 2024-04-27T03:11:10.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:10.616+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:10.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:10.616+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:10.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:10.616+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:10.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:10.617+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:10.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:10.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:10.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:10.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:10.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:10.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:10.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:10.587+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:10.995 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:10.736+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:10.996 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:10.736+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:10.996 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:10.736+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:10.996 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:10.737+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:10.996 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:10.965+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:10.966047+0000) 2024-04-27T03:11:11.380 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:11.115+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:11.116309+0000) 2024-04-27T03:11:11.380 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:10.986+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:10.987179+0000) 2024-04-27T03:11:11.630 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:11.620+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:11.630 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:11.620+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:11.630 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:11.620+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:11.630 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:11.621+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:11.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:11.585+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:11.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:11.585+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:11.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:11.585+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:11.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:11.587+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:11.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:11.772+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:11.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:11.772+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:11.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:11.772+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:11.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:11.773+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:12.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:11.987+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:11.988736+0000) 2024-04-27T03:11:12.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:11.966+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:11.967701+0000) 2024-04-27T03:11:12.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:12.117+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:12.118433+0000) 2024-04-27T03:11:12.631 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:12.603+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:12.631 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:12.603+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:12.631 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:12.603+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:12.631 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:12.604+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:12.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:12.783+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:12.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:12.783+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:12.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:12.783+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:12.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:12.784+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:12.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:12.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:12.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:12.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:12.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:12.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:12.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:12.622+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:13.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:13.117+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:13.118714+0000) 2024-04-27T03:11:13.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:12.967+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:12.967914+0000) 2024-04-27T03:11:13.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:12.988+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:12.988923+0000) 2024-04-27T03:11:13.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:13.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:13.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:13.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:13.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:13.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:13.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:13.580+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:13.930 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:11:13.734+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55136 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:11:13.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:13.732+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:13.733531+0000) 2024-04-27T03:11:13.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:13.621+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:13.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:13.621+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:13.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:13.621+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:13.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:13.622+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:13.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:13.732+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:13.733558+0000) 2024-04-27T03:11:13.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:13.733+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:13.734068+0000) 2024-04-27T03:11:13.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:13.772+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:13.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:13.772+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:13.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:13.772+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:13.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:13.772+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:14.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:13.967+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:13.968116+0000) 2024-04-27T03:11:14.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:14.118+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:14.118907+0000) 2024-04-27T03:11:14.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:13.988+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:13.989094+0000) 2024-04-27T03:11:14.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:14.605+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:14.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:14.605+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:14.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:14.605+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:14.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:14.605+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:14.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:14.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:14.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:14.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:14.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:14.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:14.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:14.592+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:15.086 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:14.990+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:14.991195+0000) 2024-04-27T03:11:15.086 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:14.784+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:15.086 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:14.784+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:15.086 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:14.784+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:15.086 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:14.785+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:15.086 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:14.970+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:14.971106+0000) 2024-04-27T03:11:15.383 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:15.120+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:15.121741+0000) 2024-04-27T03:11:15.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:15.555+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:15.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:15.555+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:15.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:15.555+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:15.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:15.556+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:15.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:15.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:15.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:15.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:15.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:15.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:15.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:15.641+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:15.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:15.811+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:15.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:15.811+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:15.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:15.811+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:15.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:15.813+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:16.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:15.993+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:15.993825+0000) 2024-04-27T03:11:16.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:16.124+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:16.125056+0000) 2024-04-27T03:11:16.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:15.972+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:15.973786+0000) 2024-04-27T03:11:16.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:16.535+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:16.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:16.535+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:16.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:16.535+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:16.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:16.536+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:16.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:16.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:16.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:16.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:16.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:16.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:16.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:16.634+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:16.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:16.797+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:16.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:16.797+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:16.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:16.797+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:16.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:16.798+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:17.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:16.995+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:16.996368+0000) 2024-04-27T03:11:17.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:17.125+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:17.126617+0000) 2024-04-27T03:11:17.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:16.974+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:16.975349+0000) 2024-04-27T03:11:17.644 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:17.578+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:17.644 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:17.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:17.644 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:17.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:17.644 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:17.579+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:17.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:17.816+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:17.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:17.816+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:17.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:17.816+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:17.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:17.817+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:17.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:17.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:17.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:17.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:17.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:17.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:17.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:17.636+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:18.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:17.974+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:17.975513+0000) 2024-04-27T03:11:18.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:18.126+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:18.126812+0000) 2024-04-27T03:11:18.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:17.995+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:17.996563+0000) 2024-04-27T03:11:18.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:18.530+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:18.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:18.530+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:18.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:18.530+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:18.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:18.530+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:18.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:18.740+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:18.742789+0000) 2024-04-27T03:11:18.931 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:11:18.740+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55175 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:11:18.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:18.740+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:18.742299+0000) 2024-04-27T03:11:18.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:18.796+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:18.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:18.796+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:18.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:18.796+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:18.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:18.797+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:18.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:18.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:18.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:18.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:18.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:18.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:18.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:18.654+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:18.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:18.738+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:18.740013+0000) 2024-04-27T03:11:19.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:19.125+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:19.127000+0000) 2024-04-27T03:11:19.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:18.973+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:18.975700+0000) 2024-04-27T03:11:19.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:18.994+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:18.996765+0000) 2024-04-27T03:11:19.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:19.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:19.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:19.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:19.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:19.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:19.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:19.494+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:19.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:19.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:19.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:19.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:19.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:19.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:19.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:19.665+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:19.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:19.816+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:19.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:19.817+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:19.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:19.817+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:19.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:19.817+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:20.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:19.976+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:19.977870+0000) 2024-04-27T03:11:20.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:19.999+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:20.000966+0000) 2024-04-27T03:11:20.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:20.127+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:20.129181+0000) 2024-04-27T03:11:20.802 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:20.645+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:20.802 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:20.645+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:20.802 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:20.645+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:20.802 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:20.646+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:20.803 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:20.501+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:20.803 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:20.501+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:20.803 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:20.501+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:20.803 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:20.501+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:21.086 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:21.002+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:21.003844+0000) 2024-04-27T03:11:21.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:20.791+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:21.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:20.791+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:21.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:20.791+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:21.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:20.792+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:21.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:20.979+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:20.981058+0000) 2024-04-27T03:11:21.386 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:21.129+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:21.131360+0000) 2024-04-27T03:11:21.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:21.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:21.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:21.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:21.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:21.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:21.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:21.478+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:21.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:21.804+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:21.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:21.804+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:21.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:21.804+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:21.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:21.805+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:21.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:21.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:21.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:21.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:21.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:21.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:21.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:21.661+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:22.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:21.980+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:21.982690+0000) 2024-04-27T03:11:22.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:22.131+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:22.133498+0000) 2024-04-27T03:11:22.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:22.004+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:22.005945+0000) 2024-04-27T03:11:22.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:22.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:22.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:22.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:22.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:22.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:22.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:22.449+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:22.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:22.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:22.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:22.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:22.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:22.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:22.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:22.622+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:22.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:22.790+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:22.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:22.790+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:22.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:22.790+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:22.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:22.791+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:23.270 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:22.981+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:22.982914+0000) 2024-04-27T03:11:23.271 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:23.004+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:23.006125+0000) 2024-04-27T03:11:23.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:23.131+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:23.133756+0000) 2024-04-27T03:11:23.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:23.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:23.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:23.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:23.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:23.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:23.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:23.430+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:23.932 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:11:23.747+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55214 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:11:23.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:23.746+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:23.747895+0000) 2024-04-27T03:11:23.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:23.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:23.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:23.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:23.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:23.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:23.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:23.662+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:23.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:23.744+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:23.746651+0000) 2024-04-27T03:11:23.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:23.744+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:23.746462+0000) 2024-04-27T03:11:23.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:23.762+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:23.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:23.762+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:23.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:23.762+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:23.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:23.763+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:24.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:24.004+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:24.006316+0000) 2024-04-27T03:11:24.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:24.132+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:24.133938+0000) 2024-04-27T03:11:24.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:23.981+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:23.983097+0000) 2024-04-27T03:11:24.662 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:24.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:24.662 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:24.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:24.662 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:24.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:24.662 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:24.404+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:24.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:24.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:24.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:24.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:24.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:24.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:24.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:24.722+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:24.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:24.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:24.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:24.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:24.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:24.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:24.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:24.653+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:25.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:24.983+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:24.985310+0000) 2024-04-27T03:11:25.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:25.007+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:25.008960+0000) 2024-04-27T03:11:25.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:25.134+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:25.136163+0000) 2024-04-27T03:11:25.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:25.370+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:25.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:25.370+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:25.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:25.370+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:25.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:25.371+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:25.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:25.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:25.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:25.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:25.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:25.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:25.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:25.687+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:25.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:25.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:25.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:25.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:25.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:25.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:25.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:25.701+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:26.336 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:25.985+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:25.987551+0000) 2024-04-27T03:11:26.337 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:26.012+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:26.013846+0000) 2024-04-27T03:11:26.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:26.136+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:26.138369+0000) 2024-04-27T03:11:26.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:26.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:26.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:26.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:26.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:26.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:26.708 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:26.326+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:26.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:26.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:26.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:26.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:26.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:26.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:26.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:26.724+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:26.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:26.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:26.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:26.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:26.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:26.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:26.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:26.698+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:27.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:27.139+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:27.141471+0000) 2024-04-27T03:11:27.296 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:27.014+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:27.016508+0000) 2024-04-27T03:11:27.297 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:26.988+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:26.990590+0000) 2024-04-27T03:11:27.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:27.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:27.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:27.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:27.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:27.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:27.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:27.287+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:27.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:27.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:27.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:27.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:27.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:27.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:27.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:27.683+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:27.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:27.770+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:27.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:27.770+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:27.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:27.770+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:27.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:27.771+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:28.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:27.989+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:27.990837+0000) 2024-04-27T03:11:28.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:28.139+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:28.141658+0000) 2024-04-27T03:11:28.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:28.014+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:28.016679+0000) 2024-04-27T03:11:28.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:28.265+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:28.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:28.265+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:28.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:28.265+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:28.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:28.266+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:28.932 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:11:28.753+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55253 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:11:28.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:28.752+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:28.754140+0000) 2024-04-27T03:11:28.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:28.751+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:28.753737+0000) 2024-04-27T03:11:28.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:28.798+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:28.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:28.798+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:28.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:28.798+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:28.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:28.799+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:28.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:28.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:28.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:28.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:28.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:28.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:28.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:28.648+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:28.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:28.753+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:28.754960+0000) 2024-04-27T03:11:29.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:29.140+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:29.141893+0000) 2024-04-27T03:11:29.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:28.989+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:28.991054+0000) 2024-04-27T03:11:29.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:29.015+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:29.016849+0000) 2024-04-27T03:11:29.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:29.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:29.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:29.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:29.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:29.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:29.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:29.240+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:29.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:29.803+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:29.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:29.803+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:29.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:29.803+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:29.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:29.805+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:29.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:29.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:29.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:29.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:29.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:29.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:29.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:29.648+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:30.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:30.142+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:30.144546+0000) 2024-04-27T03:11:30.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:29.994+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:29.995308+0000) 2024-04-27T03:11:30.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:30.017+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:30.019102+0000) 2024-04-27T03:11:30.620 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:30.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:30.620 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:30.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:30.620 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:30.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:30.620 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:30.253+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:30.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:30.797+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:30.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:30.797+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:30.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:30.797+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:30.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:30.797+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:30.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:30.609+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:30.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:30.609+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:30.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:30.609+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:30.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:30.610+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:31.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:30.996+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:30.998654+0000) 2024-04-27T03:11:31.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:31.146+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:31.146892+0000) 2024-04-27T03:11:31.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:31.020+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:31.021931+0000) 2024-04-27T03:11:31.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:31.298+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:31.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:31.298+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:31.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:31.298+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:31.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:31.299+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:31.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:31.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:31.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:31.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:31.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:31.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:31.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:31.649+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:31.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:31.831+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:31.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:31.831+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:31.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:31.831+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:31.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:31.831+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:32.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:32.000+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:32.000865+0000) 2024-04-27T03:11:32.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:32.147+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:32.148682+0000) 2024-04-27T03:11:32.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:32.023+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:32.024731+0000) 2024-04-27T03:11:32.628 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:32.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:32.628 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:32.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:32.628 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:32.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:32.628 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:32.326+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:32.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:32.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:32.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:32.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:32.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:32.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:32.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:32.619+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:32.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:32.832+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:32.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:32.832+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:32.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:32.832+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:32.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:32.833+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:33.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:33.148+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:33.148921+0000) 2024-04-27T03:11:33.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:33.000+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:33.001106+0000) 2024-04-27T03:11:33.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:33.024+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:33.025028+0000) 2024-04-27T03:11:33.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:33.306+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:33.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:33.306+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:33.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:33.306+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:33.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:33.306+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:33.878 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:11:33.761+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55292 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:11:33.878 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:33.761+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:33.761894+0000) 2024-04-27T03:11:33.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:33.759+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:33.760163+0000) 2024-04-27T03:11:33.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:33.868+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:33.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:33.868+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:33.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:33.868+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:33.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:33.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:33.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:33.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:33.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:33.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:33.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:33.596+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:33.880 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:33.759+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:33.760182+0000) 2024-04-27T03:11:34.133 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:33.869+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:34.133 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:34.000+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:34.001308+0000) 2024-04-27T03:11:34.134 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:34.024+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:34.025342+0000) 2024-04-27T03:11:34.394 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:34.148+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:34.149097+0000) 2024-04-27T03:11:34.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:34.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:34.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:34.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:34.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:34.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:34.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:34.349+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:34.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:34.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:34.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:34.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:34.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:34.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:34.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:34.572+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:35.088 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:35.028+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:35.029571+0000) 2024-04-27T03:11:35.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:34.876+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:35.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:34.876+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:35.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:34.876+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:35.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:34.878+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:35.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:35.003+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:35.003963+0000) 2024-04-27T03:11:35.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:35.151+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:35.152088+0000) 2024-04-27T03:11:35.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:35.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:35.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:35.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:35.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:35.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:35.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:35.397+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:35.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:35.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:35.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:35.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:35.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:35.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:35.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:35.595+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:36.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:35.906+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:36.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:35.906+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:36.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:35.906+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:36.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:35.907+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:36.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:36.006+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:36.007041+0000) 2024-04-27T03:11:36.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:36.030+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:36.031687+0000) 2024-04-27T03:11:36.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:36.153+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:36.154515+0000) 2024-04-27T03:11:36.627 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:36.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:36.628 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:36.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:36.628 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:36.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:36.628 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:36.617+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:36.628 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:36.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:36.628 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:36.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:36.628 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:36.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:36.628 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:36.375+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:37.165 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:37.033+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:37.033887+0000) 2024-04-27T03:11:37.165 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:36.884+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:37.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:36.884+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:37.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:36.884+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:37.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:36.884+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:37.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:37.008+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:37.009185+0000) 2024-04-27T03:11:37.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:37.156+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:37.156905+0000) 2024-04-27T03:11:37.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:37.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:37.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:37.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:37.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:37.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:37.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:37.367+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:37.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:37.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:37.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:37.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:37.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:37.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:37.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:37.604+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:38.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:38.156+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:38.157069+0000) 2024-04-27T03:11:38.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:38.033+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:38.034052+0000) 2024-04-27T03:11:38.231 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:37.934+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:38.231 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:37.934+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:38.231 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:37.934+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:38.231 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:37.935+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:38.232 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:38.008+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:38.009356+0000) 2024-04-27T03:11:38.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:38.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:38.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:38.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:38.605 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:38.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:38.605 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:38.383+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:38.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:38.765+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:38.766212+0000) 2024-04-27T03:11:38.896 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:11:38.768+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55331 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:11:38.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:38.767+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:38.768148+0000) 2024-04-27T03:11:38.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:38.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:38.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:38.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:38.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:38.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:38.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:38.596+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:38.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:38.767+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:38.768201+0000) 2024-04-27T03:11:39.165 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:39.033+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:39.034328+0000) 2024-04-27T03:11:39.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:38.906+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:39.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:38.906+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:39.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:38.907+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:39.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:38.907+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:39.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:39.008+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:39.009590+0000) 2024-04-27T03:11:39.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:39.156+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:39.157259+0000) 2024-04-27T03:11:39.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:39.401+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:39.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:39.401+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:39.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:39.401+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:39.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:39.401+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:39.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:39.557+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:39.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:39.557+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:39.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:39.557+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:39.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:39.558+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:40.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:40.037+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:40.038661+0000) 2024-04-27T03:11:40.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:40.159+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:40.159848+0000) 2024-04-27T03:11:40.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:39.953+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:40.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:39.953+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:40.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:39.953+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:40.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:39.954+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:40.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:40.012+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:40.012846+0000) 2024-04-27T03:11:40.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:40.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:40.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:40.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:40.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:40.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:40.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:40.387+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:40.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:40.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:40.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:40.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:40.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:40.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:40.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:40.565+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:41.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:41.039+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:41.040778+0000) 2024-04-27T03:11:41.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:41.162+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:41.163059+0000) 2024-04-27T03:11:41.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:40.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:41.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:40.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:41.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:40.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:41.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:40.940+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:41.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:41.014+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:41.015086+0000) 2024-04-27T03:11:41.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:41.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:41.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:41.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:41.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:41.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:41.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:41.432+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:41.703 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:41.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:41.703 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:41.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:41.703 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:41.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:41.703 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:41.548+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:41.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:41.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:41.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:41.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:41.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:41.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:41.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:41.931+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:42.401 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:42.016+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:42.016846+0000) 2024-04-27T03:11:42.401 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:42.041+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:42.042345+0000) 2024-04-27T03:11:42.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:42.164+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:42.165272+0000) 2024-04-27T03:11:42.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:42.468+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:42.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:42.468+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:42.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:42.468+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:42.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:42.468+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:42.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:42.510+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:42.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:42.510+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:42.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:42.510+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:42.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:42.511+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:43.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:43.041+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:43.042535+0000) 2024-04-27T03:11:43.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:42.895+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:43.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:42.895+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:43.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:42.895+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:43.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:42.896+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:43.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:43.016+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:43.017037+0000) 2024-04-27T03:11:43.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:43.164+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:43.165568+0000) 2024-04-27T03:11:43.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:43.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:43.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:43.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:43.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:43.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:43.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:43.506+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:43.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:43.495+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:43.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:43.495+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:43.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:43.495+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:43.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:43.496+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:44.051 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:11:43.775+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55370 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:11:44.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:43.773+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:43.774128+0000) 2024-04-27T03:11:44.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:43.774+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:43.775004+0000) 2024-04-27T03:11:44.052 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:43.774+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:43.775120+0000) 2024-04-27T03:11:44.052 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:43.857+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:44.052 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:43.857+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:44.052 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:43.857+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:44.052 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:43.857+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:44.052 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:44.016+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:44.017199+0000) 2024-04-27T03:11:44.401 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:44.041+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:44.042709+0000) 2024-04-27T03:11:44.401 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:44.164+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:44.165788+0000) 2024-04-27T03:11:44.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:44.486+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:44.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:44.486+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:44.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:44.486+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:44.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:44.486+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:44.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:44.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:44.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:44.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:44.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:44.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:44.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:44.489+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:45.028 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:44.843+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:45.029 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:44.843+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:45.029 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:44.843+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:45.029 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:44.844+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:45.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:45.019+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:45.020178+0000) 2024-04-27T03:11:45.405 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:45.167+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:45.168417+0000) 2024-04-27T03:11:45.405 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:45.045+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:45.045972+0000) 2024-04-27T03:11:45.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:45.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:45.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:45.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:45.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:45.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:45.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:45.450+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:45.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:45.522+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:45.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:45.522+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:45.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:45.522+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:45.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:45.523+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:46.059 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:45.798+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:46.060 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:45.798+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:46.060 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:45.798+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:46.060 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:45.799+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:46.060 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:46.022+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:46.023076+0000) 2024-04-27T03:11:46.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:46.170+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:46.171354+0000) 2024-04-27T03:11:46.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:46.050+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:46.051043+0000) 2024-04-27T03:11:46.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:46.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:46.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:46.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:46.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:46.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:46.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:46.454+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:46.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:46.487+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:46.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:46.487+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:46.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:46.487+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:46.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:46.487+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:47.033 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:46.770+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:47.033 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:46.770+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:47.033 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:46.770+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:47.033 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:46.771+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:47.405 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:47.024+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:47.024802+0000) 2024-04-27T03:11:47.405 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:47.052+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:47.052825+0000) 2024-04-27T03:11:47.405 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:47.173+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:47.174536+0000) 2024-04-27T03:11:47.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:47.428+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:47.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:47.428+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:47.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:47.428+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:47.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:47.429+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:47.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:47.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:47.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:47.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:47.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:47.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:47.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:47.465+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:48.061 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:47.784+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:48.061 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:47.784+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:48.061 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:47.784+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:48.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:47.786+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:48.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:48.024+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:48.025014+0000) 2024-04-27T03:11:48.405 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:48.052+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:48.052994+0000) 2024-04-27T03:11:48.405 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:48.173+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:48.174792+0000) 2024-04-27T03:11:48.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:48.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:48.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:48.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:48.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:48.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:48.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:48.508+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:48.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:48.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:48.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:48.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:48.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:48.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:48.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:48.449+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:49.023 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:11:48.781+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55409 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:11:49.023 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:48.779+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:48.780346+0000) 2024-04-27T03:11:49.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:48.780+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:48.781532+0000) 2024-04-27T03:11:49.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:48.738+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:49.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:48.738+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:49.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:48.738+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:49.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:48.739+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:49.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:48.780+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:48.781198+0000) 2024-04-27T03:11:49.405 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:49.024+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:49.025203+0000) 2024-04-27T03:11:49.405 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:49.052+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:49.053162+0000) 2024-04-27T03:11:49.406 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:49.174+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:49.174976+0000) 2024-04-27T03:11:49.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T03:11:49.608+0000 7ff0e4ed9700 0 monclient(hunting): authenticate timed out after 300 2024-04-27T03:11:49.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:49.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:49.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:49.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:49.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:49.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:49.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:49.418+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:49.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:49.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:49.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:49.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:49.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:49.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:49.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:49.481+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:50.023 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:49.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:50.023 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:49.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:50.023 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:49.744+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:50.023 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:49.744+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:50.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:50.027+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:50.027836+0000) 2024-04-27T03:11:50.408 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:50.054+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:50.055790+0000) 2024-04-27T03:11:50.409 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:50.176+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:50.177648+0000) 2024-04-27T03:11:50.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:50.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:50.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:50.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:50.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:50.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:50.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:50.464+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:50.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:50.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:50.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:50.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:50.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:50.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:50.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:50.451+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:51.023 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:50.748+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:51.023 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:50.748+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:51.023 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:50.748+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:51.023 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:50.749+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:51.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:51.029+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:51.029970+0000) 2024-04-27T03:11:51.409 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:51.180+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:51.181065+0000) 2024-04-27T03:11:51.409 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:51.057+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:51.057902+0000) 2024-04-27T03:11:51.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:51.493+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:51.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:51.493+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:51.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:51.493+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:51.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:51.494+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:51.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:51.452+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:51.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:51.452+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:51.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:51.452+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:51.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:51.453+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:52.023 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:51.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:52.023 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:51.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:52.023 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:51.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:52.023 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:51.723+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:52.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:52.032+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:52.033173+0000) 2024-04-27T03:11:52.409 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:52.182+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:52.183270+0000) 2024-04-27T03:11:52.409 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:52.058+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:52.059740+0000) 2024-04-27T03:11:52.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:52.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:52.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:52.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:52.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:52.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:52.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:52.416+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:52.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:52.449+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:52.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:52.449+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:52.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:52.449+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:52.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:52.450+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:52.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:52.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:52.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:52.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:52.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:52.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:52.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:52.699+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:53.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:53.032+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:53.033364+0000) 2024-04-27T03:11:53.409 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:53.182+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:53.183535+0000) 2024-04-27T03:11:53.409 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:53.059+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:53.059965+0000) 2024-04-27T03:11:53.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:53.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:53.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:53.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:53.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:53.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:53.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:53.459+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:53.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:53.442+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:53.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:53.442+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:53.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:53.442+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:53.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:53.442+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:54.023 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:11:53.788+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55448 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:11:54.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:53.786+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:53.787071+0000) 2024-04-27T03:11:54.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:53.786+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:53.787757+0000) 2024-04-27T03:11:54.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:53.735+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:54.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:53.735+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:54.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:53.735+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:54.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:53.736+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:54.025 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:53.785+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:53.786554+0000) 2024-04-27T03:11:54.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:54.032+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:54.033584+0000) 2024-04-27T03:11:54.409 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:54.059+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:54.060340+0000) 2024-04-27T03:11:54.409 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:54.182+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:54.183762+0000) 2024-04-27T03:11:54.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:54.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:54.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:54.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:54.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:54.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:54.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:54.499+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:54.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:54.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:54.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:54.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:54.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:54.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:54.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:54.405+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:55.023 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:54.721+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:55.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:54.721+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:55.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:54.721+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:55.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:54.721+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:55.413 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:55.034+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:55.035652+0000) 2024-04-27T03:11:55.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:55.185+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:55.185930+0000) 2024-04-27T03:11:55.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:55.063+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:55.064600+0000) 2024-04-27T03:11:55.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:55.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:55.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:55.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:55.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:55.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:55.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:55.536+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:55.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:55.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:55.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:55.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:55.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:55.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:55.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:55.439+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:56.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:55.757+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:56.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:55.757+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:56.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:55.757+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:56.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:55.758+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:56.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:56.038+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:56.038880+0000) 2024-04-27T03:11:56.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:56.189+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:56.190131+0000) 2024-04-27T03:11:56.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:56.068+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:56.068841+0000) 2024-04-27T03:11:56.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:56.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:56.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:56.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:56.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:56.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:56.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:56.493+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:56.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:56.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:56.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:56.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:56.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:56.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:56.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:56.474+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:57.049 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:56.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:57.049 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:56.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:57.049 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:56.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:57.049 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:56.780+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:57.413 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:57.039+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:57.040511+0000) 2024-04-27T03:11:57.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:57.069+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:57.070474+0000) 2024-04-27T03:11:57.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:57.191+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:57.191797+0000) 2024-04-27T03:11:57.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:57.519+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:57.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:57.519+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:57.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:57.519+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:57.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:57.519+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:57.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:57.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:57.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:57.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:57.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:57.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:57.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:57.475+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:58.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:57.761+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:58.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:57.761+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:58.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:57.761+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:58.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:57.762+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:58.413 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:58.069+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:58.070649+0000) 2024-04-27T03:11:58.414 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:58.039+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:58.040711+0000) 2024-04-27T03:11:58.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:58.191+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:58.192000+0000) 2024-04-27T03:11:58.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:58.558+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:58.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:58.558+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:58.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:58.558+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:58.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:58.558+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:58.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:58.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:58.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:58.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:58.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:58.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:58.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:58.445+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:59.050 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:11:58.795+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55487 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:11:59.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:58.793+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:58.793828+0000) 2024-04-27T03:11:59.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:58.793+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:58.794752+0000) 2024-04-27T03:11:59.051 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:58.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:59.051 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:58.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:59.051 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:58.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:11:59.051 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:58.770+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:59.051 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:58.793+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:58.793984+0000) 2024-04-27T03:11:59.414 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:59.040+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:59.040825+0000) 2024-04-27T03:11:59.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:59.191+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:59.192244+0000) 2024-04-27T03:11:59.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:59.070+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:11:59.070884+0000) 2024-04-27T03:11:59.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:59.578+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:59.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:59.578+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:59.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:59.578+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:11:59.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:11:59.579+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:11:59.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:59.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:59.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:59.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:59.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:59.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:11:59.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:11:59.456+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:00.054 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:59.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:00.054 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:59.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:00.055 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:59.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:00.055 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:11:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:11:59.788+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:00.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:00.072+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:00.073560+0000) 2024-04-27T03:12:00.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:00.045+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:00.045828+0000) 2024-04-27T03:12:00.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:00.194+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:00.195178+0000) 2024-04-27T03:12:00.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:00.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:00.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:00.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:00.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:00.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:00.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:00.494+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:00.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:00.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:00.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:00.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:00.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:00.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:00.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:00.549+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:01.059 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:00.799+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:01.059 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:00.799+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:01.059 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:00.799+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:01.059 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:00.801+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:01.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:01.049+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:01.050650+0000) 2024-04-27T03:12:01.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:01.197+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:01.197845+0000) 2024-04-27T03:12:01.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:01.076+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:01.076800+0000) 2024-04-27T03:12:01.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:01.544+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:01.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:01.544+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:01.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:01.544+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:01.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:01.544+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:01.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:01.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:01.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:01.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:01.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:01.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:01.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:01.486+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:02.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:01.799+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:02.025 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:01.799+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:02.025 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:01.799+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:02.025 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:01.800+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:02.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:02.051+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:02.052788+0000) 2024-04-27T03:12:02.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:02.199+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:02.200341+0000) 2024-04-27T03:12:02.418 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:02.077+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:02.078470+0000) 2024-04-27T03:12:02.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:02.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:02.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:02.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:02.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:02.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:02.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:02.534+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:02.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:02.583+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:02.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:02.583+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:02.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:02.583+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:02.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:02.583+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:03.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:02.845+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:03.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:02.845+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:03.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:02.845+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:03.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:02.846+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:03.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:03.052+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:03.052959+0000) 2024-04-27T03:12:03.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:03.077+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:03.078703+0000) 2024-04-27T03:12:03.418 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:03.199+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:03.200583+0000) 2024-04-27T03:12:03.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:03.538+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:03.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:03.538+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:03.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:03.538+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:03.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:03.539+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:03.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:03.564+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:03.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:03.564+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:03.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:03.564+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:03.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:03.564+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:04.063 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:12:03.802+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55526 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:12:04.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:03.799+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:03.800532+0000) 2024-04-27T03:12:04.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:03.799+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:03.800603+0000) 2024-04-27T03:12:04.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:03.801+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:03.801961+0000) 2024-04-27T03:12:04.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:03.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:04.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:03.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:04.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:03.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:04.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:03.873+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:04.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:04.052+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:04.053161+0000) 2024-04-27T03:12:04.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:04.078+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:04.078937+0000) 2024-04-27T03:12:04.418 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:04.200+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:04.200826+0000) 2024-04-27T03:12:04.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:04.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:04.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:04.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:04.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:04.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:04.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:04.536+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:04.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:04.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:04.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:04.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:04.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:04.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:04.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:04.526+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:05.091 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:04.871+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:05.091 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:04.871+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:05.092 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:04.871+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:05.092 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:04.872+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:05.092 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:05.056+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:05.057094+0000) 2024-04-27T03:12:05.420 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:05.202+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:05.203466+0000) 2024-04-27T03:12:05.420 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:05.082+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:05.083226+0000) 2024-04-27T03:12:05.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:05.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:05.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:05.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:05.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:05.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:05.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:05.572+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:05.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:05.515+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:05.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:05.515+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:05.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:05.515+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:05.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:05.515+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:06.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:05.886+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:06.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:05.886+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:06.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:05.886+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:06.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:05.887+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:06.376 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:06.060+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:06.061305+0000) 2024-04-27T03:12:06.376 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:06.207+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:06.208461+0000) 2024-04-27T03:12:06.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:06.084+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:06.085464+0000) 2024-04-27T03:12:06.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:06.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:06.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:06.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:06.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:06.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:06.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:06.615+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:06.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:06.530+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:06.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:06.530+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:06.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:06.530+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:06.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:06.530+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:07.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:06.889+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:07.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:06.889+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:07.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:06.889+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:07.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:06.890+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:07.420 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:07.064+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:07.065112+0000) 2024-04-27T03:12:07.420 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:07.086+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:07.087655+0000) 2024-04-27T03:12:07.421 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:07.211+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:07.212441+0000) 2024-04-27T03:12:07.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:07.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:07.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:07.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:07.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:07.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:07.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:07.622+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:07.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:07.531+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:07.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:07.531+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:07.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:07.531+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:07.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:07.531+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:08.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:07.935+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:08.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:07.935+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:08.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:07.935+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:08.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:07.935+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:08.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:08.064+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:08.065275+0000) 2024-04-27T03:12:08.420 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:08.211+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:08.212616+0000) 2024-04-27T03:12:08.420 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:08.087+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:08.087854+0000) 2024-04-27T03:12:08.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:08.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:08.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:08.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:08.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:08.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:08.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:08.509+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:08.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:08.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:08.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:08.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:08.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:08.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:08.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:08.652+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:09.074 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:12:08.809+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55565 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:12:09.074 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:08.808+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:08.809666+0000) 2024-04-27T03:12:09.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:08.807+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:08.808514+0000) 2024-04-27T03:12:09.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:08.807+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:08.808672+0000) 2024-04-27T03:12:09.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:08.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:09.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:08.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:09.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:08.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:09.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:08.974+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:09.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:09.064+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:09.065446+0000) 2024-04-27T03:12:09.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:09.087+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:09.088087+0000) 2024-04-27T03:12:09.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:09.212+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:09.212841+0000) 2024-04-27T03:12:09.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:09.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:09.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:09.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:09.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:09.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:09.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:09.620+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:09.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:09.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:09.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:09.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:09.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:09.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:09.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:09.499+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:10.077 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:09.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:10.077 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:09.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:10.077 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:09.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:10.077 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:09.967+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:10.424 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:10.067+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:10.068096+0000) 2024-04-27T03:12:10.424 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:10.214+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:10.215443+0000) 2024-04-27T03:12:10.424 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:10.089+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:10.090182+0000) 2024-04-27T03:12:10.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:10.478+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:10.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:10.478+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:10.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:10.478+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:10.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:10.478+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:10.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:10.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:10.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:10.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:10.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:10.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:10.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:10.650+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:11.079 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:10.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:11.079 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:10.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:11.079 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:10.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:11.079 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:10.961+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:11.424 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:11.069+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:11.070292+0000) 2024-04-27T03:12:11.424 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:11.216+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:11.217580+0000) 2024-04-27T03:12:11.424 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:11.091+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:11.092260+0000) 2024-04-27T03:12:11.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:11.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:11.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:11.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:11.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:11.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:11.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:11.455+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:11.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:11.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:11.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:11.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:11.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:11.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:11.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:11.640+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:12.083 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:11.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:12.083 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:11.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:12.083 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:11.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:12.083 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:11.998+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:12.422 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:12.073+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:12.074276+0000) 2024-04-27T03:12:12.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:12.218+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:12.219249+0000) 2024-04-27T03:12:12.423 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:12.094+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:12.094847+0000) 2024-04-27T03:12:12.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:12.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:12.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:12.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:12.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:12.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:12.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:12.481+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:12.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:12.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:12.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:12.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:12.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:12.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:12.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:12.664+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:13.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:12.991+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:13.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:12.991+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:13.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:12.991+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:13.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:12.992+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:13.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:13.073+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:13.074511+0000) 2024-04-27T03:12:13.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:13.094+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:13.095035+0000) 2024-04-27T03:12:13.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:13.218+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:13.219458+0000) 2024-04-27T03:12:13.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:13.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:13.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:13.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:13.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:13.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:13.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:13.449+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:13.938 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:12:13.816+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55604 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:12:13.938 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:13.814+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:13.815087+0000) 2024-04-27T03:12:13.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:13.815+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:13.816139+0000) 2024-04-27T03:12:13.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:13.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:13.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:13.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:13.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:13.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:13.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:13.679+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:13.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:13.814+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:13.814928+0000) 2024-04-27T03:12:14.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:14.094+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:14.095206+0000) 2024-04-27T03:12:14.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:14.031+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:14.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:14.031+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:14.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:14.031+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:14.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:14.032+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:14.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:14.073+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:14.074753+0000) 2024-04-27T03:12:14.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:14.218+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:14.219651+0000) 2024-04-27T03:12:14.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:14.437+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:14.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:14.437+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:14.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:14.437+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:14.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:14.437+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:14.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:14.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:14.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:14.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:14.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:14.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:14.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:14.703+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:15.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:15.223+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:15.224609+0000) 2024-04-27T03:12:15.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:15.096+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:15.097331+0000) 2024-04-27T03:12:15.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:15.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:15.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:15.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:15.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:15.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:15.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:15.037+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:15.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:15.078+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:15.078986+0000) 2024-04-27T03:12:15.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:15.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:15.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:15.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:15.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:15.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:15.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:15.656+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:15.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:15.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:15.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:15.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:15.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:15.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:15.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:15.455+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:16.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:16.098+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:16.099552+0000) 2024-04-27T03:12:16.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:16.031+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:16.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:16.031+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:16.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:16.031+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:16.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:16.032+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:16.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:16.080+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:16.081173+0000) 2024-04-27T03:12:16.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:16.228+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:16.229498+0000) 2024-04-27T03:12:16.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:16.472+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:16.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:16.472+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:16.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:16.472+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:16.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:16.473+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:16.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:16.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:16.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:16.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:16.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:16.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:16.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:16.625+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:17.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:17.102+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:17.102937+0000) 2024-04-27T03:12:17.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:17.028+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:17.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:17.028+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:17.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:17.028+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:17.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:17.029+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:17.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:17.083+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:17.084488+0000) 2024-04-27T03:12:17.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:17.232+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:17.233446+0000) 2024-04-27T03:12:17.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:17.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:17.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:17.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:17.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:17.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:17.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:17.446+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:17.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:17.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:17.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:17.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:17.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:17.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:17.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:17.661+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:18.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:18.102+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:18.103104+0000) 2024-04-27T03:12:18.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:18.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:18.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:18.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:18.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:18.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:18.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:18.061+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:18.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:18.083+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:18.084668+0000) 2024-04-27T03:12:18.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:18.232+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:18.233687+0000) 2024-04-27T03:12:18.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:18.476+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:18.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:18.476+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:18.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:18.476+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:18.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:18.476+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:18.953 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:12:18.823+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55643 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:12:18.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:18.822+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:18.823051+0000) 2024-04-27T03:12:18.954 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:18.821+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:18.822445+0000) 2024-04-27T03:12:18.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:18.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:18.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:18.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:18.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:18.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:18.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:18.687+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:18.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:18.820+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:18.821749+0000) 2024-04-27T03:12:19.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:19.102+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:19.103291+0000) 2024-04-27T03:12:19.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:19.084+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:19.084896+0000) 2024-04-27T03:12:19.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:19.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:19.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:19.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:19.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:19.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:19.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:19.088+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:19.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:19.233+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:19.233937+0000) 2024-04-27T03:12:19.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:19.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:19.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:19.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:19.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:19.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:19.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:19.456+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:19.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:19.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:19.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:19.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:19.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:19.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:19.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:19.689+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:20.430 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:20.235+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:20.236151+0000) 2024-04-27T03:12:20.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:20.105+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:20.105944+0000) 2024-04-27T03:12:20.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:20.086+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:20.086979+0000) 2024-04-27T03:12:20.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:20.134+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:20.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:20.134+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:20.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:20.134+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:20.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:20.135+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:20.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:20.434+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:20.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:20.434+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:20.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:20.434+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:20.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:20.435+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:20.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:20.675+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:20.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:20.675+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:20.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:20.675+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:20.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:20.676+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:21.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:21.110+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:21.110962+0000) 2024-04-27T03:12:21.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:21.088+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:21.089103+0000) 2024-04-27T03:12:21.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:21.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:21.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:21.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:21.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:21.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:21.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:21.106+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:21.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:21.238+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:21.238905+0000) 2024-04-27T03:12:21.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:21.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:21.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:21.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:21.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:21.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:21.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:21.468+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:21.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:21.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:21.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:21.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:21.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:21.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:21.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:21.677+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:22.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:22.114+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:22.114973+0000) 2024-04-27T03:12:22.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:22.090+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:22.091623+0000) 2024-04-27T03:12:22.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:22.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:22.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:22.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:22.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:22.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:22.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:22.124+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:22.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:22.239+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:22.240704+0000) 2024-04-27T03:12:22.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:22.475+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:22.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:22.475+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:22.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:22.475+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:22.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:22.475+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:22.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:22.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:22.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:22.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:22.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:22.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:22.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:22.720+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:23.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:23.114+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:23.115200+0000) 2024-04-27T03:12:23.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:23.090+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:23.091780+0000) 2024-04-27T03:12:23.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:23.140+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:23.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:23.140+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:23.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:23.140+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:23.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:23.141+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:23.531 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:23.240+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:23.240929+0000) 2024-04-27T03:12:23.785 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:23.521+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:23.785 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:23.521+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:23.785 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:23.521+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:23.785 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:23.521+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:23.785 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:23.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:23.785 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:23.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:23.785 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:23.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:23.785 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:23.703+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:24.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:23.827+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:23.829133+0000) 2024-04-27T03:12:24.094 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:23.826+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:23.828515+0000) 2024-04-27T03:12:24.094 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:12:23.829+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55682 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:12:24.094 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:23.828+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:23.829887+0000) 2024-04-27T03:12:24.430 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:24.239+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:24.241172+0000) 2024-04-27T03:12:24.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:24.113+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:24.115448+0000) 2024-04-27T03:12:24.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:24.090+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:24.091963+0000) 2024-04-27T03:12:24.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:24.185+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:24.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:24.185+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:24.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:24.185+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:24.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:24.186+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:24.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:24.527+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:24.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:24.527+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:24.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:24.527+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:24.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:24.527+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:24.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:24.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:24.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:24.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:24.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:24.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:24.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:24.703+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:25.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:25.115+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:25.117606+0000) 2024-04-27T03:12:25.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:25.244+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:25.246220+0000) 2024-04-27T03:12:25.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:25.094+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:25.095822+0000) 2024-04-27T03:12:25.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:25.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:25.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:25.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:25.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:25.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:25.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:25.151+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:25.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:25.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:25.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:25.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:25.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:25.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:25.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:25.483+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:26.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:25.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:26.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:25.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:26.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:25.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:26.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:25.738+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:26.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:26.118+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:26.120372+0000) 2024-04-27T03:12:26.378 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:26.249+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:26.251211+0000) 2024-04-27T03:12:26.379 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:26.099+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:26.100816+0000) 2024-04-27T03:12:26.379 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:26.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:26.379 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:26.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:26.379 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:26.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:26.379 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:26.182+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:26.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:26.530+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:26.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:26.530+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:26.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:26.530+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:26.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:26.531+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:27.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:26.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:27.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:26.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:27.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:26.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:27.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:26.721+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:27.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:27.253+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:27.255259+0000) 2024-04-27T03:12:27.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:27.120+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:27.122205+0000) 2024-04-27T03:12:27.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:27.103+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:27.104827+0000) 2024-04-27T03:12:27.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:27.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:27.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:27.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:27.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:27.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:27.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:27.147+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:27.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:27.558+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:27.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:27.559+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:27.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:27.559+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:27.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:27.559+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:27.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:27.684+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:27.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:27.684+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:27.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:27.684+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:27.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:27.686+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:28.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:28.253+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:28.255488+0000) 2024-04-27T03:12:28.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:28.120+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:28.122386+0000) 2024-04-27T03:12:28.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:28.103+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:28.105072+0000) 2024-04-27T03:12:28.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:28.124+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:28.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:28.124+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:28.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:28.124+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:28.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:28.125+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:28.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:28.587+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:28.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:28.587+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:28.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:28.587+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:28.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:28.587+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:28.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:28.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:28.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:28.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:28.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:28.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:28.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:28.649+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:29.094 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:12:28.836+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55721 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:12:29.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:28.834+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:28.836335+0000) 2024-04-27T03:12:29.095 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:28.833+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:28.835797+0000) 2024-04-27T03:12:29.095 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:28.833+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:28.835633+0000) 2024-04-27T03:12:29.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:29.103+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:29.105343+0000) 2024-04-27T03:12:29.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:29.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:29.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:29.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:29.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:29.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:29.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:29.112+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:29.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:29.120+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:29.122637+0000) 2024-04-27T03:12:29.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:29.253+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:29.255785+0000) 2024-04-27T03:12:29.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:29.610+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:29.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:29.610+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:29.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:29.610+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:29.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:29.611+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:29.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:29.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:29.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:29.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:29.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:29.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:29.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:29.695+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:30.440 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:30.123+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:30.124825+0000) 2024-04-27T03:12:30.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:30.256+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:30.257965+0000) 2024-04-27T03:12:30.440 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:30.077+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:30.440 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:30.077+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:30.441 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:30.077+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:30.441 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:30.077+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:30.441 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:30.106+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:30.107943+0000) 2024-04-27T03:12:30.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:30.590+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:30.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:30.590+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:30.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:30.590+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:30.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:30.591+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:30.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:30.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:30.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:30.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:30.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:30.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:30.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:30.669+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:31.121 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:31.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:31.121 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:31.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:31.121 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:31.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:31.121 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:31.061+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:31.440 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:31.110+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:31.112233+0000) 2024-04-27T03:12:31.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:31.259+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:31.261144+0000) 2024-04-27T03:12:31.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:31.125+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:31.127454+0000) 2024-04-27T03:12:31.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:31.544+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:31.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:31.544+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:31.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:31.544+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:31.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:31.544+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:31.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:31.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:31.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:31.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:31.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:31.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:31.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:31.658+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:32.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:32.127+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:32.129274+0000) 2024-04-27T03:12:32.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:32.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:32.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:32.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:32.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:32.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:32.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:32.055+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:32.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:32.112+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:32.114451+0000) 2024-04-27T03:12:32.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:32.261+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:32.262902+0000) 2024-04-27T03:12:32.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:32.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:32.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:32.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:32.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:32.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:32.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:32.509+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:32.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:32.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:32.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:32.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:32.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:32.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:32.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:32.637+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:33.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:33.127+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:33.129553+0000) 2024-04-27T03:12:33.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:33.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:33.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:33.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:33.231 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:33.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:33.231 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:33.105+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:33.231 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:33.113+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:33.114675+0000) 2024-04-27T03:12:33.505 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:33.261+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:33.263068+0000) 2024-04-27T03:12:33.505 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:33.495+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:33.505 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:33.495+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:33.505 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:33.495+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:33.786 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:33.495+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:33.786 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:33.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:33.786 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:33.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:33.786 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:33.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:33.786 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:33.621+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:34.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:12:33.843+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55760 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:12:34.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:33.841+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:33.843563+0000) 2024-04-27T03:12:34.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:33.842+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:33.844712+0000) 2024-04-27T03:12:34.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:34.059+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:34.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:34.059+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:34.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:34.059+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:34.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:33.841+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:33.842909+0000) 2024-04-27T03:12:34.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:34.261+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:34.263261+0000) 2024-04-27T03:12:34.441 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:34.059+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:34.441 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:34.113+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:34.114909+0000) 2024-04-27T03:12:34.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:34.127+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:34.129785+0000) 2024-04-27T03:12:34.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:34.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:34.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:34.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:34.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:34.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:34.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:34.654+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:34.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:34.530+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:34.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:34.530+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:34.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:34.530+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:34.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:34.531+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:35.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:35.130+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:35.131842+0000) 2024-04-27T03:12:35.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:35.045+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:35.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:35.045+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:35.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:35.045+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:35.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:35.046+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:35.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:35.115+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:35.117683+0000) 2024-04-27T03:12:35.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:35.264+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:35.265571+0000) 2024-04-27T03:12:35.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:35.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:35.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:35.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:35.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:35.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:35.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:35.505+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:35.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:35.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:35.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:35.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:35.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:35.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:35.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:35.675+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:36.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:36.134+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:36.135079+0000) 2024-04-27T03:12:36.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:36.084+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:36.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:36.084+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:36.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:36.084+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:36.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:36.085+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:36.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:36.119+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:36.119880+0000) 2024-04-27T03:12:36.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:36.267+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:36.268357+0000) 2024-04-27T03:12:36.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:36.533+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:36.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:36.533+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:36.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:36.533+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:36.941 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:36.533+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:36.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:36.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:36.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:36.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:36.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:36.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:36.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:36.661+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:37.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:37.138+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:37.138924+0000) 2024-04-27T03:12:37.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:37.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:37.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:37.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:37.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:37.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:37.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:37.062+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:37.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:37.122+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:37.123233+0000) 2024-04-27T03:12:37.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:37.269+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:37.270063+0000) 2024-04-27T03:12:37.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:37.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:37.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:37.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:37.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:37.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:37.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:37.509+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:37.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:37.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:37.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:37.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:37.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:37.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:37.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:37.656+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:38.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:38.138+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:38.139085+0000) 2024-04-27T03:12:38.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:38.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:38.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:38.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:38.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:38.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:38.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:38.077+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:38.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:38.122+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:38.123483+0000) 2024-04-27T03:12:38.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:38.269+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:38.270254+0000) 2024-04-27T03:12:38.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:38.464+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:38.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:38.464+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:38.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:38.464+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:38.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:38.465+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:38.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:38.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:38.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:38.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:38.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:38.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:38.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:38.624+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:39.132 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:12:38.851+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55799 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:12:39.133 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:38.850+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:38.851092+0000) 2024-04-27T03:12:39.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:38.850+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:38.851091+0000) 2024-04-27T03:12:39.133 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:38.850+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:38.851382+0000) 2024-04-27T03:12:39.133 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:39.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:39.133 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:39.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:39.133 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:39.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:39.133 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:39.089+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:39.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:39.122+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:39.123683+0000) 2024-04-27T03:12:39.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:39.138+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:39.139320+0000) 2024-04-27T03:12:39.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:39.269+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:39.270441+0000) 2024-04-27T03:12:39.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:39.460+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:39.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:39.460+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:39.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:39.460+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:39.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:39.460+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:39.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:39.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:39.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:39.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:39.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:39.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:39.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:39.632+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:40.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:40.141+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:40.142251+0000) 2024-04-27T03:12:40.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:40.127+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:40.128081+0000) 2024-04-27T03:12:40.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:40.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:40.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:40.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:40.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:40.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:40.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:40.129+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:40.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:40.273+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:40.274645+0000) 2024-04-27T03:12:40.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:40.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:40.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:40.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:40.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:40.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:40.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:40.428+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:40.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:40.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:40.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:40.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:40.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:40.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:40.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:40.625+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:41.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:41.144+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:41.144843+0000) 2024-04-27T03:12:41.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:41.129+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:41.130518+0000) 2024-04-27T03:12:41.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:41.174+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:41.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:41.174+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:41.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:41.174+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:41.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:41.175+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:41.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:41.275+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:41.276762+0000) 2024-04-27T03:12:41.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:41.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:41.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:41.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:41.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:41.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:41.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:41.407+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:41.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:41.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:41.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:41.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:41.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:41.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:41.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:41.655+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:42.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:42.146+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:42.147275+0000) 2024-04-27T03:12:42.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:42.131+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:42.132768+0000) 2024-04-27T03:12:42.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:42.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:42.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:42.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:42.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:42.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:42.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:42.216+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:42.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:42.277+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:42.278534+0000) 2024-04-27T03:12:42.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:42.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:42.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:42.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:42.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:42.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:42.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:42.423+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:42.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:42.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:42.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:42.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:42.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:42.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:42.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:42.706+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:43.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:43.132+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:43.132953+0000) 2024-04-27T03:12:43.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:43.146+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:43.147510+0000) 2024-04-27T03:12:43.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:43.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:43.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:43.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:43.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:43.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:43.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:43.195+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:43.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:43.277+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:43.278788+0000) 2024-04-27T03:12:43.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:43.462+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:43.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:43.462+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:43.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:43.462+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:43.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:43.463+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:43.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:43.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:43.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:43.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:43.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:43.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:43.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:43.662+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:44.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:43.855+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:43.856731+0000) 2024-04-27T03:12:44.096 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:43.856+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:43.856944+0000) 2024-04-27T03:12:44.096 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:12:43.858+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55838 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:12:44.097 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:43.856+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:43.857601+0000) 2024-04-27T03:12:44.429 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:44.146+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:44.147713+0000) 2024-04-27T03:12:44.430 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:44.278+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:44.278971+0000) 2024-04-27T03:12:44.430 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:44.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:44.430 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:44.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:44.430 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:44.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:44.430 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:44.132+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:44.133179+0000) 2024-04-27T03:12:44.430 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:44.185+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:44.430 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:44.185+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:44.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:44.185+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:44.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:44.186+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:44.689 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:44.420+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:44.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:44.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:44.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:44.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:44.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:44.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:44.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:44.680+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:45.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:45.134+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:45.135456+0000) 2024-04-27T03:12:45.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:45.149+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:45.149847+0000) 2024-04-27T03:12:45.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:45.227+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:45.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:45.227+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:45.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:45.227+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:45.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:45.228+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:45.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:45.280+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:45.281366+0000) 2024-04-27T03:12:45.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:45.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:45.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:45.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:45.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:45.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:45.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:45.394+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:45.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:45.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:45.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:45.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:45.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:45.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:45.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:45.652+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:46.215 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:46.136+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:46.137552+0000) 2024-04-27T03:12:46.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:46.151+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:46.152641+0000) 2024-04-27T03:12:46.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:46.257+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:46.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:46.257+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:46.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:46.257+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:46.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:46.258+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:46.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:46.282+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:46.283385+0000) 2024-04-27T03:12:46.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:46.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:46.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:46.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:46.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:46.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:46.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:46.376+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:46.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:46.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:46.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:46.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:46.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:46.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:46.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:46.623+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:47.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:47.154+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:47.155193+0000) 2024-04-27T03:12:47.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:47.139+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:47.139951+0000) 2024-04-27T03:12:47.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:47.237+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:47.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:47.237+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:47.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:47.237+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:47.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:47.238+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:47.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:47.284+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:47.285691+0000) 2024-04-27T03:12:47.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:47.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:47.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:47.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:47.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:47.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:47.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:47.380+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:47.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:47.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:47.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:47.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:47.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:47.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:47.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:47.648+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:48.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:48.154+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:48.155517+0000) 2024-04-27T03:12:48.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:48.139+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:48.140194+0000) 2024-04-27T03:12:48.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:48.264+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:48.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:48.264+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:48.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:48.264+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:48.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:48.264+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:48.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:48.285+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:48.285888+0000) 2024-04-27T03:12:48.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:48.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:48.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:48.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:48.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:48.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:48.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:48.344+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:48.839 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:48.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:48.840 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:48.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:48.840 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:48.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:48.840 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:48.667+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:49.142 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:12:48.864+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55877 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:12:49.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:48.862+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:48.863564+0000) 2024-04-27T03:12:49.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:48.863+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:48.864553+0000) 2024-04-27T03:12:49.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:48.862+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:48.863239+0000) 2024-04-27T03:12:49.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:49.155+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:49.155830+0000) 2024-04-27T03:12:49.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:49.285+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:49.286229+0000) 2024-04-27T03:12:49.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:49.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:49.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:49.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:49.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:49.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:49.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:49.369+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:49.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:49.139+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:49.140445+0000) 2024-04-27T03:12:49.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:49.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:49.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:49.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:49.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:49.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:49.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:49.254+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:49.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:49.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:49.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:49.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:49.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:49.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:49.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:49.672+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:50.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:50.142+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:50.143269+0000) 2024-04-27T03:12:50.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:50.158+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:50.159020+0000) 2024-04-27T03:12:50.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:50.229+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:50.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:50.229+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:50.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:50.229+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:50.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:50.230+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:50.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:50.288+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:50.289233+0000) 2024-04-27T03:12:50.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:50.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:50.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:50.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:50.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:50.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:50.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:50.374+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:50.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:50.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:50.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:50.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:50.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:50.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:50.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:50.681+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:51.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:51.162+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:51.163373+0000) 2024-04-27T03:12:51.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:51.145+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:51.145855+0000) 2024-04-27T03:12:51.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:51.257+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:51.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:51.257+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:51.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:51.257+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:51.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:51.258+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:51.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:51.290+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:51.291609+0000) 2024-04-27T03:12:51.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:51.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:51.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:51.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:51.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:51.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:51.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:51.382+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:51.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:51.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:51.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:51.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:51.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:51.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:51.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:51.663+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:52.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:52.148+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:52.149372+0000) 2024-04-27T03:12:52.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:52.164+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:52.165682+0000) 2024-04-27T03:12:52.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:52.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:52.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:52.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:52.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:52.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:52.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:52.257+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:52.579 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:52.293+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:52.294225+0000) 2024-04-27T03:12:52.579 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:52.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:52.579 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:52.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:52.579 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:52.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:52.579 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:52.374+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:52.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:52.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:52.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:52.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:52.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:52.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:52.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:52.632+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:53.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:53.148+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:53.149679+0000) 2024-04-27T03:12:53.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:53.165+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:53.165893+0000) 2024-04-27T03:12:53.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:53.217+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:53.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:53.217+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:53.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:53.217+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:53.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:53.218+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:53.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:53.293+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:53.294475+0000) 2024-04-27T03:12:53.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:53.345+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:53.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:53.345+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:53.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:53.345+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:53.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:53.345+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:53.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:53.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:53.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:53.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:53.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:53.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:53.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:53.673+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:54.143 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:12:53.872+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55916 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:12:54.144 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:53.871+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:53.872456+0000) 2024-04-27T03:12:54.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:53.871+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:53.872035+0000) 2024-04-27T03:12:54.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:53.870+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:53.871183+0000) 2024-04-27T03:12:54.454 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:54.165+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:54.166222+0000) 2024-04-27T03:12:54.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:54.293+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:54.294719+0000) 2024-04-27T03:12:54.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:54.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:54.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:54.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:54.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:54.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:54.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:54.308+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:54.455 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:54.149+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:54.149874+0000) 2024-04-27T03:12:54.455 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:54.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:54.456 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:54.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:54.456 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:54.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:54.456 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:54.194+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:54.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:54.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:54.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:54.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:54.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:54.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:54.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:54.658+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:55.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:55.151+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:55.152466+0000) 2024-04-27T03:12:55.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:55.169+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:55.170597+0000) 2024-04-27T03:12:55.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:55.296+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:55.297536+0000) 2024-04-27T03:12:55.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:55.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:55.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:55.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:55.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:55.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:55.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:55.298+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:55.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:55.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:55.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:55.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:55.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:55.234+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:55.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:55.235+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:55.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:55.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:55.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:55.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:55.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:55.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:55.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:55.673+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:56.215 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:56.155+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:56.156538+0000) 2024-04-27T03:12:56.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:56.172+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:56.172820+0000) 2024-04-27T03:12:56.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:56.205+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:56.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:56.205+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:56.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:56.205+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:56.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:56.207+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:56.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:56.300+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:56.300951+0000) 2024-04-27T03:12:56.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:56.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:56.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:56.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:56.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:56.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:56.481 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:56.310+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:56.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:56.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:56.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:56.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:56.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:56.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:56.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:56.634+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:57.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:57.158+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:57.158958+0000) 2024-04-27T03:12:57.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:57.174+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:57.175092+0000) 2024-04-27T03:12:57.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:57.234+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:57.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:57.234+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:57.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:57.234+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:57.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:57.235+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:57.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:57.302+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:57.303270+0000) 2024-04-27T03:12:57.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:57.321+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:57.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:57.321+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:57.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:57.321+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:57.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:57.321+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:57.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:57.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:57.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:57.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:57.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:57.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:57.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:57.640+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:58.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:58.158+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:58.159112+0000) 2024-04-27T03:12:58.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:58.174+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:58.175274+0000) 2024-04-27T03:12:58.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:58.272+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:58.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:58.272+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:58.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:58.272+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:58.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:58.274+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:58.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:58.302+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:58.303447+0000) 2024-04-27T03:12:58.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:58.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:58.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:58.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:58.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:58.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:58.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:58.337+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:58.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:58.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:58.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:58.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:58.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:58.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:58.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:58.595+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:59.144 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:12:58.880+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55955 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:12:59.144 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:58.877+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:58.878751+0000) 2024-04-27T03:12:59.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:58.877+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:58.877847+0000) 2024-04-27T03:12:59.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:58.878+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:58.879309+0000) 2024-04-27T03:12:59.458 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:59.174+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:59.175439+0000) 2024-04-27T03:12:59.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:59.158+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:59.159295+0000) 2024-04-27T03:12:59.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:59.260+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:59.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:59.260+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:59.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:59.260+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:12:59.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:12:59.261+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:59.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:59.302+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:12:59.303618+0000) 2024-04-27T03:12:59.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:59.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:59.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:59.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:59.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:59.340+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:12:59.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:12:59.340+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:12:59.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:59.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:59.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:59.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:59.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:59.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:12:59.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:12:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:12:59.559+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:00.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:00.161+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:00.162206+0000) 2024-04-27T03:13:00.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:00.177+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:00.178316+0000) 2024-04-27T03:13:00.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:00.217+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:00.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:00.217+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:00.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:00.217+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:00.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:00.218+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:00.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:00.304+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:00.305758+0000) 2024-04-27T03:13:00.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:00.385+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:00.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:00.385+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:00.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:00.385+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:00.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:00.386+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:00.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:00.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:00.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:00.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:00.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:00.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:00.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:00.536+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:01.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:01.163+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:01.164266+0000) 2024-04-27T03:13:01.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:01.180+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:01.181264+0000) 2024-04-27T03:13:01.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:01.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:01.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:01.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:01.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:01.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:01.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:01.513+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:01.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:01.264+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:01.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:01.264+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:01.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:01.264+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:01.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:01.265+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:01.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:01.307+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:01.308561+0000) 2024-04-27T03:13:01.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:01.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:01.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:01.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:01.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:01.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:01.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:01.408+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:02.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:02.165+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:02.166563+0000) 2024-04-27T03:13:02.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:02.183+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:02.183904+0000) 2024-04-27T03:13:02.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:02.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:02.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:02.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:02.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:02.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:02.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:02.277+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:02.577 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:02.310+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:02.310895+0000) 2024-04-27T03:13:02.577 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:02.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:02.577 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:02.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:02.577 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:02.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:02.577 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:02.456+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:02.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:02.505+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:02.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:02.505+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:02.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:02.505+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:02.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:02.505+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:03.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:03.165+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:03.166743+0000) 2024-04-27T03:13:03.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:03.183+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:03.184071+0000) 2024-04-27T03:13:03.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:03.310+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:03.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:03.310+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:03.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:03.310+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:03.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:03.311+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:03.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:03.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:03.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:03.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:03.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:03.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:03.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:03.498+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:03.577 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:03.310+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:03.311083+0000) 2024-04-27T03:13:03.577 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:03.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:03.577 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:03.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:03.577 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:03.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:03.578 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:03.488+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:04.144 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:13:03.887+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 55994 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:13:04.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:03.885+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:03.886070+0000) 2024-04-27T03:13:04.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:03.884+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:03.885511+0000) 2024-04-27T03:13:04.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:03.884+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:03.885616+0000) 2024-04-27T03:13:04.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:04.310+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:04.311294+0000) 2024-04-27T03:13:04.464 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:04.183+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:04.184263+0000) 2024-04-27T03:13:04.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:04.166+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:04.166939+0000) 2024-04-27T03:13:04.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:04.301+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:04.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:04.301+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:04.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:04.301+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:04.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:04.302+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:04.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:04.471+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:04.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:04.471+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:04.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:04.471+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:04.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:04.472+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:04.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:04.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:04.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:04.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:04.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:04.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:04.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:04.464+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:05.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:05.168+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:05.169195+0000) 2024-04-27T03:13:05.466 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:05.185+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:05.186581+0000) 2024-04-27T03:13:05.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:05.314+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:05.315034+0000) 2024-04-27T03:13:05.469 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:05.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:05.469 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:05.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:05.469 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:05.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:05.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:05.257+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:05.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:05.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:05.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:05.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:05.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:05.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:05.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:05.461+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:05.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:05.479+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:05.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:05.479+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:05.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:05.479+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:05.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:05.480+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:06.199 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:06.172+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:06.173444+0000) 2024-04-27T03:13:06.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:06.189+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:06.190776+0000) 2024-04-27T03:13:06.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:06.319+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:06.319846+0000) 2024-04-27T03:13:06.469 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:06.217+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:06.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:06.217+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:06.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:06.217+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:06.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:06.217+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:06.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:06.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:06.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:06.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:06.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:06.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:06.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:06.498+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:06.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:06.473+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:06.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:06.473+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:06.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:06.473+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:06.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:06.474+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:07.201 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:07.175+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:07.175818+0000) 2024-04-27T03:13:07.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:07.321+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:07.322371+0000) 2024-04-27T03:13:07.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:07.191+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:07.192658+0000) 2024-04-27T03:13:07.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:07.221+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:07.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:07.221+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:07.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:07.221+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:07.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:07.222+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:07.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:07.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:07.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:07.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:07.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:07.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:07.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:07.504+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:07.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:07.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:07.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:07.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:07.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:07.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:07.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:07.522+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:08.202 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:08.175+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:08.175994+0000) 2024-04-27T03:13:08.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:08.321+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:08.322551+0000) 2024-04-27T03:13:08.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:08.192+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:08.192840+0000) 2024-04-27T03:13:08.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:08.239+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:08.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:08.239+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:08.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:08.239+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:08.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:08.239+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:08.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:08.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:08.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:08.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:08.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:08.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:08.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:08.534+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:08.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:08.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:08.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:08.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:08.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:08.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:08.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:08.519+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:09.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:08.892+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:08.893355+0000) 2024-04-27T03:13:09.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:08.891+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:08.892652+0000) 2024-04-27T03:13:09.186 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:13:08.893+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56033 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:13:09.186 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:08.891+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:08.892307+0000) 2024-04-27T03:13:09.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:09.321+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:09.322735+0000) 2024-04-27T03:13:09.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:09.175+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:09.176234+0000) 2024-04-27T03:13:09.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:09.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:09.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:09.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:09.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:09.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:09.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:09.290+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:09.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:09.192+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:09.193005+0000) 2024-04-27T03:13:09.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:09.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:09.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:09.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:09.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:09.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:09.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:09.531+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:09.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:09.522+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:09.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:09.522+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:09.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:09.522+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:09.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:09.523+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:10.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:10.178+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:10.178911+0000) 2024-04-27T03:13:10.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:10.194+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:10.195307+0000) 2024-04-27T03:13:10.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:10.324+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:10.325426+0000) 2024-04-27T03:13:10.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:10.261+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:10.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:10.261+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:10.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:10.261+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:10.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:10.262+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:10.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:10.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:10.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:10.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:10.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:10.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:10.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:10.577+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:10.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:10.507+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:10.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:10.507+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:10.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:10.507+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:10.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:10.508+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:11.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:11.196+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:11.197540+0000) 2024-04-27T03:13:11.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:11.327+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:11.328314+0000) 2024-04-27T03:13:11.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:11.180+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:11.181575+0000) 2024-04-27T03:13:11.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:11.224+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:11.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:11.224+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:11.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:11.224+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:11.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:11.224+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:11.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:11.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:11.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:11.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:11.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:11.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:11.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:11.565+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:11.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:11.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:11.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:11.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:11.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:11.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:11.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:11.470+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:12.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:12.177+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:12.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:12.177+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:12.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:12.177+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:12.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:12.177+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:12.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:12.182+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:12.183401+0000) 2024-04-27T03:13:12.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:12.200+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:12.201167+0000) 2024-04-27T03:13:12.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:12.331+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:12.331833+0000) 2024-04-27T03:13:12.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:12.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:12.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:12.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:12.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:12.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:12.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:12.542+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:12.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:12.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:12.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:12.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:12.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:12.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:12.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:12.493+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:13.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:13.182+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:13.183587+0000) 2024-04-27T03:13:13.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:13.200+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:13.201334+0000) 2024-04-27T03:13:13.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:13.331+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:13.332055+0000) 2024-04-27T03:13:13.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:13.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:13.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:13.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:13.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:13.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:13.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:13.194+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:13.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:13.471+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:13.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:13.471+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:13.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:13.471+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:13.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:13.471+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:13.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:13.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:13.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:13.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:13.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:13.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:13.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:13.572+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:14.099 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:13:13.901+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56072 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:13:14.099 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:13.899+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:13.899897+0000) 2024-04-27T03:13:14.100 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:13.898+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:13.899712+0000) 2024-04-27T03:13:14.100 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:13.898+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:13.898943+0000) 2024-04-27T03:13:14.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:14.200+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:14.201524+0000) 2024-04-27T03:13:14.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:14.331+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:14.332235+0000) 2024-04-27T03:13:14.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:14.155+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:14.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:14.155+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:14.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:14.155+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:14.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:14.156+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:14.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:14.182+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:14.183703+0000) 2024-04-27T03:13:14.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:14.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:14.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:14.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:14.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:14.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:14.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:14.597+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:14.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:14.521+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:14.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:14.521+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:14.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:14.521+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:14.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:14.521+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:15.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:15.143+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:15.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:15.143+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:15.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:15.143+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:15.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:15.144+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:15.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:15.333+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:15.334404+0000) 2024-04-27T03:13:15.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:15.187+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:15.188319+0000) 2024-04-27T03:13:15.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:15.202+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:15.203795+0000) 2024-04-27T03:13:15.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:15.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:15.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:15.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:15.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:15.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:15.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:15.523+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:15.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:15.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:15.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:15.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:15.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:15.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:15.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:15.597+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:16.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:16.178+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:16.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:16.179+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:16.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:16.179+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:16.200 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:16.180+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:16.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:16.337+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:16.338698+0000) 2024-04-27T03:13:16.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:16.190+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:16.191401+0000) 2024-04-27T03:13:16.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:16.205+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:16.205895+0000) 2024-04-27T03:13:16.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:16.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:16.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:16.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:16.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:16.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:16.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:16.573+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:16.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:16.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:16.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:16.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:16.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:16.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:16.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:16.540+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:17.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:17.209+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:17.209898+0000) 2024-04-27T03:13:17.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:17.339+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:17.340638+0000) 2024-04-27T03:13:17.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:17.191+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:17.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:17.191+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:17.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:17.191+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:17.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:17.192+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:17.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:17.192+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:17.193513+0000) 2024-04-27T03:13:17.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:17.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:17.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:17.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:17.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:17.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:17.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:17.569+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:17.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:17.556+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:17.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:17.556+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:17.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:17.556+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:17.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:17.557+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:18.202 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:18.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:18.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:18.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:18.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:18.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:18.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:18.171+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:18.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:18.192+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:18.193713+0000) 2024-04-27T03:13:18.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:18.340+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:18.340815+0000) 2024-04-27T03:13:18.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:18.209+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:18.210061+0000) 2024-04-27T03:13:18.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:18.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:18.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:18.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:18.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:18.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:18.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:18.568+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:18.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:18.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:18.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:18.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:18.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:18.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:18.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:18.569+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:19.203 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:13:18.908+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56111 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:13:19.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:18.905+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:18.906204+0000) 2024-04-27T03:13:19.203 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:18.905+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:18.906778+0000) 2024-04-27T03:13:19.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:18.905+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:18.906477+0000) 2024-04-27T03:13:19.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:19.122+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:19.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:19.122+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:19.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:19.122+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:19.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:19.124+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:19.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:19.193+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:19.193918+0000) 2024-04-27T03:13:19.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:19.209+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:19.210245+0000) 2024-04-27T03:13:19.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:19.340+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:19.340992+0000) 2024-04-27T03:13:19.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:19.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:19.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:19.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:19.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:19.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:19.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:19.556+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:19.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:19.600+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:19.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:19.600+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:19.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:19.600+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:19.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:19.601+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:20.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:20.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:20.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:20.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:20.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:20.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:20.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:20.130+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:20.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:20.196+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:20.197071+0000) 2024-04-27T03:13:20.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:20.211+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:20.212446+0000) 2024-04-27T03:13:20.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:20.343+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:20.343831+0000) 2024-04-27T03:13:20.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:20.509+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:20.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:20.509+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:20.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:20.509+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:20.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:20.510+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:20.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:20.581+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:20.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:20.581+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:20.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:20.581+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:20.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:20.581+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:21.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:21.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:21.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:21.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:21.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:21.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:21.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:21.162+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:21.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:21.200+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:21.201752+0000) 2024-04-27T03:13:21.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:21.213+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:21.214555+0000) 2024-04-27T03:13:21.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:21.346+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:21.347063+0000) 2024-04-27T03:13:21.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:21.571+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:21.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:21.571+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:21.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:21.571+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:21.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:21.572+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:21.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:21.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:21.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:21.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:21.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:21.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:21.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:21.490+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:22.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:22.348+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:22.348939+0000) 2024-04-27T03:13:22.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:22.216+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:22.217019+0000) 2024-04-27T03:13:22.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:22.203+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:22.203950+0000) 2024-04-27T03:13:22.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:22.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:22.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:22.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:22.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:22.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:22.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:22.211+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:22.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:22.507+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:22.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:22.507+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:22.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:22.507+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:22.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:22.508+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:22.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:22.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:22.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:22.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:22.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:22.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:22.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:22.576+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:23.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:23.348+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:23.349091+0000) 2024-04-27T03:13:23.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:23.216+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:23.217195+0000) 2024-04-27T03:13:23.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:23.203+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:23.204038+0000) 2024-04-27T03:13:23.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:23.208+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:23.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:23.208+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:23.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:23.208+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:23.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:23.209+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:23.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:23.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:23.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:23.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:23.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:23.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:23.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:23.492+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:23.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:23.626+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:23.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:23.626+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:23.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:23.626+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:23.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:23.626+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:24.044 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:13:23.914+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56150 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:13:24.045 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:23.913+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:23.914680+0000) 2024-04-27T03:13:24.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:23.913+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:23.914008+0000) 2024-04-27T03:13:24.045 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:23.912+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:23.913451+0000) 2024-04-27T03:13:24.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:24.216+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:24.217383+0000) 2024-04-27T03:13:24.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:24.203+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:24.204249+0000) 2024-04-27T03:13:24.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:24.243+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:24.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:24.243+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:24.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:24.243+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:24.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:24.244+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:24.644 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:24.348+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:24.349242+0000) 2024-04-27T03:13:24.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:24.634+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:24.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:24.476+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:24.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:24.476+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:24.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:24.476+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:24.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:24.477+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:24.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:24.634+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:24.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:24.634+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:24.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:24.635+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:25.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:25.352+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:25.352903+0000) 2024-04-27T03:13:25.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:25.205+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:25.206387+0000) 2024-04-27T03:13:25.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:25.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:25.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:25.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:25.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:25.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:25.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:25.254+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:25.481 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:25.219+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:25.220674+0000) 2024-04-27T03:13:25.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:25.517+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:25.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:25.517+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:25.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:25.517+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:25.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:25.518+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:25.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:25.628+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:25.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:25.628+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:25.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:25.628+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:25.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:25.629+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:26.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:26.354+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:26.355103+0000) 2024-04-27T03:13:26.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:26.223+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:26.223987+0000) 2024-04-27T03:13:26.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:26.208+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:26.209509+0000) 2024-04-27T03:13:26.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:26.262+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:26.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:26.262+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:26.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:26.262+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:26.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:26.263+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:26.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:26.583+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:26.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:26.583+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:26.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:26.583+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:26.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:26.584+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:26.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:26.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:26.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:26.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:26.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:26.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:26.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:26.512+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:27.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:27.357+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:27.358568+0000) 2024-04-27T03:13:27.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:27.225+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:27.226274+0000) 2024-04-27T03:13:27.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:27.211+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:27.211974+0000) 2024-04-27T03:13:27.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:27.263+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:27.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:27.263+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:27.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:27.263+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:27.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:27.265+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:27.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:27.546+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:27.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:27.546+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:27.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:27.546+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:27.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:27.547+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:27.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:27.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:27.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:27.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:27.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:27.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:27.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:27.521+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:28.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:28.357+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:28.358830+0000) 2024-04-27T03:13:28.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:28.224+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:28.226496+0000) 2024-04-27T03:13:28.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:28.210+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:28.212177+0000) 2024-04-27T03:13:28.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:28.238+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:28.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:28.238+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:28.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:28.238+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:28.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:28.239+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:28.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:28.553+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:28.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:28.553+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:28.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:28.553+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:28.927 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:28.554+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:28.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:28.493+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:28.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:28.493+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:28.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:28.493+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:28.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:28.494+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:29.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:28.918+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:28.920238+0000) 2024-04-27T03:13:29.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:28.920+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:28.922242+0000) 2024-04-27T03:13:29.222 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:13:28.920+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56189 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:13:29.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:28.918+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:28.920628+0000) 2024-04-27T03:13:29.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:29.357+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:29.359003+0000) 2024-04-27T03:13:29.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:29.224+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:29.226682+0000) 2024-04-27T03:13:29.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:29.210+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:29.212356+0000) 2024-04-27T03:13:29.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:29.260+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:29.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:29.260+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:29.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:29.260+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:29.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:29.261+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:29.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:29.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:29.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:29.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:29.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:29.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:29.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:29.568+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:29.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:29.539+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:29.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:29.539+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:29.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:29.539+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:29.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:29.540+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:30.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:30.227+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:30.228908+0000) 2024-04-27T03:13:30.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:30.361+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:30.363710+0000) 2024-04-27T03:13:30.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:30.213+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:30.214920+0000) 2024-04-27T03:13:30.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:30.262+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:30.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:30.262+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:30.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:30.262+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:30.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:30.263+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:30.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:30.548+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:30.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:30.548+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:30.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:30.548+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:30.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:30.548+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:30.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:30.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:30.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:30.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:30.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:30.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:30.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:30.550+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:31.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:31.230+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:31.232084+0000) 2024-04-27T03:13:31.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:31.364+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:31.366073+0000) 2024-04-27T03:13:31.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:31.217+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:31.219100+0000) 2024-04-27T03:13:31.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:31.239+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:31.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:31.239+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:31.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:31.239+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:31.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:31.240+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:31.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:31.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:31.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:31.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:31.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:31.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:31.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:31.519+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:31.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:31.542+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:31.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:31.542+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:31.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:31.542+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:31.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:31.542+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:32.485 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:32.233+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:32.235747+0000) 2024-04-27T03:13:32.485 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:32.220+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:32.222483+0000) 2024-04-27T03:13:32.485 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:32.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:32.485 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:32.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:32.485 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:32.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:32.486 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:32.257+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:32.486 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:32.368+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:32.370208+0000) 2024-04-27T03:13:32.486 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:32.469+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:32.486 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:32.469+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:32.486 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:32.469+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:32.486 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:32.470+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:32.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:32.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:32.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:32.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:32.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:32.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:32.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:32.533+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:33.485 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:33.234+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:33.235976+0000) 2024-04-27T03:13:33.485 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:33.368+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:33.370449+0000) 2024-04-27T03:13:33.486 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:33.220+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:33.222678+0000) 2024-04-27T03:13:33.486 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:33.285+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:33.486 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:33.285+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:33.486 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:33.285+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:33.486 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:33.286+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:33.789 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:33.512+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:33.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:33.512+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:33.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:33.512+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:33.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:33.513+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:33.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:33.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:33.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:33.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:33.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:33.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:33.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:33.565+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:34.061 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:13:33.928+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56228 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:13:34.061 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:33.926+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:33.928228+0000) 2024-04-27T03:13:34.061 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:33.925+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:33.927179+0000) 2024-04-27T03:13:34.061 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:33.925+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:33.926815+0000) 2024-04-27T03:13:34.380 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:34.234+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:34.236145+0000) 2024-04-27T03:13:34.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:34.221+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:34.222824+0000) 2024-04-27T03:13:34.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:34.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:34.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:34.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:34.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:34.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:34.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:34.277+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:34.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:34.368+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:34.370695+0000) 2024-04-27T03:13:34.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:34.513+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:34.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:34.513+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:34.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:34.513+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:34.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:34.514+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:34.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:34.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:34.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:34.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:34.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:34.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:34.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:34.590+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:35.488 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:35.371+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:35.373408+0000) 2024-04-27T03:13:35.488 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:35.238+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:35.240409+0000) 2024-04-27T03:13:35.488 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:35.225+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:35.227591+0000) 2024-04-27T03:13:35.488 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:35.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:35.488 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:35.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:35.488 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:35.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:35.488 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:35.323+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:35.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:35.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:35.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:35.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:35.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:35.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:35.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:35.639+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:35.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:35.487+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:35.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:35.487+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:35.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:35.487+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:35.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:35.488+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:36.535 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:36.374+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:36.375820+0000) 2024-04-27T03:13:36.535 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:36.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:36.535 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:36.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:36.535 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:36.230+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:36.231835+0000) 2024-04-27T03:13:36.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:36.308+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:36.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:36.308+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:36.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:36.308+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:36.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:36.309+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:36.536 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:36.240+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:36.242663+0000) 2024-04-27T03:13:36.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:36.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:36.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:36.523+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:36.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:36.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:36.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:36.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:36.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:36.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:36.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:36.650+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:37.559 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:37.376+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:37.377810+0000) 2024-04-27T03:13:37.559 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:37.548+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:37.560 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:37.231+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:37.233668+0000) 2024-04-27T03:13:37.560 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:37.318+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:37.560 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:37.318+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:37.560 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:37.318+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:37.560 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:37.319+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:37.560 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:37.243+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:37.245267+0000) 2024-04-27T03:13:37.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:37.548+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:37.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:37.548+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:37.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:37.549+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:37.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:37.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:37.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:37.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:37.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:37.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:37.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:37.638+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:38.539 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:38.243+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:38.245436+0000) 2024-04-27T03:13:38.539 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:38.376+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:38.377992+0000) 2024-04-27T03:13:38.540 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:38.232+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:38.233845+0000) 2024-04-27T03:13:38.540 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:38.302+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:38.540 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:38.302+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:38.540 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:38.302+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:38.540 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:38.303+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:38.940 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:38.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:38.940 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:38.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:38.940 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:38.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:38.941 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:38.529+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:38.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:38.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:38.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:38.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:38.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:38.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:38.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:38.633+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:39.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:38.931+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:38.933758+0000) 2024-04-27T03:13:39.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:38.932+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:38.934691+0000) 2024-04-27T03:13:39.230 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:13:38.935+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56267 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:13:39.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:38.934+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:38.936577+0000) 2024-04-27T03:13:39.504 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:39.243+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:39.245617+0000) 2024-04-27T03:13:39.504 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:39.377+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:39.378249+0000) 2024-04-27T03:13:39.504 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:39.232+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:39.234036+0000) 2024-04-27T03:13:39.504 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:39.292+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:39.504 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:39.292+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:39.504 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:39.292+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:39.504 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:39.293+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:39.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:39.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:39.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:39.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:39.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:39.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:39.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:39.493+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:39.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:39.654+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:39.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:39.654+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:39.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:39.654+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:39.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:39.654+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:40.580 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:40.247+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:40.247829+0000) 2024-04-27T03:13:40.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:40.234+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:40.236148+0000) 2024-04-27T03:13:40.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:40.286+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:40.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:40.286+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:40.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:40.286+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:40.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:40.286+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:40.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:40.378+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:40.380658+0000) 2024-04-27T03:13:40.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:40.479+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:40.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:40.479+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:40.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:40.479+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:40.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:40.479+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:40.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:40.701+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:40.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:40.701+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:40.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:40.701+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:40.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:40.702+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:41.514 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:41.249+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:41.249879+0000) 2024-04-27T03:13:41.515 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:41.237+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:41.238236+0000) 2024-04-27T03:13:41.515 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:41.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:41.515 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:41.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:41.515 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:41.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:41.515 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:41.313+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:41.515 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:41.382+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:41.382807+0000) 2024-04-27T03:13:41.515 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:41.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:41.516 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:41.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:41.516 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:41.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:41.516 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:41.484+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:41.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:41.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:41.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:41.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:41.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:41.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:41.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:41.740+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:42.580 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:42.250+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:42.251748+0000) 2024-04-27T03:13:42.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:42.239+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:42.240069+0000) 2024-04-27T03:13:42.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:42.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:42.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:42.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:42.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:42.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:42.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:42.354+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:42.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:42.384+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:42.385267+0000) 2024-04-27T03:13:42.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:42.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:42.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:42.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:42.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:42.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:42.582 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:42.459+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:42.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:42.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:42.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:42.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:42.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:42.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:42.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:42.787+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:43.505 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:43.384+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:43.385546+0000) 2024-04-27T03:13:43.506 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:43.239+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:43.240278+0000) 2024-04-27T03:13:43.506 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:43.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:43.506 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:43.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:43.506 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:43.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:43.506 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:43.335+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:43.506 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:43.251+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:43.251977+0000) 2024-04-27T03:13:43.781 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:43.495+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:43.781 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:43.495+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:43.781 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:43.495+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:43.781 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:43.496+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:44.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:43.941+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:43.942013+0000) 2024-04-27T03:13:44.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:43.940+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:43.941610+0000) 2024-04-27T03:13:44.036 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:13:43.943+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56306 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:13:44.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:43.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:44.037 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:43.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:44.037 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:43.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:44.037 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:43.772+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:44.037 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:43.941+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:43.942280+0000) 2024-04-27T03:13:44.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:44.251+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:44.252224+0000) 2024-04-27T03:13:44.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:44.239+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:44.240466+0000) 2024-04-27T03:13:44.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:44.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:44.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:44.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:44.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:44.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:44.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:44.323+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:44.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:44.384+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:44.385730+0000) 2024-04-27T03:13:44.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:44.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:44.584 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:44.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:44.584 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:44.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:44.584 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:44.501+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:44.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:44.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:44.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:44.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:44.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:44.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:44.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:44.741+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:45.508 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:45.255+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:45.256483+0000) 2024-04-27T03:13:45.509 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:45.242+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:45.243268+0000) 2024-04-27T03:13:45.509 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:45.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:45.509 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:45.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:45.509 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:45.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:45.509 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:45.324+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:45.509 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:45.388+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:45.388942+0000) 2024-04-27T03:13:45.510 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:45.498+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:45.510 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:45.498+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:45.510 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:45.498+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:45.764 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:45.499+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:46.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:45.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:46.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:45.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:46.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:45.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:46.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:45.754+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:46.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:46.258+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:46.259322+0000) 2024-04-27T03:13:46.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:46.244+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:46.245381+0000) 2024-04-27T03:13:46.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:46.342+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:46.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:46.342+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:46.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:46.342+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:46.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:46.342+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:46.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:46.392+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:46.393202+0000) 2024-04-27T03:13:46.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:46.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:46.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:46.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:46.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:46.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:46.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:46.530+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:47.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:46.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:47.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:46.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:47.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:46.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:47.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:46.742+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:47.387 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:47.246+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:47.247176+0000) 2024-04-27T03:13:47.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:47.261+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:47.261830+0000) 2024-04-27T03:13:47.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:47.395+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:47.396028+0000) 2024-04-27T03:13:47.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:47.496+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:47.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:47.496+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:47.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:47.496+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:47.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:47.496+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:47.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:47.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:47.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:47.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:47.657 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:47.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:47.657 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:47.378+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:47.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:47.697+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:47.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:47.697+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:47.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:47.697+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:47.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:47.698+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:48.395 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:48.246+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:48.247357+0000) 2024-04-27T03:13:48.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:48.261+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:48.262011+0000) 2024-04-27T03:13:48.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:48.384+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:48.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:48.384+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:48.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:48.384+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:48.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:48.385+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:48.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:48.395+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:48.396219+0000) 2024-04-27T03:13:48.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:48.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:48.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:48.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:48.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:48.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:48.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:48.539+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:48.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:48.675+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:48.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:48.675+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:48.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:48.675+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:48.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:48.677+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:49.230 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:13:48.949+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56346 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:13:49.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:48.948+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:48.948893+0000) 2024-04-27T03:13:49.231 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:48.947+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:48.948679+0000) 2024-04-27T03:13:49.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:48.949+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:48.949840+0000) 2024-04-27T03:13:49.578 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:49.395+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:49.396387+0000) 2024-04-27T03:13:49.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:49.246+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:49.247549+0000) 2024-04-27T03:13:49.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:49.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:49.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:49.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:49.579 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:49.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:49.579 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:49.420+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:49.579 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:49.261+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:49.262204+0000) 2024-04-27T03:13:49.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:49.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:49.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:49.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:49.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:49.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:49.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:49.643+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:49.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:49.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:49.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:49.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:49.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:49.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:49.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:49.575+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:50.499 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:50.251+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:50.252426+0000) 2024-04-27T03:13:50.499 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:50.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:50.499 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:50.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:50.499 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:50.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:50.499 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:50.431+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:50.499 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:50.263+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:50.264296+0000) 2024-04-27T03:13:50.500 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:50.398+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:50.399709+0000) 2024-04-27T03:13:50.792 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:50.590+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:50.792 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:50.590+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:50.792 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:50.590+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:50.792 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:50.590+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:50.793 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:50.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:50.793 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:50.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:50.793 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:50.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:50.793 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:50.673+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:51.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:51.401+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:51.401845+0000) 2024-04-27T03:13:51.582 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:51.265+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:51.266440+0000) 2024-04-27T03:13:51.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:51.254+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:51.255206+0000) 2024-04-27T03:13:51.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:51.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:51.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:51.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:51.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:51.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:51.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:51.405+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:51.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:51.629+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:51.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:51.629+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:51.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:51.629+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:51.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:51.629+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:51.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:51.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:51.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:51.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:51.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:51.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:51.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:51.716+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:52.582 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:52.268+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:52.268841+0000) 2024-04-27T03:13:52.582 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:52.403+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:52.403953+0000) 2024-04-27T03:13:52.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:52.258+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:52.258906+0000) 2024-04-27T03:13:52.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:52.396+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:52.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:52.396+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:52.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:52.396+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:52.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:52.397+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:52.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:52.616+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:52.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:52.616+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:52.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:52.616+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:52.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:52.617+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:52.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:52.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:52.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:52.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:52.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:52.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:52.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:52.719+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:53.582 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:53.268+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:53.269030+0000) 2024-04-27T03:13:53.582 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:53.403+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:53.404131+0000) 2024-04-27T03:13:53.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:53.258+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:53.259080+0000) 2024-04-27T03:13:53.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:53.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:53.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:53.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:53.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:53.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:53.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:53.421+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:53.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:53.650+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:53.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:53.650+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:53.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:53.650+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:53.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:53.651+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:53.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:53.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:53.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:53.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:53.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:53.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:53.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:53.675+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:54.230 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:13:53.956+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56384 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:13:54.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:53.954+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:53.955230+0000) 2024-04-27T03:13:54.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:53.954+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:53.955534+0000) 2024-04-27T03:13:54.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:53.956+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:53.956903+0000) 2024-04-27T03:13:54.582 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:54.268+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:54.269286+0000) 2024-04-27T03:13:54.582 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:54.403+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:54.404300+0000) 2024-04-27T03:13:54.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:54.258+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:54.259269+0000) 2024-04-27T03:13:54.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:54.469+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:54.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:54.469+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:54.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:54.469+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:54.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:54.470+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:54.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:54.697+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:54.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:54.697+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:54.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:54.697+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:54.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:54.698+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:54.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:54.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:54.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:54.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:54.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:54.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:54.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:54.667+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:55.582 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:55.406+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:55.406901+0000) 2024-04-27T03:13:55.582 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:55.270+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:55.271489+0000) 2024-04-27T03:13:55.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:55.260+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:55.261340+0000) 2024-04-27T03:13:55.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:55.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:55.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:55.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:55.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:55.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:55.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:55.478+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:55.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:55.695+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:55.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:55.695+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:55.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:55.695+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:55.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:55.695+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:55.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:55.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:55.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:55.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:55.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:55.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:55.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:55.680+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:56.502 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:56.262+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:56.263611+0000) 2024-04-27T03:13:56.502 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:56.272+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:56.273580+0000) 2024-04-27T03:13:56.502 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:56.409+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:56.410092+0000) 2024-04-27T03:13:56.872 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:56.510+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:56.872 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:56.510+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:56.872 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:56.510+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:56.872 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:56.511+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:56.873 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:56.704+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:56.873 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:56.704+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:56.873 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:56.704+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:56.873 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:56.704+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:56.873 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:56.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:56.873 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:56.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:56.873 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:56.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:56.873 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:56.721+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:57.534 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:57.264+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:57.265627+0000) 2024-04-27T03:13:57.535 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:57.274+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:57.275613+0000) 2024-04-27T03:13:57.535 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:57.411+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:57.411979+0000) 2024-04-27T03:13:57.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:57.661+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:57.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:57.661+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:57.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:57.661+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:57.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:57.661+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:57.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:57.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:57.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:57.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:57.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:57.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:57.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:57.525+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:57.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:57.701+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:57.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:57.701+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:57.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:57.701+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:57.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:57.703+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:58.565 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:58.265+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:58.265827+0000) 2024-04-27T03:13:58.565 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:58.411+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:58.412161+0000) 2024-04-27T03:13:58.566 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:58.274+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:58.275772+0000) 2024-04-27T03:13:58.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:58.555+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:58.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:58.555+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:58.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:58.555+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:58.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:58.556+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:58.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:58.736+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:58.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:58.736+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:58.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:58.736+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:58.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:58.736+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:58.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:58.690+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:58.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:58.690+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:58.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:58.690+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:58.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:58.691+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:59.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:58.961+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:58.962582+0000) 2024-04-27T03:13:59.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:58.961+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:58.962478+0000) 2024-04-27T03:13:59.230 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:13:58.963+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56424 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:13:59.231 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:58.961+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:58.962787+0000) 2024-04-27T03:13:59.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:59.411+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:59.412406+0000) 2024-04-27T03:13:59.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:59.265+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:59.266016+0000) 2024-04-27T03:13:59.572 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:59.275+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:13:59.275954+0000) 2024-04-27T03:13:59.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:59.684+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:59.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:59.684+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:59.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:59.684+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:13:59.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:13:59.684+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:59.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:59.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:59.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:59.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:59.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:59.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:13:59.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:13:59.562+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:13:59.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:59.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:59.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:59.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:59.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:59.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:13:59.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:13:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:13:59.782+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:00.507 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:00.269+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:00.270209+0000) 2024-04-27T03:14:00.508 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:00.413+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:00.414574+0000) 2024-04-27T03:14:00.508 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:00.277+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:00.278072+0000) 2024-04-27T03:14:00.772 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:00.585+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:00.772 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:00.585+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:00.772 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:00.585+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:00.773 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:00.586+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:00.773 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:00.650+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:00.773 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:00.650+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:00.773 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:00.650+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:00.773 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:00.650+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:01.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:00.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:01.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:00.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:01.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:00.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:01.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:00.763+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:01.290 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:01.271+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:01.272327+0000) 2024-04-27T03:14:01.582 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:01.416+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:01.416818+0000) 2024-04-27T03:14:01.583 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:01.280+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:01.281183+0000) 2024-04-27T03:14:01.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:01.615+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:01.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:01.615+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:01.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:01.615+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:01.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:01.616+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:01.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:01.602+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:01.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:01.602+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:01.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:01.602+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:01.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:01.602+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:01.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:01.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:01.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:01.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:01.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:01.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:01.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:01.765+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:02.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:02.273+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:02.274217+0000) 2024-04-27T03:14:02.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:02.282+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:02.283321+0000) 2024-04-27T03:14:02.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:02.419+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:02.419837+0000) 2024-04-27T03:14:02.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:02.560+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:02.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:02.560+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:02.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:02.560+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:02.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:02.560+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:02.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:02.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:02.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:02.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:02.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:02.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:02.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:02.807+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:02.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:02.641+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:02.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:02.641+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:02.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:02.641+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:02.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:02.643+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:03.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:03.273+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:03.274407+0000) 2024-04-27T03:14:03.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:03.419+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:03.420057+0000) 2024-04-27T03:14:03.583 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:03.282+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:03.283551+0000) 2024-04-27T03:14:03.855 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:03.591+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:03.855 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:03.591+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:03.856 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:03.591+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:03.856 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:03.591+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:03.856 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:03.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:03.856 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:03.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:03.857 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:03.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:03.857 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:03.655+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:04.151 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:14:03.971+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56462 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:14:04.152 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:03.968+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:03.969158+0000) 2024-04-27T03:14:04.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:03.967+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:03.968294+0000) 2024-04-27T03:14:04.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:03.845+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:04.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:03.845+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:04.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:03.845+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:04.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:03.846+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:04.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:03.968+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:03.969601+0000) 2024-04-27T03:14:04.429 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:04.273+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:04.274599+0000) 2024-04-27T03:14:04.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:04.282+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:04.283786+0000) 2024-04-27T03:14:04.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:04.419+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:04.420233+0000) 2024-04-27T03:14:04.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:04.634+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:04.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:04.634+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:04.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:04.634+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:04.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:04.635+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:04.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:04.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:04.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:04.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:04.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:04.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:04.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:04.674+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:04.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:04.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:04.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:04.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:04.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:04.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:04.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:04.820+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:05.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:05.276+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:05.277372+0000) 2024-04-27T03:14:05.584 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:05.285+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:05.285846+0000) 2024-04-27T03:14:05.584 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:05.422+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:05.423013+0000) 2024-04-27T03:14:05.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:05.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:05.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:05.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:05.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:05.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:05.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:05.628+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:05.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:05.618+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:05.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:05.618+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:05.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:05.618+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:05.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:05.618+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:05.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:05.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:05.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:05.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:05.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:05.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:05.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:05.805+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:06.512 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:06.278+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:06.279608+0000) 2024-04-27T03:14:06.512 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:06.288+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:06.289040+0000) 2024-04-27T03:14:06.513 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:06.424+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:06.425163+0000) 2024-04-27T03:14:06.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:06.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:06.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:06.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:06.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:06.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:06.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:06.582+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:06.831 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:06.616+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:06.831 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:06.616+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:06.831 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:06.616+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:06.831 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:06.616+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:07.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:06.819+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:07.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:06.819+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:07.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:06.819+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:07.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:06.821+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:07.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:07.282+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:07.283345+0000) 2024-04-27T03:14:07.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:07.290+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:07.291515+0000) 2024-04-27T03:14:07.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:07.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:07.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:07.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:07.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:07.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:07.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:07.564+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:07.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:07.426+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:07.427782+0000) 2024-04-27T03:14:07.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:07.637+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:07.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:07.637+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:07.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:07.637+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:07.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:07.638+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:08.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:07.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:08.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:07.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:08.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:07.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:08.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:07.805+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:08.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:08.282+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:08.283517+0000) 2024-04-27T03:14:08.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:08.290+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:08.291700+0000) 2024-04-27T03:14:08.690 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:08.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:08.690 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:08.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:08.690 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:08.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:08.690 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:08.548+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:08.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:08.427+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:08.428172+0000) 2024-04-27T03:14:08.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:08.679+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:08.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:08.679+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:08.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:08.679+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:08.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:08.680+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:08.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:08.848+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:08.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:08.848+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:08.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:08.848+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:08.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:08.848+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:09.230 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:14:08.978+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56502 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:14:09.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:08.975+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:08.976256+0000) 2024-04-27T03:14:09.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:08.977+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:08.978154+0000) 2024-04-27T03:14:09.231 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:08.976+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:08.977216+0000) 2024-04-27T03:14:09.561 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:09.291+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:09.291911+0000) 2024-04-27T03:14:09.561 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:09.427+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:09.428377+0000) 2024-04-27T03:14:09.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:09.282+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:09.283697+0000) 2024-04-27T03:14:09.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:09.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:09.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:09.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:09.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:09.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:09.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:09.552+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:09.883 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:09.718+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:09.883 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:09.718+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:09.883 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:09.718+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:09.883 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:09.719+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:10.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:09.872+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:10.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:09.872+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:10.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:09.872+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:10.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:09.873+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:10.440 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:10.285+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:10.285865+0000) 2024-04-27T03:14:10.440 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:10.294+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:10.295105+0000) 2024-04-27T03:14:10.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:10.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:10.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:10.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:10.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:10.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:10.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:10.587+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:10.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:10.430+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:10.431258+0000) 2024-04-27T03:14:10.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:10.702+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:10.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:10.702+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:10.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:10.702+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:10.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:10.703+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:10.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:10.863+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:10.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:10.863+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:10.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:10.863+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:10.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:10.864+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:11.560 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:11.287+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:11.288028+0000) 2024-04-27T03:14:11.561 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:11.297+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:11.297799+0000) 2024-04-27T03:14:11.561 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:11.435+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:11.435963+0000) 2024-04-27T03:14:11.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:11.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:11.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:11.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:11.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:11.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:11.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:11.551+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:11.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:11.673+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:11.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:11.673+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:11.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:11.673+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:11.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:11.673+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:12.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:11.909+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:12.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:11.909+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:12.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:11.909+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:12.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:11.909+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:12.516 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:12.290+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:12.290835+0000) 2024-04-27T03:14:12.517 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:12.438+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:12.438948+0000) 2024-04-27T03:14:12.517 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:12.300+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:12.300799+0000) 2024-04-27T03:14:12.880 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:12.666+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:12.880 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:12.666+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:12.880 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:12.666+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:12.880 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:12.667+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:12.880 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:12.580+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:12.880 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:12.580+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:12.880 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:12.580+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:12.880 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:12.581+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:13.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:12.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:13.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:12.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:13.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:12.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:13.153 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:12.870+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:13.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:13.290+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:13.291036+0000) 2024-04-27T03:14:13.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:13.300+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:13.301124+0000) 2024-04-27T03:14:13.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:13.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:13.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:13.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:13.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:13.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:13.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:13.572+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:13.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:13.438+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:13.439145+0000) 2024-04-27T03:14:13.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:13.660+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:13.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:13.660+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:13.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:13.660+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:13.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:13.660+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:14.106 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:14:13.984+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56540 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:14:14.106 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:13.982+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:13.983638+0000) 2024-04-27T03:14:14.106 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:13.983+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:13.984096+0000) 2024-04-27T03:14:14.106 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:13.861+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:14.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:13.861+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:14.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:13.861+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:14.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:13.861+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:14.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:13.982+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:13.983335+0000) 2024-04-27T03:14:14.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:14.290+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:14.291308+0000) 2024-04-27T03:14:14.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:14.300+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:14.301366+0000) 2024-04-27T03:14:14.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:14.438+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:14.439389+0000) 2024-04-27T03:14:14.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:14.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:14.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:14.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:14.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:14.566+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:14.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:14.566+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:14.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:14.566+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:14.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:14.567+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:14.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:14.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:14.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:14.711+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:14.980 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:14.883+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:14.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:14.883+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:14.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:14.883+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:14.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:14.885+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:15.584 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:15.440+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:15.441635+0000) 2024-04-27T03:14:15.585 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:15.303+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:15.304281+0000) 2024-04-27T03:14:15.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:15.294+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:15.295531+0000) 2024-04-27T03:14:15.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:15.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:15.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:15.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:15.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:15.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:15.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:15.613+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:15.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:15.759+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:15.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:15.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:15.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:15.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:15.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:15.760+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:16.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:15.895+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:16.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:15.895+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:16.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:15.895+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:16.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:15.896+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:16.584 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:16.305+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:16.306530+0000) 2024-04-27T03:14:16.585 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:16.445+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:16.446377+0000) 2024-04-27T03:14:16.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:16.298+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:16.299762+0000) 2024-04-27T03:14:16.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:16.594+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:16.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:16.594+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:16.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:16.594+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:16.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:16.595+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:16.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:16.725+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:16.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:16.725+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:16.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:16.725+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:16.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:16.726+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:17.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:16.906+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:17.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:16.906+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:17.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:16.906+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:17.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:16.908+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:17.580 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:17.307+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:17.308526+0000) 2024-04-27T03:14:17.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:17.449+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:17.450113+0000) 2024-04-27T03:14:17.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:17.300+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:17.301753+0000) 2024-04-27T03:14:17.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:17.570+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:17.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:17.570+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:17.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:17.570+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:17.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:17.571+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:17.895 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:17.731+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:17.895 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:17.731+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:17.895 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:17.731+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:17.895 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:17.732+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:18.153 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:17.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:18.153 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:17.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:18.153 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:17.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:18.153 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:17.885+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:18.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:18.301+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:18.301930+0000) 2024-04-27T03:14:18.459 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:18.307+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:18.308773+0000) 2024-04-27T03:14:18.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:18.449+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:18.450309+0000) 2024-04-27T03:14:18.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:18.700+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:18.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:18.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:18.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:18.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:18.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:18.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:18.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:18.602+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:18.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:18.875+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:18.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:18.875+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:18.982 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:18.875+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:18.982 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:18.876+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:18.982 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:18.700+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:18.982 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:18.700+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:18.982 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:18.701+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:19.311 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:14:18.991+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56576 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:14:19.312 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:18.990+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:18.991336+0000) 2024-04-27T03:14:19.312 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:18.989+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:18.990392+0000) 2024-04-27T03:14:19.312 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:18.989+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:18.990431+0000) 2024-04-27T03:14:19.583 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:19.308+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:19.308974+0000) 2024-04-27T03:14:19.584 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:19.449+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:19.450536+0000) 2024-04-27T03:14:19.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:19.301+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:19.302110+0000) 2024-04-27T03:14:19.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:19.573+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:19.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:19.573+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:19.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:19.573+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:19.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:19.574+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:19.844 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:19.704+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:19.844 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:19.704+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:19.844 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:19.704+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:19.844 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:19.704+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:20.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:19.833+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:20.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:19.833+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:20.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:19.833+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:20.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:19.834+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:20.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:20.304+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:20.304823+0000) 2024-04-27T03:14:20.463 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:20.311+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:20.312188+0000) 2024-04-27T03:14:20.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:20.453+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:20.453919+0000) 2024-04-27T03:14:20.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:20.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:20.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:20.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:20.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:20.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:20.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:20.527+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:21.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:20.720+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:21.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:20.720+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:21.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:20.720+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:21.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:20.721+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:21.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:20.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:21.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:20.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:21.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:20.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:21.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:20.882+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:21.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:21.306+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:21.306823+0000) 2024-04-27T03:14:21.585 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:21.455+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:21.456600+0000) 2024-04-27T03:14:21.585 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:21.314+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:21.314836+0000) 2024-04-27T03:14:21.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:21.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:21.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:21.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:21.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:21.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:21.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:21.575+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:21.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:21.746+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:21.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:21.746+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:21.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:21.746+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:21.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:21.746+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:21.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:21.922+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:21.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:21.922+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:21.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:21.922+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:22.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:21.923+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:22.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:22.308+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:22.308953+0000) 2024-04-27T03:14:22.586 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:22.317+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:22.317939+0000) 2024-04-27T03:14:22.586 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:22.458+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:22.459434+0000) 2024-04-27T03:14:22.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:22.605+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:22.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:22.605+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:22.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:22.605+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:22.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:22.605+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:22.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:22.699+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:22.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:22.699+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:22.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:22.699+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:22.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:22.700+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:23.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:22.957+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:23.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:22.957+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:23.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:22.957+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:23.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:22.958+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:23.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:23.308+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:23.309218+0000) 2024-04-27T03:14:23.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:23.458+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:23.459688+0000) 2024-04-27T03:14:23.581 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:23.317+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:23.318092+0000) 2024-04-27T03:14:23.954 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:23.702+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:23.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:23.702+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:23.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:23.702+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:23.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:23.703+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:23.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:23.570+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:23.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:23.570+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:23.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:23.570+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:23.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:23.572+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:24.230 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:14:23.999+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56614 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:14:24.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:23.996+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:23.997463+0000) 2024-04-27T03:14:24.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:23.995+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:23.996613+0000) 2024-04-27T03:14:24.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:23.986+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:24.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:23.986+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:24.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:23.986+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:24.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:23.987+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:24.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:23.996+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:23.997163+0000) 2024-04-27T03:14:24.585 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:24.317+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:24.318321+0000) 2024-04-27T03:14:24.586 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:24.459+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:24.459865+0000) 2024-04-27T03:14:24.586 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:24.308+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:24.309428+0000) 2024-04-27T03:14:24.586 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:24.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:24.586 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:24.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:24.586 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:24.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:24.586 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:24.549+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:24.927 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:24.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:24.927 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:24.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:24.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:24.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:24.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:24.665+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:25.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:25.017+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:25.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:25.017+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:25.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:25.018+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:25.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:25.019+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:25.580 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:25.320+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:25.320821+0000) 2024-04-27T03:14:25.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:25.461+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:25.462517+0000) 2024-04-27T03:14:25.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:25.310+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:25.311652+0000) 2024-04-27T03:14:25.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:25.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:25.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:25.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:25.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:25.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:25.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:25.570+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:25.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:25.690+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:25.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:25.690+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:25.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:25.690+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:25.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:25.691+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:26.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:26.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:26.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:26.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:26.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:26.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:26.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:26.043+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:26.526 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:26.464+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:26.464836+0000) 2024-04-27T03:14:26.527 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:26.313+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:26.313911+0000) 2024-04-27T03:14:26.527 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:26.322+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:26.322996+0000) 2024-04-27T03:14:26.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:26.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:26.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:26.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:26.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:26.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:26.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:26.572+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:26.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:26.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:26.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:26.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:26.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:26.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:26.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:26.729+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:27.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:27.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:27.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:27.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:27.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:27.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:27.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:27.002+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:27.586 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:27.466+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:27.467030+0000) 2024-04-27T03:14:27.586 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:27.326+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:27.326967+0000) 2024-04-27T03:14:27.586 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:27.314+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:27.315772+0000) 2024-04-27T03:14:27.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:27.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:27.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:27.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:27.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:27.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:27.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:27.550+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:27.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:27.683+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:27.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:27.683+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:27.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:27.683+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:27.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:27.684+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:28.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:28.040+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:28.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:28.040+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:28.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:28.040+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:28.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:28.041+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:28.586 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:28.466+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:28.467201+0000) 2024-04-27T03:14:28.608 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:28.326+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:28.327197+0000) 2024-04-27T03:14:28.608 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:28.315+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:28.315950+0000) 2024-04-27T03:14:28.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:28.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:28.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:28.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:28.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:28.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:28.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:28.504+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:28.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:28.647+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:28.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:28.647+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:28.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:28.647+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:28.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:28.648+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:29.230 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:14:29.006+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56654 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:14:29.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:29.005+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:29.006420+0000) 2024-04-27T03:14:29.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:29.003+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:29.004674+0000) 2024-04-27T03:14:29.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:29.004+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:29.005453+0000) 2024-04-27T03:14:29.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:29.011+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:29.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:29.011+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:29.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:29.011+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:29.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:29.012+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:29.586 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:29.466+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:29.467383+0000) 2024-04-27T03:14:29.586 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:29.326+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:29.327379+0000) 2024-04-27T03:14:29.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:29.315+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:29.316128+0000) 2024-04-27T03:14:29.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:29.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:29.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:29.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:29.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:29.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:29.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:29.469+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:29.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:29.686+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:29.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:29.686+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:29.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:29.686+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:29.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:29.687+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:30.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:30.003+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:30.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:30.003+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:30.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:30.003+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:30.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:30.004+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:30.531 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:30.470+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:30.470841+0000) 2024-04-27T03:14:30.531 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:30.331+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:30.332027+0000) 2024-04-27T03:14:30.531 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:30.319+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:30.320310+0000) 2024-04-27T03:14:30.532 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:30.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:30.532 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:30.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:30.532 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:30.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:30.532 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:30.456+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:30.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:30.688+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:30.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:30.688+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:30.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:30.688+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:30.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:30.689+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:31.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:31.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:31.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:31.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:31.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:31.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:31.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:31.017+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:31.586 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:31.336+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:31.336830+0000) 2024-04-27T03:14:31.587 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:31.473+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:31.474061+0000) 2024-04-27T03:14:31.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:31.321+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:31.322404+0000) 2024-04-27T03:14:31.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:31.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:31.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:31.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:31.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:31.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:31.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:31.425+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:31.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:31.646+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:31.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:31.646+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:31.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:31.646+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:31.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:31.646+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:32.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:32.004+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:32.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:32.004+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:32.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:32.004+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:32.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:32.005+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:32.586 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:32.474+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:32.476274+0000) 2024-04-27T03:14:32.587 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:32.338+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:32.340752+0000) 2024-04-27T03:14:32.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:32.322+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:32.324376+0000) 2024-04-27T03:14:32.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:32.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:32.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:32.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:32.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:32.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:32.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:32.415+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:32.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:32.640+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:32.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:32.640+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:32.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:32.640+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:32.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:32.641+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:33.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:32.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:33.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:32.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:33.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:32.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:33.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:32.996+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:33.587 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:33.474+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:33.476466+0000) 2024-04-27T03:14:33.587 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:33.339+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:33.340946+0000) 2024-04-27T03:14:33.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:33.322+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:33.324564+0000) 2024-04-27T03:14:33.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:33.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:33.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:33.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:33.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:33.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:33.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:33.432+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:33.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:33.598+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:33.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:33.598+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:33.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:33.598+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:33.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:33.598+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:34.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:34.010+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:34.012326+0000) 2024-04-27T03:14:34.230 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:14:34.012+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56692 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:14:34.231 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:34.010+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:34.012238+0000) 2024-04-27T03:14:34.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:33.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:34.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:33.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:34.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:33.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:34.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:33.996+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:34.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:34.012+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:34.014060+0000) 2024-04-27T03:14:34.587 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:34.474+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:34.476640+0000) 2024-04-27T03:14:34.587 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:34.339+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:34.341120+0000) 2024-04-27T03:14:34.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:34.322+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:34.324749+0000) 2024-04-27T03:14:34.588 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:34.409+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:34.588 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:34.409+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:34.588 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:34.409+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:34.588 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:34.410+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:34.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:34.644+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:34.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:34.644+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:34.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:34.644+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:34.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:34.645+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:35.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:34.953+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:35.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:34.953+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:35.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:34.953+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:35.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:34.953+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:35.641 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:35.327+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:35.329761+0000) 2024-04-27T03:14:35.641 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:35.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:35.641 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:35.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:35.641 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:35.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:35.641 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:35.403+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:35.642 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:35.341+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:35.343267+0000) 2024-04-27T03:14:35.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:35.476+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:35.478752+0000) 2024-04-27T03:14:35.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:35.629+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:35.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:35.629+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:35.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:35.629+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:35.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:35.629+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:36.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:35.951+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:36.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:35.951+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:36.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:35.951+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:36.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:35.952+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:36.637 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:36.346+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:36.348290+0000) 2024-04-27T03:14:36.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:36.480+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:36.482088+0000) 2024-04-27T03:14:36.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:36.626+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:36.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:36.626+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:36.638 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:36.626+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:36.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:36.330+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:36.331827+0000) 2024-04-27T03:14:36.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:36.391+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:36.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:36.391+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:36.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:36.391+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:36.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:36.392+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:36.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:36.626+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:37.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:36.966+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:37.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:36.966+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:37.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:36.966+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:37.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:36.966+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:37.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:37.482+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:37.484666+0000) 2024-04-27T03:14:37.637 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:37.349+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:37.350811+0000) 2024-04-27T03:14:37.637 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:37.332+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:37.334238+0000) 2024-04-27T03:14:37.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:37.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:37.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:37.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:37.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:37.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:37.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:37.364+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:37.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:37.625+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:37.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:37.625+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:37.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:37.625+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:37.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:37.626+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:38.155 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:37.988+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:38.155 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:37.988+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:38.155 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:37.988+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:38.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:37.990+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:38.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:38.349+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:38.350991+0000) 2024-04-27T03:14:38.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:38.332+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:38.334401+0000) 2024-04-27T03:14:38.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:38.365+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:38.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:38.365+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:38.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:38.365+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:38.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:38.366+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:38.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:38.483+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:38.484853+0000) 2024-04-27T03:14:38.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:38.648+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:38.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:38.648+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:38.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:38.648+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:38.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:38.648+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:39.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:39.017+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:39.019110+0000) 2024-04-27T03:14:39.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:39.017+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:39.018932+0000) 2024-04-27T03:14:39.231 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:14:39.019+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56732 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:14:39.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:39.016+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:39.017936+0000) 2024-04-27T03:14:39.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:39.023+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:39.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:39.023+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:39.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:39.023+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:39.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:39.024+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:39.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:39.349+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:39.351231+0000) 2024-04-27T03:14:39.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:39.483+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:39.485091+0000) 2024-04-27T03:14:39.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:39.600+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:39.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:39.600+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:39.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:39.332+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:39.334559+0000) 2024-04-27T03:14:39.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:39.348+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:39.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:39.348+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:39.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:39.348+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:39.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:39.348+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:39.880 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:39.600+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:39.880 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:39.601+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:40.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:40.055+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:40.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:40.055+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:40.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:40.055+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:40.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:40.056+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:40.537 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:40.353+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:40.355466+0000) 2024-04-27T03:14:40.538 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:40.485+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:40.487270+0000) 2024-04-27T03:14:40.538 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:40.331+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:40.538 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:40.331+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:40.538 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:40.331+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:40.538 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:40.332+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:40.538 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:40.335+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:40.337583+0000) 2024-04-27T03:14:40.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:40.650+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:40.957 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:40.650+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:40.957 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:40.650+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:40.957 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:40.651+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:41.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:41.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:41.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:41.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:41.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:41.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:41.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:41.014+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:41.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:41.355+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:41.357607+0000) 2024-04-27T03:14:41.682 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:41.488+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:41.490268+0000) 2024-04-27T03:14:41.682 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:41.670+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:41.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:41.327+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:41.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:41.327+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:41.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:41.327+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:41.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:41.328+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:41.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:41.338+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:41.340336+0000) 2024-04-27T03:14:41.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:41.670+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:41.957 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:41.670+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:41.957 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:41.671+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:42.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:42.031+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:42.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:42.031+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:42.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:42.031+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:42.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:42.032+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:42.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:42.357+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:42.359641+0000) 2024-04-27T03:14:42.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:42.492+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:42.494663+0000) 2024-04-27T03:14:42.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:42.716+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:42.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:42.716+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:42.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:42.716+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:42.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:42.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:42.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:42.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:42.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:42.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:42.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:42.340+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:42.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:42.341+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:42.342810+0000) 2024-04-27T03:14:42.980 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:42.716+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:43.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:43.034+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:43.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:43.034+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:43.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:43.034+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:43.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:43.035+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:43.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:43.358+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:43.359835+0000) 2024-04-27T03:14:43.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:43.493+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:43.494858+0000) 2024-04-27T03:14:43.732 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:43.341+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:43.342973+0000) 2024-04-27T03:14:43.732 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:43.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:43.732 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:43.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:43.732 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:43.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:43.732 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:43.359+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:44.032 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:43.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:44.032 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:43.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:44.033 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:43.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:44.033 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:43.762+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:44.350 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:14:44.028+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56770 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:14:44.350 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:44.024+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:44.025923+0000) 2024-04-27T03:14:44.350 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:44.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:44.350 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:44.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:44.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:44.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:44.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:44.024+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:44.025906+0000) 2024-04-27T03:14:44.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:44.025+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:44.026856+0000) 2024-04-27T03:14:44.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:44.033+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:44.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:44.033+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:44.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:44.033+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:44.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:44.034+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:44.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:44.359+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:44.360007+0000) 2024-04-27T03:14:44.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:44.494+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:44.495026+0000) 2024-04-27T03:14:44.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:44.339+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:44.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:44.342+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:44.343091+0000) 2024-04-27T03:14:45.042 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:44.775+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:45.043 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:44.775+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:45.043 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:44.775+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:45.043 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:44.775+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:45.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:45.074+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:45.356 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:45.074+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:45.356 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:45.074+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:45.356 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:45.075+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:45.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:45.496+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:45.497148+0000) 2024-04-27T03:14:45.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:45.362+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:45.363131+0000) 2024-04-27T03:14:45.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:45.345+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:45.346181+0000) 2024-04-27T03:14:45.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:45.371+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:45.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:45.371+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:45.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:45.371+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:45.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:45.372+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:46.090 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:45.789+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:46.091 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:45.789+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:46.091 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:45.789+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:46.091 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:45.790+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:46.358 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:46.080+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:46.358 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:46.080+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:46.358 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:46.080+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:46.359 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:46.081+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:46.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:46.364+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:46.365325+0000) 2024-04-27T03:14:46.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:46.498+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:46.499347+0000) 2024-04-27T03:14:46.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:46.348+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:46.349085+0000) 2024-04-27T03:14:46.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:46.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:46.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:46.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:46.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:46.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:46.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:46.398+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:47.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:46.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:47.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:46.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:47.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:46.793+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:47.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:46.793+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:47.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:47.040+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:47.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:47.040+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:47.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:47.040+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:47.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:47.041+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:47.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:47.351+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:47.351904+0000) 2024-04-27T03:14:47.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:47.501+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:47.501875+0000) 2024-04-27T03:14:47.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:47.367+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:47.368370+0000) 2024-04-27T03:14:47.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:47.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:47.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:47.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:47.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:47.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:47.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:47.432+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:48.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:47.810+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:48.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:47.810+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:48.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:47.810+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:48.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:47.811+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:48.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:48.044+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:48.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:48.044+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:48.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:48.044+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:48.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:48.045+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:48.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:48.351+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:48.352222+0000) 2024-04-27T03:14:48.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:48.501+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:48.502072+0000) 2024-04-27T03:14:48.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:48.367+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:48.368575+0000) 2024-04-27T03:14:48.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:48.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:48.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:48.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:48.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:48.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:48.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:48.454+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:49.112 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:14:49.035+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56810 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:14:49.112 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:49.034+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:49.035288+0000) 2024-04-27T03:14:49.113 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:48.838+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:49.113 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:48.838+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:49.113 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:48.838+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:49.113 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:48.839+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:49.113 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:49.034+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:49.034884+0000) 2024-04-27T03:14:49.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:49.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:49.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:49.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:49.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:49.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:49.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:49.023+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:49.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:49.034+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:49.034842+0000) 2024-04-27T03:14:49.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:49.351+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:49.352399+0000) 2024-04-27T03:14:49.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:49.367+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:49.368742+0000) 2024-04-27T03:14:49.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:49.501+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:49.502245+0000) 2024-04-27T03:14:49.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:49.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:49.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:49.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:49.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:49.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:49.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:49.441+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:50.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:49.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:50.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:49.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:50.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:49.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:50.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:49.843+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:50.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:49.982+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:50.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:49.982+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:50.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:49.982+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:50.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:49.983+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:50.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:50.370+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:50.371645+0000) 2024-04-27T03:14:50.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:50.503+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:50.504786+0000) 2024-04-27T03:14:50.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:50.353+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:50.354603+0000) 2024-04-27T03:14:50.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:50.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:50.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:50.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:50.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:50.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:50.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:50.459+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:51.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:50.970+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:51.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:50.970+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:51.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:50.970+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:51.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:50.971+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:51.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:50.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:51.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:50.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:51.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:50.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:51.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:50.862+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:51.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:51.356+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:51.356844+0000) 2024-04-27T03:14:51.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:51.375+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:51.375887+0000) 2024-04-27T03:14:51.863 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:51.506+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:51.506958+0000) 2024-04-27T03:14:51.864 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:51.853+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:51.864 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:51.853+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:51.864 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:51.853+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:51.864 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:51.480+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:51.864 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:51.480+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:51.864 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:51.480+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:51.864 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:51.481+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:52.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:51.853+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:52.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:51.947+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:52.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:51.947+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:52.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:51.947+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:52.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:51.948+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:52.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:52.358+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:52.358831+0000) 2024-04-27T03:14:52.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:52.377+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:52.378397+0000) 2024-04-27T03:14:52.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:52.510+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:52.511446+0000) 2024-04-27T03:14:52.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:52.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:52.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:52.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:52.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:52.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:52.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:52.530+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:53.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:52.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:53.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:52.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:53.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:52.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:53.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:52.895+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:53.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:52.948+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:53.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:52.948+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:53.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:52.948+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:53.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:52.949+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:53.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:53.377+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:53.378633+0000) 2024-04-27T03:14:53.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:53.358+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:53.358988+0000) 2024-04-27T03:14:53.794 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:53.510+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:53.511619+0000) 2024-04-27T03:14:53.795 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:53.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:53.795 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:53.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:53.795 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:53.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:53.795 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:53.491+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:54.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:53.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:54.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:53.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:54.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:53.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:54.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:53.897+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:54.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:53.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:54.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:53.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:54.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:53.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:54.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:53.969+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:54.368 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:14:54.043+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56848 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:14:54.369 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:54.041+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:54.042145+0000) 2024-04-27T03:14:54.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:54.040+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:54.041001+0000) 2024-04-27T03:14:54.369 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:54.040+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:54.040880+0000) 2024-04-27T03:14:54.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:54.378+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:54.378817+0000) 2024-04-27T03:14:54.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:54.510+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:54.511784+0000) 2024-04-27T03:14:54.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:54.358+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:54.359195+0000) 2024-04-27T03:14:54.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:54.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:54.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:54.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:54.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:54.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:54.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:54.466+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:55.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:54.933+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:55.211 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:54.933+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:55.211 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:54.933+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:55.211 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:54.934+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:55.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:54.903+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:55.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:54.903+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:55.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:54.903+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:55.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:54.903+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:55.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:55.382+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:55.383073+0000) 2024-04-27T03:14:55.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:55.362+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:55.363367+0000) 2024-04-27T03:14:55.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:55.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:55.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:55.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:55.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:55.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:55.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:55.434+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:55.867 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:55.513+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:55.514583+0000) 2024-04-27T03:14:55.867 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:55.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:55.867 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:55.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:55.867 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:55.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:56.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:55.857+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:56.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:55.887+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:56.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:55.887+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:56.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:55.887+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:56.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:55.887+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:56.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:56.364+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:56.365683+0000) 2024-04-27T03:14:56.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:56.386+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:56.387366+0000) 2024-04-27T03:14:56.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:56.516+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:56.517459+0000) 2024-04-27T03:14:56.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:56.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:56.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:56.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:56.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:56.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:56.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:56.434+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:57.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:56.921+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:57.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:56.921+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:57.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:56.921+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:57.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:56.922+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:57.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:56.903+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:57.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:56.903+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:57.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:56.903+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:57.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:56.904+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:57.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:57.518+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:57.519567+0000) 2024-04-27T03:14:57.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:57.388+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:57.389341+0000) 2024-04-27T03:14:57.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:57.367+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:57.367824+0000) 2024-04-27T03:14:57.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:57.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:57.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:57.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:57.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:57.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:57.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:57.477+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:58.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:57.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:58.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:57.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:58.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:57.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:58.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:57.953+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:58.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:57.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:58.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:57.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:58.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:57.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:58.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:57.875+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:58.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:58.388+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:58.389576+0000) 2024-04-27T03:14:58.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:58.367+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:58.368072+0000) 2024-04-27T03:14:58.891 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:58.519+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:58.519824+0000) 2024-04-27T03:14:58.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:58.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:58.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:58.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:58.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:58.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:58.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:58.477+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:59.157 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:14:59.049+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56888 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:14:59.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:59.048+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:59.049189+0000) 2024-04-27T03:14:59.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:58.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:59.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:58.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:59.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:58.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:14:59.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:58.915+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:59.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:59.048+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:59.049259+0000) 2024-04-27T03:14:59.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:58.881+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:59.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:58.881+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:59.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:58.881+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:14:59.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:58.882+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:14:59.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:59.047+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:59.048085+0000) 2024-04-27T03:14:59.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:59.367+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:59.368309+0000) 2024-04-27T03:14:59.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:59.389+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:59.389812+0000) 2024-04-27T03:14:59.872 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:59.519+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:14:59.519983+0000) 2024-04-27T03:14:59.873 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:59.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:59.873 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:59.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:59.873 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:59.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:14:59.873 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:14:59.475+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:00.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:59.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:00.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:59.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:00.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:59.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:00.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:14:59.863+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:00.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:59.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:00.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:59.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:00.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:59.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:00.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:14:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:14:59.945+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:00.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:00.372+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:00.372933+0000) 2024-04-27T03:15:00.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:00.393+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:00.394349+0000) 2024-04-27T03:15:00.858 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:00.521+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:00.522641+0000) 2024-04-27T03:15:00.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:00.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:00.859 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:00.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:00.859 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:00.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:00.859 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:00.502+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:01.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:00.847+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:01.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:00.847+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:01.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:00.847+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:01.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:00.848+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:01.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:00.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:01.115 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:00.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:01.115 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:00.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:01.115 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:00.931+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:01.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:01.395+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:01.396524+0000) 2024-04-27T03:15:01.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:01.524+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:01.524846+0000) 2024-04-27T03:15:01.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:01.374+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:01.375030+0000) 2024-04-27T03:15:01.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:01.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:01.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:01.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:01.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:01.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:01.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:01.506+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:02.113 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:01.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:02.113 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:01.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:02.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:01.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:02.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:01.912+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:02.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:01.843+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:02.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:01.843+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:02.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:01.843+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:02.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:01.844+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:02.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:02.526+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:02.526953+0000) 2024-04-27T03:15:02.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:02.397+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:02.398500+0000) 2024-04-27T03:15:02.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:02.376+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:02.376955+0000) 2024-04-27T03:15:02.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:02.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:02.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:02.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:02.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:02.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:02.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:02.511+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:03.113 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:02.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:03.113 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:02.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:03.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:02.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:03.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:02.884+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:03.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:02.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:03.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:02.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:03.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:02.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:03.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:02.813+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:03.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:03.526+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:03.527118+0000) 2024-04-27T03:15:03.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:03.397+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:03.398676+0000) 2024-04-27T03:15:03.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:03.376+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:03.377150+0000) 2024-04-27T03:15:03.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:03.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:03.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:03.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:03.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:03.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:03.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:03.501+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:04.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:03.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:04.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:03.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:04.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:03.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:04.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:03.831+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:04.045 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:03.855+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:04.045 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:03.855+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:04.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:03.855+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:04.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:03.856+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:04.386 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:15:04.056+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56926 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:15:04.387 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:04.055+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:04.056152+0000) 2024-04-27T03:15:04.387 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:04.055+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:04.056048+0000) 2024-04-27T03:15:04.387 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:04.054+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:04.055028+0000) 2024-04-27T03:15:04.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:04.398+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:04.398848+0000) 2024-04-27T03:15:04.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:04.526+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:04.527276+0000) 2024-04-27T03:15:04.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:04.376+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:04.377344+0000) 2024-04-27T03:15:04.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:04.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:04.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:04.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:04.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:04.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:04.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:04.526+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:05.113 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:04.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:05.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:04.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:05.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:04.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:05.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:04.902+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:05.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:04.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:05.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:04.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:05.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:04.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:05.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:04.794+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:05.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:05.400+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:05.401045+0000) 2024-04-27T03:15:05.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:05.528+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:05.529378+0000) 2024-04-27T03:15:05.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:05.378+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:05.379479+0000) 2024-04-27T03:15:05.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:05.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:05.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:05.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:05.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:05.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:05.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:05.523+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:06.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:05.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:06.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:05.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:06.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:05.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:06.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:05.880+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:06.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:05.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:06.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:05.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:06.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:05.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:06.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:05.816+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:06.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:06.404+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:06.405628+0000) 2024-04-27T03:15:06.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:06.530+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:06.531601+0000) 2024-04-27T03:15:06.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:06.382+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:06.383613+0000) 2024-04-27T03:15:06.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:06.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:06.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:06.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:06.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:06.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:06.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:06.505+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:07.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:06.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:07.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:06.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:07.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:06.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:07.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:06.852+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:07.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:06.849+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:07.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:06.849+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:07.114 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:06.849+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:07.115 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:06.850+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:07.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:07.407+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:07.408298+0000) 2024-04-27T03:15:07.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:07.540+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:07.541535+0000) 2024-04-27T03:15:07.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:07.386+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:07.387622+0000) 2024-04-27T03:15:07.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:07.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:07.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:07.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:07.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:07.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:07.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:07.505+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:08.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:07.880+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:08.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:07.880+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:08.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:07.880+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:08.046 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:07.881+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:08.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:07.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:08.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:07.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:08.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:07.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:08.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:07.857+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:08.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:08.407+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:08.408466+0000) 2024-04-27T03:15:08.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:08.540+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:08.541727+0000) 2024-04-27T03:15:08.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:08.387+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:08.387830+0000) 2024-04-27T03:15:08.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:08.484+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:08.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:08.484+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:08.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:08.484+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:08.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:08.485+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:09.196 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:15:09.064+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 56966 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:15:09.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:09.061+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:09.062336+0000) 2024-04-27T03:15:09.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:08.904+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:09.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:08.904+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:09.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:08.904+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:09.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:08.905+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:09.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:09.061+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:09.062652+0000) 2024-04-27T03:15:09.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:08.904+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:09.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:08.904+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:09.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:08.904+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:09.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:08.904+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:09.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:09.060+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:09.061450+0000) 2024-04-27T03:15:09.462 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:09.407+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:09.408628+0000) 2024-04-27T03:15:09.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:09.387+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:09.388031+0000) 2024-04-27T03:15:09.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:09.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:09.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:09.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:09.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:09.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:09.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:09.541+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:09.541879+0000) 2024-04-27T03:15:09.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:09.453+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:10.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:09.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:10.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:09.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:10.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:09.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:10.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:09.883+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:10.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:09.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:10.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:09.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:10.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:09.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:10.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:09.927+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:10.421 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:10.391+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:10.392775+0000) 2024-04-27T03:15:10.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:10.543+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:10.543932+0000) 2024-04-27T03:15:10.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:10.410+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:10.411723+0000) 2024-04-27T03:15:10.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:10.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:10.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:10.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:10.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:10.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:10.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:10.491+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:11.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:10.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:11.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:10.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:11.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:10.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:11.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:10.902+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:11.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:10.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:11.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:10.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:11.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:10.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:11.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:10.976+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:11.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:11.545+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:11.545964+0000) 2024-04-27T03:15:11.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:11.413+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:11.413999+0000) 2024-04-27T03:15:11.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:11.396+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:11.397500+0000) 2024-04-27T03:15:11.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:11.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:11.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:11.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:11.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:11.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:11.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:11.461+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:12.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:12.019+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:12.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:12.019+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:12.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:12.019+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:12.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:12.020+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:12.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:11.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:12.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:11.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:12.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:11.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:12.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:11.874+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:12.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:12.417+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:12.418576+0000) 2024-04-27T03:15:12.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:12.547+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:12.548114+0000) 2024-04-27T03:15:12.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:12.399+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:12.400240+0000) 2024-04-27T03:15:12.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:12.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:12.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:12.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:12.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:12.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:12.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:12.450+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:13.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:12.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:13.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:12.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:13.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:12.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:13.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:12.886+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:13.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:12.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:13.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:12.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:13.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:12.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:13.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:12.980+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:13.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:13.418+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:13.418818+0000) 2024-04-27T03:15:13.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:13.547+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:13.548300+0000) 2024-04-27T03:15:13.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:13.399+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:13.400435+0000) 2024-04-27T03:15:13.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:13.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:13.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:13.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:13.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:13.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:13.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:13.479+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:14.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:13.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:14.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:13.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:14.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:13.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:14.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:13.915+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:14.046 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:13.942+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:14.046 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:13.942+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:14.047 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:13.942+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:14.047 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:13.944+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:14.410 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:15:14.071+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57005 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:15:14.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:14.070+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:14.071010+0000) 2024-04-27T03:15:14.410 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:14.069+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:14.070090+0000) 2024-04-27T03:15:14.411 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:14.069+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:14.070387+0000) 2024-04-27T03:15:14.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:14.547+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:14.548472+0000) 2024-04-27T03:15:14.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:14.418+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:14.419002+0000) 2024-04-27T03:15:14.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:14.399+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:14.400614+0000) 2024-04-27T03:15:14.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:14.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:14.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:14.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:14.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:14.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:14.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:14.497+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:15.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:14.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:15.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:14.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:15.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:14.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:15.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:14.930+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:15.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:14.966+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:15.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:14.966+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:15.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:14.966+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:15.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:14.967+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:15.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:15.423+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:15.424024+0000) 2024-04-27T03:15:15.731 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:15.551+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:15.552653+0000) 2024-04-27T03:15:15.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:15.402+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:15.403014+0000) 2024-04-27T03:15:15.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:15.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:15.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:15.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:15.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:15.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:15.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:15.499+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:16.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:15.983+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:16.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:15.983+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:16.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:15.983+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:16.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:15.984+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:16.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:15.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:16.232 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:15.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:16.232 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:15.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:16.232 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:15.958+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:16.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:16.426+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:16.427113+0000) 2024-04-27T03:15:16.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:16.554+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:16.555380+0000) 2024-04-27T03:15:16.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:16.404+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:16.405741+0000) 2024-04-27T03:15:16.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:16.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:16.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:16.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:16.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:16.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:16.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:16.526+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:17.018 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:16.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:17.018 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:16.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:17.018 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:16.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:17.018 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:16.918+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:17.419 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:17.008+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:17.419 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:17.008+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:17.419 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:17.008+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:17.419 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:17.008+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:17.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:17.428+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:17.429147+0000) 2024-04-27T03:15:17.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:17.556+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:17.557617+0000) 2024-04-27T03:15:17.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:17.408+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:17.409748+0000) 2024-04-27T03:15:17.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:17.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:17.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:17.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:17.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:17.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:17.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:17.573+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:18.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:18.008+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:18.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:18.008+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:18.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:18.008+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:18.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:18.009+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:18.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:17.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:18.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:17.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:18.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:17.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:18.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:17.931+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:18.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:18.428+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:18.429314+0000) 2024-04-27T03:15:18.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:18.557+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:18.557800+0000) 2024-04-27T03:15:18.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:18.409+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:18.409996+0000) 2024-04-27T03:15:18.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:18.599+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:18.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:18.599+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:18.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:18.599+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:18.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:18.601+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:19.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:15:19.078+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57045 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:15:19.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:19.075+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:19.076605+0000) 2024-04-27T03:15:19.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:18.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:19.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:18.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:19.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:18.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:19.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:18.930+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:19.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:19.075+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:19.076471+0000) 2024-04-27T03:15:19.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:19.014+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:19.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:19.014+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:19.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:19.014+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:19.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:19.015+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:19.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:19.076+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:19.077366+0000) 2024-04-27T03:15:19.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:19.409+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:19.410251+0000) 2024-04-27T03:15:19.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:19.428+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:19.429486+0000) 2024-04-27T03:15:19.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:19.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:19.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:19.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:19.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:19.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:19.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:19.583+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:19.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:19.557+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:19.558001+0000) 2024-04-27T03:15:20.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:19.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:20.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:19.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:20.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:19.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:20.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:19.948+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:20.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:19.986+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:20.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:19.986+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:20.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:19.986+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:20.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:19.987+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:20.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:20.413+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:20.414497+0000) 2024-04-27T03:15:20.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:20.430+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:20.431602+0000) 2024-04-27T03:15:20.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:20.559+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:20.560623+0000) 2024-04-27T03:15:20.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:20.945+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:20.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:20.945+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:20.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:20.945+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:20.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:20.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:20.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:20.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:20.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:20.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:20.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:20.579+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:21.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:20.945+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:21.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:21.037+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:21.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:21.037+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:21.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:21.037+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:21.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:21.038+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:21.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:21.562+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:21.562909+0000) 2024-04-27T03:15:21.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:21.415+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:21.416635+0000) 2024-04-27T03:15:21.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:21.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:21.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:21.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:21.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:21.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:21.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:21.572+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:21.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:21.433+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:21.433924+0000) 2024-04-27T03:15:22.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:21.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:22.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:21.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:22.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:21.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:22.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:21.916+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:22.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:22.068+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:22.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:22.068+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:22.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:22.068+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:22.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:22.069+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:22.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:22.435+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:22.436550+0000) 2024-04-27T03:15:22.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:22.565+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:22.565976+0000) 2024-04-27T03:15:22.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:22.418+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:22.419637+0000) 2024-04-27T03:15:22.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:22.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:22.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:22.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:22.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:22.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:22.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:22.565+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:23.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:23.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:23.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:23.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:23.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:23.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:23.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:23.094+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:23.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:22.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:23.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:22.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:23.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:22.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:23.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:22.966+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:23.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:23.435+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:23.436723+0000) 2024-04-27T03:15:23.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:23.565+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:23.566163+0000) 2024-04-27T03:15:23.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:23.419+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:23.419824+0000) 2024-04-27T03:15:23.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:23.589+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:23.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:23.589+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:23.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:23.589+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:23.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:23.589+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:24.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:24.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:24.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:24.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:24.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:24.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:24.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:24.012+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:24.429 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:15:24.085+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57084 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:15:24.429 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:24.082+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:24.083547+0000) 2024-04-27T03:15:24.429 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:24.083+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:24.084592+0000) 2024-04-27T03:15:24.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:24.073+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:24.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:24.073+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:24.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:24.073+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:24.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:24.074+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:24.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:24.084+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:24.085036+0000) 2024-04-27T03:15:24.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:24.436+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:24.436897+0000) 2024-04-27T03:15:24.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:24.565+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:24.566315+0000) 2024-04-27T03:15:24.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:24.419+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:24.420005+0000) 2024-04-27T03:15:24.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:24.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:24.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:24.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:24.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:24.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:24.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:24.584+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:25.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:25.069+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:25.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:25.069+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:25.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:25.069+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:25.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:25.070+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:25.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:24.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:25.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:24.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:25.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:24.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:25.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:24.979+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:25.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:25.567+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:25.568389+0000) 2024-04-27T03:15:25.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:25.439+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:25.440032+0000) 2024-04-27T03:15:25.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:25.422+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:25.422975+0000) 2024-04-27T03:15:25.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:25.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:25.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:25.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:25.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:25.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:25.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:25.579+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:26.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:25.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:26.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:25.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:26.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:25.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:26.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:25.982+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:26.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:26.067+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:26.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:26.067+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:26.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:26.067+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:26.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:26.068+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:26.563 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:26.424+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:26.425050+0000) 2024-04-27T03:15:26.564 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:26.441+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:26.442186+0000) 2024-04-27T03:15:26.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:26.571+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:26.572677+0000) 2024-04-27T03:15:26.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:26.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:26.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:26.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:26.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:26.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:26.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:26.554+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:27.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:26.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:27.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:26.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:27.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:26.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:27.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:26.979+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:27.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:27.088+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:27.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:27.088+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:27.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:27.088+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:27.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:27.089+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:27.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:27.574+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:27.575316+0000) 2024-04-27T03:15:27.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:27.445+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:27.446337+0000) 2024-04-27T03:15:27.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:27.428+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:27.429143+0000) 2024-04-27T03:15:27.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:27.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:27.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:27.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:27.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:27.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:27.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:27.512+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:28.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:28.094+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:28.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:28.094+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:28.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:28.094+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:28.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:28.095+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:28.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:28.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:28.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:28.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:28.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:28.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:28.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:28.018+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:28.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:28.445+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:28.446583+0000) 2024-04-27T03:15:28.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:28.574+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:28.575481+0000) 2024-04-27T03:15:28.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:28.428+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:28.429395+0000) 2024-04-27T03:15:28.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:28.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:28.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:28.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:28.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:28.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:28.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:28.505+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:29.220 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:15:29.093+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57124 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:15:29.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:29.091+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:29.091863+0000) 2024-04-27T03:15:29.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:28.972+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:29.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:28.972+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:29.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:28.972+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:29.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:28.972+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:29.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:29.091+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:29.091939+0000) 2024-04-27T03:15:29.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:29.046+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:29.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:29.046+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:29.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:29.046+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:29.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:29.047+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:29.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:29.090+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:29.091428+0000) 2024-04-27T03:15:29.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:29.446+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:29.446795+0000) 2024-04-27T03:15:29.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:29.428+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:29.429567+0000) 2024-04-27T03:15:29.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:29.574+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:29.575656+0000) 2024-04-27T03:15:29.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:29.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:29.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:29.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:29.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:29.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:29.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:29.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:29.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:29.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:29.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:29.475+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:30.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:29.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:30.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:29.941+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:30.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:30.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:30.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:30.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:30.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:30.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:30.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:30.060+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:30.544 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:30.448+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:30.448863+0000) 2024-04-27T03:15:30.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:30.430+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:30.431729+0000) 2024-04-27T03:15:30.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:30.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:30.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:30.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:30.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:30.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:30.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:30.436+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:30.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:30.579+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:30.579806+0000) 2024-04-27T03:15:31.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:30.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:31.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:30.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:31.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:30.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:31.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:30.909+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:31.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:31.087+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:31.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:31.087+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:31.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:31.087+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:31.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:31.088+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:31.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:31.582+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:31.583065+0000) 2024-04-27T03:15:31.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:31.450+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:31.450955+0000) 2024-04-27T03:15:31.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:31.434+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:31.434936+0000) 2024-04-27T03:15:31.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:31.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:31.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:31.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:31.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:31.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:31.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:31.461+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:32.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:31.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:32.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:31.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:32.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:31.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:32.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:31.936+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:32.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:32.076+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:32.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:32.076+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:32.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:32.076+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:32.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:32.078+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:32.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:32.584+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:32.585296+0000) 2024-04-27T03:15:32.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:32.454+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:32.455143+0000) 2024-04-27T03:15:32.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:32.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:32.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:32.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:32.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:32.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:32.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:32.435+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:32.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:32.438+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:32.438994+0000) 2024-04-27T03:15:33.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:33.048+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:33.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:33.048+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:33.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:33.048+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:33.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:33.049+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:33.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:32.922+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:33.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:32.922+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:33.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:32.922+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:33.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:32.922+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:33.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:33.584+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:33.585494+0000) 2024-04-27T03:15:33.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:33.454+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:33.455335+0000) 2024-04-27T03:15:33.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:33.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:33.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:33.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:33.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:33.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:33.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:33.434+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:33.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:33.438+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:33.439184+0000) 2024-04-27T03:15:34.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:33.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:34.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:33.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:34.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:33.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:34.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:33.912+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:34.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:34.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:34.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:34.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:34.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:34.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:34.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:34.002+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:34.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:34.098+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:34.098968+0000) 2024-04-27T03:15:34.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:34.098+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:34.099455+0000) 2024-04-27T03:15:34.449 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:15:34.100+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57163 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:15:34.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:34.097+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:34.097863+0000) 2024-04-27T03:15:34.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:34.584+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:34.585684+0000) 2024-04-27T03:15:34.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:34.454+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:34.455526+0000) 2024-04-27T03:15:34.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:34.438+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:34.439401+0000) 2024-04-27T03:15:34.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:34.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:34.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:34.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:34.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:34.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:34.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:34.467+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:35.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:34.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:35.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:34.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:35.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:34.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:35.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:34.942+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:35.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:34.984+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:35.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:34.984+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:35.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:34.984+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:35.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:34.984+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:35.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:35.456+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:35.457801+0000) 2024-04-27T03:15:35.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:35.585+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:35.587751+0000) 2024-04-27T03:15:35.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:35.439+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:35.441720+0000) 2024-04-27T03:15:35.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:35.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:35.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:35.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:35.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:35.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:35.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:35.501+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:36.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:35.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:36.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:35.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:36.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:35.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:36.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:35.912+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:36.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:35.999+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:36.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:35.999+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:36.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:35.999+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:36.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:35.999+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:36.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:36.588+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:36.590591+0000) 2024-04-27T03:15:36.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:36.458+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:36.460046+0000) 2024-04-27T03:15:36.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:36.442+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:36.443823+0000) 2024-04-27T03:15:36.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:36.523+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:36.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:36.523+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:36.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:36.523+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:36.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:36.524+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:37.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:36.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:37.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:36.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:37.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:36.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:37.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:36.912+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:37.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:37.043+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:37.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:37.043+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:37.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:37.043+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:37.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:37.044+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:37.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:37.460+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:37.462670+0000) 2024-04-27T03:15:37.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:37.590+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:37.592790+0000) 2024-04-27T03:15:37.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:37.444+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:37.446430+0000) 2024-04-27T03:15:37.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:37.488+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:37.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:37.488+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:37.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:37.488+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:37.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:37.489+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:38.161 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:37.882+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:38.161 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:37.882+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:38.161 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:37.882+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:38.161 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:37.882+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:38.161 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:38.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:38.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:38.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:38.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:38.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:38.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:38.026+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:38.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:38.591+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:38.592985+0000) 2024-04-27T03:15:38.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:38.461+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:38.462827+0000) 2024-04-27T03:15:38.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:38.444+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:38.446611+0000) 2024-04-27T03:15:38.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:38.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:38.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:38.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:38.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:38.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:38.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:38.482+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:39.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:38.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:39.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:38.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:39.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:38.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:39.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:38.844+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:39.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:39.058+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:39.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:39.058+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:39.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:39.058+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:39.113 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:39.059+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:39.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:39.105+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:39.107432+0000) 2024-04-27T03:15:39.445 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:15:39.108+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57203 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:15:39.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:39.104+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:39.106281+0000) 2024-04-27T03:15:39.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:39.105+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:39.106971+0000) 2024-04-27T03:15:39.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:39.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:39.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:39.591+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:39.593233+0000) 2024-04-27T03:15:39.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:39.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:39.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:39.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:39.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:39.435+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:39.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:39.445+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:39.446797+0000) 2024-04-27T03:15:39.731 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:39.461+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:39.463035+0000) 2024-04-27T03:15:40.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:39.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:40.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:39.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:40.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:39.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:40.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:39.853+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:40.117 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:40.068+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:40.118 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:40.068+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:40.118 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:40.068+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:40.118 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:40.068+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:40.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:40.464+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:40.466146+0000) 2024-04-27T03:15:40.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:40.595+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:40.597442+0000) 2024-04-27T03:15:40.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:40.448+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:40.450003+0000) 2024-04-27T03:15:40.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:40.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:40.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:40.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:40.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:40.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:40.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:40.483+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:41.161 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:40.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:41.161 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:40.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:41.161 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:40.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:41.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:40.894+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:41.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:41.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:41.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:41.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:41.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:41.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:41.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:41.097+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:41.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:41.467+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:41.469014+0000) 2024-04-27T03:15:41.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:41.598+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:41.600423+0000) 2024-04-27T03:15:41.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:41.450+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:41.452778+0000) 2024-04-27T03:15:41.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:41.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:41.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:41.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:41.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:41.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:41.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:41.458+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:42.161 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:41.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:42.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:41.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:42.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:41.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:42.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:41.868+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:42.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:42.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:42.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:42.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:42.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:42.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:42.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:42.082+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:42.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:42.601+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:42.603183+0000) 2024-04-27T03:15:42.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:42.469+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:42.471253+0000) 2024-04-27T03:15:42.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:42.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:42.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:42.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:42.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:42.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:42.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:42.440+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:42.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:42.453+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:42.455074+0000) 2024-04-27T03:15:43.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:42.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:43.106 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:42.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:43.106 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:42.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:43.106 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:42.843+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:43.464 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:43.094+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:43.464 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:43.094+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:43.464 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:43.094+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:43.464 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:43.095+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:43.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:43.469+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:43.471493+0000) 2024-04-27T03:15:43.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:43.453+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:43.455332+0000) 2024-04-27T03:15:43.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:43.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:43.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:43.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:43.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:43.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:43.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:43.470+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:43.732 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:43.601+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:43.603375+0000) 2024-04-27T03:15:44.049 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:43.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:44.049 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:43.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:44.049 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:43.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:44.049 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:43.820+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:44.394 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:15:44.115+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57242 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:15:44.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:44.113+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:44.115280+0000) 2024-04-27T03:15:44.394 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:44.114+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:44.116307+0000) 2024-04-27T03:15:44.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:44.112+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:44.114687+0000) 2024-04-27T03:15:44.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:44.144+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:44.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:44.144+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:44.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:44.144+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:44.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:44.146+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:44.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:44.470+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:44.471984+0000) 2024-04-27T03:15:44.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:44.601+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:44.603630+0000) 2024-04-27T03:15:44.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:44.453+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:44.455447+0000) 2024-04-27T03:15:44.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:44.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:44.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:44.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:44.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:44.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:44.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:44.491+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:45.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:44.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:45.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:44.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:45.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:44.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:45.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:44.823+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:45.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:45.154+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:45.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:45.154+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:45.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:45.154+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:45.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:45.155+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:45.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:45.604+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:45.605980+0000) 2024-04-27T03:15:45.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:45.472+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:45.474159+0000) 2024-04-27T03:15:45.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:45.457+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:45.459649+0000) 2024-04-27T03:15:45.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:45.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:45.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:45.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:45.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:45.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:45.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:45.493+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:46.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:45.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:46.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:45.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:46.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:45.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:46.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:45.841+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:46.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:46.157+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:46.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:46.157+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:46.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:46.157+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:46.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:46.158+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:46.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:46.476+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:46.478263+0000) 2024-04-27T03:15:46.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:46.608+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:46.610222+0000) 2024-04-27T03:15:46.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:46.460+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:46.462090+0000) 2024-04-27T03:15:46.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:46.473+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:46.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:46.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:46.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:46.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:46.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:46.474+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:47.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:46.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:47.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:46.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:47.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:46.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:47.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:46.885+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:47.463 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:47.157+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:47.463 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:47.157+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:47.464 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:47.157+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:47.464 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:47.158+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:47.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:47.479+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:47.481221+0000) 2024-04-27T03:15:47.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:47.611+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:47.613286+0000) 2024-04-27T03:15:47.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:47.462+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:47.464333+0000) 2024-04-27T03:15:47.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:47.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:47.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:47.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:47.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:47.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:47.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:47.482+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:48.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:47.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:48.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:47.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:48.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:47.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:48.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:47.867+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:48.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:48.140+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:48.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:48.140+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:48.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:48.140+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:48.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:48.141+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:48.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:48.612+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:48.613459+0000) 2024-04-27T03:15:48.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:48.480+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:48.481395+0000) 2024-04-27T03:15:48.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:48.463+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:48.464653+0000) 2024-04-27T03:15:48.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:48.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:48.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:48.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:48.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:48.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:48.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:48.486+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:49.118 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:48.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:49.118 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:48.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:49.118 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:48.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:49.118 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:48.851+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:49.473 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:15:49.123+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57282 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:15:49.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:49.122+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:49.123485+0000) 2024-04-27T03:15:49.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:49.122+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:49.122886+0000) 2024-04-27T03:15:49.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:49.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:49.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:49.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:49.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:49.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:49.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:49.120+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:49.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:49.122+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:49.123032+0000) 2024-04-27T03:15:49.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:49.612+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:49.613661+0000) 2024-04-27T03:15:49.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:49.480+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:49.481590+0000) 2024-04-27T03:15:49.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:49.464+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:49.464824+0000) 2024-04-27T03:15:49.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:49.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:49.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:49.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:49.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:49.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:49.731 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:49.490+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:50.118 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:49.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:50.118 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:49.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:50.118 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:49.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:50.118 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:49.845+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:50.118 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:50.078+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:50.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:50.078+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:50.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:50.078+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:50.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:50.079+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:50.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:50.482+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:50.483777+0000) 2024-04-27T03:15:50.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:50.614+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:50.615791+0000) 2024-04-27T03:15:50.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:50.469+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:50.469909+0000) 2024-04-27T03:15:50.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:50.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:50.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:50.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:50.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:50.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:50.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:50.494+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:51.049 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:50.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:51.049 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:50.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:51.049 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:50.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:51.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:50.835+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:51.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:51.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:51.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:51.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:51.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:51.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:51.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:51.066+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:51.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:51.485+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:51.486592+0000) 2024-04-27T03:15:51.881 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:51.474+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:51.474981+0000) 2024-04-27T03:15:51.881 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:51.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:51.881 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:51.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:51.881 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:51.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:51.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:51.506+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:51.882 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:51.619+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:51.620019+0000) 2024-04-27T03:15:51.882 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:51.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:51.882 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:51.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:51.882 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:51.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:52.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:51.872+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:52.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:52.040+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:52.163 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:52.040+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:52.163 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:52.040+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:52.163 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:52.041+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:52.842 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:52.623+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:52.624203+0000) 2024-04-27T03:15:52.842 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:52.479+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:52.479995+0000) 2024-04-27T03:15:52.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:52.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:52.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:52.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:52.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:52.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:52.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:52.548+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:52.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:52.487+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:52.488682+0000) 2024-04-27T03:15:53.100 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:52.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:53.100 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:52.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:53.100 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:52.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:53.100 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:52.833+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:53.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:53.090+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:53.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:53.090+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:53.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:53.090+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:53.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:53.091+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:53.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:53.488+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:53.488843+0000) 2024-04-27T03:15:53.797 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:53.623+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:53.624437+0000) 2024-04-27T03:15:53.797 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:53.479+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:53.480188+0000) 2024-04-27T03:15:53.797 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:53.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:53.797 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:53.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:53.797 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:53.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:53.797 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:53.558+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:54.049 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:53.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:54.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:53.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:54.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:53.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:54.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:53.871+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:54.479 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:15:54.131+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57321 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:15:54.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:54.129+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:54.130457+0000) 2024-04-27T03:15:54.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:54.128+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:54.129148+0000) 2024-04-27T03:15:54.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:54.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:54.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:54.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:54.481 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:54.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:54.481 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:54.126+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:54.481 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:54.129+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:54.130267+0000) 2024-04-27T03:15:54.871 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:54.623+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:54.624598+0000) 2024-04-27T03:15:54.872 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:54.488+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:54.489080+0000) 2024-04-27T03:15:54.872 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:54.479+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:54.480388+0000) 2024-04-27T03:15:54.872 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:54.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:54.872 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:54.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:54.872 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:54.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:54.872 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:54.554+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:55.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:55.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:55.163 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:55.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:55.163 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:55.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:55.163 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:55.094+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:55.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:54.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:55.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:54.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:55.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:54.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:55.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:54.862+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:55.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:55.490+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:55.491257+0000) 2024-04-27T03:15:55.817 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:55.628+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:55.629608+0000) 2024-04-27T03:15:55.817 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:55.483+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:55.484637+0000) 2024-04-27T03:15:55.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:55.551+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:55.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:55.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:55.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:55.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:55.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:55.553+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:56.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:55.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:56.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:55.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:56.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:55.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:56.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:55.826+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:56.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:56.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:56.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:56.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:56.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:56.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:56.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:56.067+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:56.643 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:56.486+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:56.486827+0000) 2024-04-27T03:15:56.643 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:56.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:56.643 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:56.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:56.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:56.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:56.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:56.572+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:56.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:56.493+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:56.493901+0000) 2024-04-27T03:15:56.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:56.633+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:56.634603+0000) 2024-04-27T03:15:56.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:56.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:56.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:56.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:56.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:56.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:56.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:56.833+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:57.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:57.103+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:57.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:57.103+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:57.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:57.103+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:57.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:57.104+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:57.830 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:57.497+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:57.498245+0000) 2024-04-27T03:15:57.831 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:57.636+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:57.636976+0000) 2024-04-27T03:15:57.831 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:57.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:57.831 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:57.488+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:57.489528+0000) 2024-04-27T03:15:57.831 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:57.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:57.831 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:57.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:57.831 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:57.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:57.831 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:57.572+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:58.093 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:57.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:58.094 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:57.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:58.094 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:57.821+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:58.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:58.083+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:58.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:58.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:58.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:58.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:58.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:58.084+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:58.824 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:58.497+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:58.498483+0000) 2024-04-27T03:15:58.825 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:58.488+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:58.489724+0000) 2024-04-27T03:15:58.825 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:58.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:58.825 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:58.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:58.825 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:58.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:58.825 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:58.573+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:58.825 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:58.636+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:58.637131+0000) 2024-04-27T03:15:58.825 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:58.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:58.825 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:58.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:58.825 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:58.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:15:59.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:58.815+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:59.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:59.064+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:59.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:59.064+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:59.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:59.064+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:15:59.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:59.065+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:59.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:59.136+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:59.137434+0000) 2024-04-27T03:15:59.480 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:15:59.138+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57361 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:15:59.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:59.135+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:59.136083+0000) 2024-04-27T03:15:59.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:59.135+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:59.136488+0000) 2024-04-27T03:15:59.751 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:59.636+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:59.637300+0000) 2024-04-27T03:15:59.752 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:59.489+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:59.489883+0000) 2024-04-27T03:15:59.752 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:59.543+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:59.752 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:59.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:59.752 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:59.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:15:59.752 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:15:59.544+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:15:59.752 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:15:59.497+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:15:59.498719+0000) 2024-04-27T03:16:00.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:59.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:00.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:59.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:00.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:59.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:00.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:15:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:15:59.827+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:00.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:00.095+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:00.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:00.095+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:00.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:00.095+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:00.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:00.096+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:00.864 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:00.639+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:00.640642+0000) 2024-04-27T03:16:00.865 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:00.500+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:00.500945+0000) 2024-04-27T03:16:00.865 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:00.492+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:00.493284+0000) 2024-04-27T03:16:00.865 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:00.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:00.865 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:00.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:00.865 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:00.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:00.865 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:00.507+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:01.119 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:00.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:01.119 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:00.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:01.119 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:00.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:01.119 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:00.876+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:01.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:01.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:01.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:01.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:01.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:01.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:01.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:01.141+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:01.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:01.643+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:01.643978+0000) 2024-04-27T03:16:01.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:01.502+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:01.503188+0000) 2024-04-27T03:16:01.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:01.494+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:01.495484+0000) 2024-04-27T03:16:01.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:01.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:01.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:01.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:01.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:01.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:01.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:01.502+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:02.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:01.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:02.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:01.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:02.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:01.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:02.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:01.923+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:02.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:02.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:02.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:02.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:02.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:02.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:02.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:02.176+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:02.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:02.505+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:02.506220+0000) 2024-04-27T03:16:02.823 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:02.648+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:02.649060+0000) 2024-04-27T03:16:02.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:02.499+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:02.500247+0000) 2024-04-27T03:16:02.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:02.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:02.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:02.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:02.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:02.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:02.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:02.504+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:03.119 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:02.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:03.119 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:02.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:03.119 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:02.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:03.119 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:02.916+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:03.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:03.142+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:03.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:03.142+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:03.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:03.142+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:03.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:03.143+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:03.797 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:03.648+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:03.649260+0000) 2024-04-27T03:16:03.798 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:03.505+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:03.506458+0000) 2024-04-27T03:16:03.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:03.499+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:03.500427+0000) 2024-04-27T03:16:03.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:03.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:03.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:03.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:03.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:03.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:03.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:03.517+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:04.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:03.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:04.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:03.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:04.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:03.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:04.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:03.961+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:04.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:04.142+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:04.143558+0000) 2024-04-27T03:16:04.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:04.143+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:04.144502+0000) 2024-04-27T03:16:04.480 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:16:04.146+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57400 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:16:04.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:04.143+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:04.143946+0000) 2024-04-27T03:16:04.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:04.168+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:04.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:04.168+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:04.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:04.168+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:04.481 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:04.170+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:04.942 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:04.648+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:04.649453+0000) 2024-04-27T03:16:04.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:04.499+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:04.500680+0000) 2024-04-27T03:16:04.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:04.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:04.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:04.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:04.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:04.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:04.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:04.506+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:04.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:04.505+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:04.506713+0000) 2024-04-27T03:16:05.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:05.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:05.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:05.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:05.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:05.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:05.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:05.161+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:05.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:04.933+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:05.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:04.933+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:05.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:04.933+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:05.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:04.933+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:05.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:05.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:05.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:05.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:05.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:05.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:05.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:05.483+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:05.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:05.502+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:05.502858+0000) 2024-04-27T03:16:05.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:05.508+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:05.508824+0000) 2024-04-27T03:16:05.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:05.651+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:05.652567+0000) 2024-04-27T03:16:06.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:05.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:06.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:05.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:06.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:05.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:06.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:05.976+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:06.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:06.167+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:06.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:06.167+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:06.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:06.167+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:06.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:06.168+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:06.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:06.512+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:06.513192+0000) 2024-04-27T03:16:06.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:06.656+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:06.657308+0000) 2024-04-27T03:16:06.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:06.504+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:06.504926+0000) 2024-04-27T03:16:06.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:06.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:06.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:06.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:06.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:06.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:06.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:06.518+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:07.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:07.172+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:07.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:07.172+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:07.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:07.172+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:07.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:07.173+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:07.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:06.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:07.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:06.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:07.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:06.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:07.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:06.981+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:07.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:07.514+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:07.515375+0000) 2024-04-27T03:16:07.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:07.659+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:07.660377+0000) 2024-04-27T03:16:07.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:07.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:07.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:07.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:07.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:07.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:07.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:07.492+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:07.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:07.506+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:07.507077+0000) 2024-04-27T03:16:08.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:08.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:08.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:08.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:08.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:08.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:08.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:08.132+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:08.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:07.945+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:08.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:07.945+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:08.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:07.945+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:08.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:07.946+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:08.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:08.514+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:08.515561+0000) 2024-04-27T03:16:08.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:08.659+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:08.660586+0000) 2024-04-27T03:16:08.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:08.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:08.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:08.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:08.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:08.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:08.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:08.462+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:08.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:08.506+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:08.507282+0000) 2024-04-27T03:16:09.229 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:16:09.152+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57440 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:16:09.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:09.150+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:09.151346+0000) 2024-04-27T03:16:09.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:09.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:09.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:09.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:09.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:09.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:09.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:09.098+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:09.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:09.150+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:09.151590+0000) 2024-04-27T03:16:09.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:08.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:09.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:08.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:09.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:08.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:09.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:08.949+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:09.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:09.150+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:09.151403+0000) 2024-04-27T03:16:09.596 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:09.514+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:09.515745+0000) 2024-04-27T03:16:09.596 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:09.492+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:09.596 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:09.492+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:09.596 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:09.492+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:09.596 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:09.493+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:09.596 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:09.506+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:09.507462+0000) 2024-04-27T03:16:09.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:09.659+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:09.660768+0000) 2024-04-27T03:16:10.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:10.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:10.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:10.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:10.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:10.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:10.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:10.146+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:10.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:09.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:10.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:09.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:10.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:09.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:10.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:09.909+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:10.904 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:10.662+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:10.663030+0000) 2024-04-27T03:16:10.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:10.517+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:10.518328+0000) 2024-04-27T03:16:10.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:10.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:10.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:10.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:10.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:10.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:10.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:10.491+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:10.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:10.508+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:10.509591+0000) 2024-04-27T03:16:11.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:10.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:11.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:10.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:11.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:10.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:11.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:10.895+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:11.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:11.157+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:11.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:11.157+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:11.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:11.157+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:11.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:11.158+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:11.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:11.664+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:11.665255+0000) 2024-04-27T03:16:11.857 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:11.520+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:11.521274+0000) 2024-04-27T03:16:11.857 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:11.510+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:11.511688+0000) 2024-04-27T03:16:11.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:11.522+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:11.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:11.522+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:11.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:11.522+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:11.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:11.524+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:12.120 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:11.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:12.120 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:11.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:12.120 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:11.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:12.120 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:11.848+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:12.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:12.196+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:12.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:12.196+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:12.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:12.196+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:12.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:12.197+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:12.892 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:12.522+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:12.523759+0000) 2024-04-27T03:16:12.892 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:12.666+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:12.667402+0000) 2024-04-27T03:16:12.892 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:12.882+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:12.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:12.515+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:12.516432+0000) 2024-04-27T03:16:12.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:12.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:12.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:12.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:12.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:12.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:12.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:12.570+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:13.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:12.882+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:13.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:12.882+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:13.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:12.883+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:13.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:13.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:13.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:13.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:13.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:13.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:13.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:13.163+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:13.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:13.515+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:13.516620+0000) 2024-04-27T03:16:13.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:13.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:13.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:13.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:13.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:13.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:13.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:13.549+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:13.798 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:13.666+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:13.667695+0000) 2024-04-27T03:16:13.799 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:13.523+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:13.523945+0000) 2024-04-27T03:16:14.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:13.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:14.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:13.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:14.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:13.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:14.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:13.868+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:14.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:14.158+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:14.159318+0000) 2024-04-27T03:16:14.480 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:16:14.159+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57479 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:16:14.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:14.157+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:14.158413+0000) 2024-04-27T03:16:14.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:14.157+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:14.157929+0000) 2024-04-27T03:16:14.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:14.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:14.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:14.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:14.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:14.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:14.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:14.189+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:14.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:14.523+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:14.524101+0000) 2024-04-27T03:16:14.922 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:14.667+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:14.667892+0000) 2024-04-27T03:16:14.922 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:14.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:14.922 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:14.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:14.922 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:14.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:14.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:14.516+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:14.516845+0000) 2024-04-27T03:16:14.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:14.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:14.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:14.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:14.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:14.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:14.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:14.547+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:15.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:14.913+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:15.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:15.197+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:15.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:15.197+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:15.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:15.197+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:15.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:15.198+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:15.879 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:15.669+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:15.670003+0000) 2024-04-27T03:16:15.880 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:15.518+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:15.519142+0000) 2024-04-27T03:16:15.880 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:15.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:15.880 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:15.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:15.880 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:15.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:15.880 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:15.596+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:15.880 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:15.526+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:15.527551+0000) 2024-04-27T03:16:16.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:15.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:16.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:15.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:16.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:15.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:16.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:15.870+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:16.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:16.154+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:16.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:16.154+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:16.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:16.154+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:16.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:16.155+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:16.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:16.529+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:16.530362+0000) 2024-04-27T03:16:16.889 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:16.521+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:16.522180+0000) 2024-04-27T03:16:16.889 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:16.566+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:16.889 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:16.566+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:16.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:16.566+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:16.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:16.566+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:16.890 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:16.674+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:16.675089+0000) 2024-04-27T03:16:16.890 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:16.879+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:16.890 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:16.879+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:16.890 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:16.879+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:17.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:16.880+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:17.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:17.159+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:17.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:17.159+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:17.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:17.159+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:17.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:17.159+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:17.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:17.533+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:17.534672+0000) 2024-04-27T03:16:17.861 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:17.677+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:17.678184+0000) 2024-04-27T03:16:17.861 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:17.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:17.861 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:17.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:17.861 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:17.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:17.861 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:17.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:17.862 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:17.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:17.862 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:17.520+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:17.862 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:17.523+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:17.524532+0000) 2024-04-27T03:16:18.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:17.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:18.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:17.852+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:18.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:18.140+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:18.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:18.140+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:18.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:18.140+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:18.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:18.141+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:18.892 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:18.534+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:18.534822+0000) 2024-04-27T03:16:18.893 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:18.677+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:18.678375+0000) 2024-04-27T03:16:18.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:18.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:18.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:18.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:18.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:18.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:18.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:18.514+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:18.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:18.523+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:18.524724+0000) 2024-04-27T03:16:19.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:18.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:19.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:18.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:19.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:18.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:19.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:18.883+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:19.479 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:16:19.167+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57519 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:16:19.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:19.163+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:19.164762+0000) 2024-04-27T03:16:19.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:19.165+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:19.166189+0000) 2024-04-27T03:16:19.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:19.165+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:19.166494+0000) 2024-04-27T03:16:19.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:19.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:19.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:19.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:19.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:19.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:19.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:19.189+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:19.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:19.534+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:19.535006+0000) 2024-04-27T03:16:19.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:19.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:19.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:19.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:19.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:19.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:19.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:19.479+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:19.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:19.524+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:19.524898+0000) 2024-04-27T03:16:19.844 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:19.677+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:19.678572+0000) 2024-04-27T03:16:19.845 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:19.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:19.845 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:19.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:19.845 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:19.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:20.120 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:19.835+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:20.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:20.182+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:20.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:20.182+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:20.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:20.182+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:20.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:20.183+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:20.881 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:20.679+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:20.680667+0000) 2024-04-27T03:16:20.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:20.538+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:20.539319+0000) 2024-04-27T03:16:20.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:20.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:20.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:20.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:20.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:20.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:20.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:20.483+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:20.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:20.526+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:20.526934+0000) 2024-04-27T03:16:21.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:20.872+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:21.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:20.872+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:21.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:20.872+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:21.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:20.872+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:21.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:21.137+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:21.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:21.137+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:21.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:21.137+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:21.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:21.138+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:21.885 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:21.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:21.885 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:21.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:21.885 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:21.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:21.885 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:21.492+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:21.885 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:21.529+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:21.530039+0000) 2024-04-27T03:16:21.886 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:21.542+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:21.543642+0000) 2024-04-27T03:16:21.886 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:21.681+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:21.682812+0000) 2024-04-27T03:16:21.886 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:21.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:21.886 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:21.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:21.886 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:21.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:22.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:21.876+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:22.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:22.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:22.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:22.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:22.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:22.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:22.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:22.157+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:22.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:22.545+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:22.545840+0000) 2024-04-27T03:16:22.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:22.684+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:22.684957+0000) 2024-04-27T03:16:22.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:22.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:22.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:22.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:22.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:22.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:22.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:22.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:22.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:22.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:22.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:22.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:22.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:22.496+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:22.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:22.533+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:22.534359+0000) 2024-04-27T03:16:23.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:22.911+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:23.456 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:23.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:23.456 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:23.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:23.457 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:23.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:23.457 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:23.165+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:23.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:23.545+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:23.546021+0000) 2024-04-27T03:16:23.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:23.684+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:23.685213+0000) 2024-04-27T03:16:23.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:23.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:23.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:23.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:23.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:23.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:23.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:23.448+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:23.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:23.533+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:23.534583+0000) 2024-04-27T03:16:24.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:23.877+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:24.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:23.877+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:24.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:23.877+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:24.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:23.878+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:24.329 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:16:24.173+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57558 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:16:24.329 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:24.171+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:24.172433+0000) 2024-04-27T03:16:24.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:24.172+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:24.172874+0000) 2024-04-27T03:16:24.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:24.171+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:24.172236+0000) 2024-04-27T03:16:24.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:24.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:24.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:24.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:24.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:24.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:24.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:24.203+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:24.597 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:24.545+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:24.546236+0000) 2024-04-27T03:16:24.598 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:24.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:24.598 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:24.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:24.598 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:24.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:24.598 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:24.416+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:24.598 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:24.534+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:24.534860+0000) 2024-04-27T03:16:24.877 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:24.684+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:24.685460+0000) 2024-04-27T03:16:24.878 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:24.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:24.878 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:24.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:24.878 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:24.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:25.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:24.869+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:25.429 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:25.155+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:25.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:25.155+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:25.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:25.155+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:25.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:25.157+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:25.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:25.548+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:25.549152+0000) 2024-04-27T03:16:25.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:25.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:25.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:25.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:25.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:25.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:25.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:25.421+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:25.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:25.537+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:25.537993+0000) 2024-04-27T03:16:25.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:25.687+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:25.688359+0000) 2024-04-27T03:16:25.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:25.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:25.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:25.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:25.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:25.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:25.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:25.821+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:26.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:26.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:26.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:26.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:26.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:26.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:26.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:26.135+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:26.548 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:26.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:26.548 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:26.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:26.549 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:26.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:26.549 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:26.443+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:26.857 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:26.539+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:26.540183+0000) 2024-04-27T03:16:26.858 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:26.550+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:26.551457+0000) 2024-04-27T03:16:26.858 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:26.690+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:26.690921+0000) 2024-04-27T03:16:27.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:26.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:27.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:26.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:27.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:26.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:27.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:26.849+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:27.455 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:27.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:27.455 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:27.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:27.455 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:27.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:27.455 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:27.157+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:27.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:27.553+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:27.554304+0000) 2024-04-27T03:16:27.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:27.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:27.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:27.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:27.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:27.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:27.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:27.446+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:27.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:27.542+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:27.543015+0000) 2024-04-27T03:16:27.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:27.693+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:27.694114+0000) 2024-04-27T03:16:28.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:27.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:28.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:27.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:28.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:27.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:28.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:27.863+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:28.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:28.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:28.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:28.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:28.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:28.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:28.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:28.129+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:28.828 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:28.553+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:28.554538+0000) 2024-04-27T03:16:28.828 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:28.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:28.828 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:28.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:28.828 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:28.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:28.828 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:28.472+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:28.829 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:28.542+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:28.543427+0000) 2024-04-27T03:16:28.829 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:28.693+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:28.694330+0000) 2024-04-27T03:16:28.829 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:28.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:28.829 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:28.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:28.829 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:28.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:29.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:28.819+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:29.465 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:16:29.181+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57598 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:16:29.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:29.178+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:29.179093+0000) 2024-04-27T03:16:29.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:29.179+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:29.180131+0000) 2024-04-27T03:16:29.466 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:29.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:29.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:29.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:29.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:29.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:29.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:29.132+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:29.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:29.178+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:29.179137+0000) 2024-04-27T03:16:29.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:29.553+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:29.554706+0000) 2024-04-27T03:16:29.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:29.693+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:29.694521+0000) 2024-04-27T03:16:29.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:29.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:29.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:29.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:29.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:29.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:29.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:29.457+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:29.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:29.542+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:29.543681+0000) 2024-04-27T03:16:30.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:29.770+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:30.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:29.770+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:30.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:29.770+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:30.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:29.770+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:30.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:30.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:30.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:30.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:30.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:30.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:30.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:30.172+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:30.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:30.696+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:30.696947+0000) 2024-04-27T03:16:30.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:30.558+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:30.559727+0000) 2024-04-27T03:16:30.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:30.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:30.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:30.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:30.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:30.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:30.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:30.465+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:30.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:30.545+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:30.545826+0000) 2024-04-27T03:16:31.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:30.797+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:31.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:30.797+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:31.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:30.797+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:31.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:30.798+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:31.462 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:31.212+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:31.463 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:31.212+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:31.463 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:31.212+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:31.463 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:31.214+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:31.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:31.563+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:31.564777+0000) 2024-04-27T03:16:31.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:31.699+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:31.700118+0000) 2024-04-27T03:16:31.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:31.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:31.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:31.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:31.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:31.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:31.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:31.454+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:31.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:31.549+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:31.550005+0000) 2024-04-27T03:16:32.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:31.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:32.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:31.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:32.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:31.836+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:32.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:31.836+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:32.421 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:32.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:32.421 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:32.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:32.421 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:32.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:32.421 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:32.210+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:32.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:32.569+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:32.569846+0000) 2024-04-27T03:16:32.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:32.412+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:32.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:32.412+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:32.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:32.412+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:32.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:32.413+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:32.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:32.551+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:32.552158+0000) 2024-04-27T03:16:32.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:32.701+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:32.702497+0000) 2024-04-27T03:16:32.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:32.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:32.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:32.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:32.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:32.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:32.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:32.835+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:33.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:33.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:33.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:33.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:33.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:33.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:33.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:33.376+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:33.561 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:33.225+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:33.561 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:33.225+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:33.561 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:33.225+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:33.561 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:33.226+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:33.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:33.551+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:33.552422+0000) 2024-04-27T03:16:33.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:33.569+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:33.570226+0000) 2024-04-27T03:16:33.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:33.702+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:33.702879+0000) 2024-04-27T03:16:33.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:33.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:33.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:33.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:33.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:33.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:33.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:33.786+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:34.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:34.187+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:34.188267+0000) 2024-04-27T03:16:34.229 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:16:34.189+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57637 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:16:34.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:34.186+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:34.187293+0000) 2024-04-27T03:16:34.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:34.184+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:34.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:34.184+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:34.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:34.184+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:34.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:34.185+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:34.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:34.187+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:34.187970+0000) 2024-04-27T03:16:34.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:34.390+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:34.579 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:34.390+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:34.579 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:34.390+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:34.579 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:34.391+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:34.579 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:34.551+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:34.552689+0000) 2024-04-27T03:16:34.830 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:34.702+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:34.703070+0000) 2024-04-27T03:16:34.830 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:34.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:34.830 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:34.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:34.830 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:34.569+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:34.570412+0000) 2024-04-27T03:16:35.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:34.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:35.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:34.821+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:35.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:35.190+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:35.415 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:35.190+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:35.415 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:35.190+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:35.415 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:35.191+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:35.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:35.572+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:35.573459+0000) 2024-04-27T03:16:35.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:35.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:35.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:35.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:35.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:35.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:35.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:35.406+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:35.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:35.554+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:35.554991+0000) 2024-04-27T03:16:35.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:35.706+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:35.707667+0000) 2024-04-27T03:16:35.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:35.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:35.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:35.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:35.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:35.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:35.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:35.844+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:36.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:36.152+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:36.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:36.152+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:36.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:36.152+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:36.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:36.154+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:36.637 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:36.576+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:36.577781+0000) 2024-04-27T03:16:36.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:36.412+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:36.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:36.412+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:36.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:36.412+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:36.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:36.412+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:36.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:36.556+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:36.557226+0000) 2024-04-27T03:16:36.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:36.711+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:36.711969+0000) 2024-04-27T03:16:36.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:36.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:36.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:36.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:36.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:36.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:36.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:36.842+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:37.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:37.180+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:37.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:37.180+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:37.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:37.180+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:37.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:37.182+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:37.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:37.579+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:37.579927+0000) 2024-04-27T03:16:37.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:37.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:37.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:37.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:37.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:37.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:37.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:37.387+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:37.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:37.559+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:37.560292+0000) 2024-04-27T03:16:37.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:37.713+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:37.714781+0000) 2024-04-27T03:16:37.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:37.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:37.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:37.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:37.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:37.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:37.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:37.819+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:38.387 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:38.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:38.387 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:38.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:38.387 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:38.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:38.387 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:38.216+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:38.638 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:38.578+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:38.580153+0000) 2024-04-27T03:16:38.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:38.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:38.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:38.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:38.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:38.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:38.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:38.378+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:38.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:38.558+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:38.560667+0000) 2024-04-27T03:16:38.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:38.713+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:38.715113+0000) 2024-04-27T03:16:38.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:38.855+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:38.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:38.855+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:38.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:38.855+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:38.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:38.855+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:39.224 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:16:39.195+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57677 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:16:39.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:39.192+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:39.194647+0000) 2024-04-27T03:16:39.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:39.194+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:39.196048+0000) 2024-04-27T03:16:39.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:39.193+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:39.195309+0000) 2024-04-27T03:16:39.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:39.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:39.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:39.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:39.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:39.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:39.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:39.402+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:39.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:39.242+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:39.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:39.242+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:39.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:39.242+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:39.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:39.243+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:39.840 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:39.559+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:39.560904+0000) 2024-04-27T03:16:39.840 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:39.713+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:39.715306+0000) 2024-04-27T03:16:39.840 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:39.578+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:39.580396+0000) 2024-04-27T03:16:40.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:39.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:40.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:39.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:40.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:39.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:40.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:39.830+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:40.386 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:40.236+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:40.386 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:40.236+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:40.386 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:40.236+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:40.386 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:40.236+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:40.640 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:40.581+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:40.583285+0000) 2024-04-27T03:16:40.640 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:40.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:40.641 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:40.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:40.641 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:40.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:40.641 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:40.376+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:40.641 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:40.562+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:40.564055+0000) 2024-04-27T03:16:40.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:40.715+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:40.717469+0000) 2024-04-27T03:16:40.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:40.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:40.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:40.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:40.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:40.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:40.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:40.815+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:41.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:41.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:41.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:41.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:41.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:41.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:41.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:41.414+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:41.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:41.210+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:41.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:41.210+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:41.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:41.210+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:41.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:41.211+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:41.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:41.564+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:41.566204+0000) 2024-04-27T03:16:41.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:41.586+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:41.588354+0000) 2024-04-27T03:16:41.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:41.717+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:41.719754+0000) 2024-04-27T03:16:41.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:41.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:41.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:41.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:41.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:41.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:41.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:41.775+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:42.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:42.185+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:42.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:42.185+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:42.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:42.185+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:42.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:42.186+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:42.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:42.591+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:42.593399+0000) 2024-04-27T03:16:42.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:42.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:42.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:42.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:42.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:42.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:42.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:42.449+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:42.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:42.566+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:42.568373+0000) 2024-04-27T03:16:43.054 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:42.720+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:42.722632+0000) 2024-04-27T03:16:43.054 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:42.797+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:43.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:42.797+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:43.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:42.797+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:43.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:42.797+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:43.440 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:43.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:43.440 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:43.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:43.440 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:43.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:43.440 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:43.201+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:43.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:43.592+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:43.593792+0000) 2024-04-27T03:16:43.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:43.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:43.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:43.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:43.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:43.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:43.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:43.431+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:43.730 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:43.566+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:43.568684+0000) 2024-04-27T03:16:44.054 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:43.721+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:43.722969+0000) 2024-04-27T03:16:44.054 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:43.830+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:44.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:43.830+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:44.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:43.830+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:44.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:43.831+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:44.334 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:16:44.202+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57716 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:16:44.334 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:44.200+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:44.202353+0000) 2024-04-27T03:16:44.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:44.201+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:44.203380+0000) 2024-04-27T03:16:44.335 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:44.193+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:44.335 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:44.193+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:44.335 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:44.193+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:44.335 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:44.193+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:44.335 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:44.201+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:44.203238+0000) 2024-04-27T03:16:44.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:44.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:44.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:44.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:44.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:44.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:44.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:44.430+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:44.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:44.567+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:44.568849+0000) 2024-04-27T03:16:44.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:44.592+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:44.594033+0000) 2024-04-27T03:16:44.889 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:44.721+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:44.723161+0000) 2024-04-27T03:16:44.890 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:44.879+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:44.890 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:44.879+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:44.890 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:44.879+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:45.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:44.880+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:45.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:45.169+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:45.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:45.169+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:45.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:45.169+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:45.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:45.170+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:45.736 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:45.594+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:45.596219+0000) 2024-04-27T03:16:45.736 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:45.469+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:45.736 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:45.469+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:45.736 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:45.469+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:45.736 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:45.470+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:45.737 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:45.571+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:45.573078+0000) 2024-04-27T03:16:46.054 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:45.726+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:45.727918+0000) 2024-04-27T03:16:46.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:45.886+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:46.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:45.886+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:46.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:45.886+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:46.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:45.887+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:46.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:46.155+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:46.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:46.155+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:46.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:46.155+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:46.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:46.156+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:46.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:46.599+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:46.601274+0000) 2024-04-27T03:16:46.645 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:46.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:46.645 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:46.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:46.645 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:46.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:46.645 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:46.462+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:46.645 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:46.576+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:46.578170+0000) 2024-04-27T03:16:46.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:46.729+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:46.730900+0000) 2024-04-27T03:16:46.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:46.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:46.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:46.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:46.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:46.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:47.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:46.924+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:47.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:47.112+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:47.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:47.112+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:47.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:47.112+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:47.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:47.114+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:47.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:47.732+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:47.734221+0000) 2024-04-27T03:16:47.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:47.604+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:47.606263+0000) 2024-04-27T03:16:47.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:47.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:47.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:47.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:47.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:47.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:47.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:47.506+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:47.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:47.581+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:47.583240+0000) 2024-04-27T03:16:48.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:48.078+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:48.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:48.078+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:48.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:48.078+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:48.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:48.078+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:48.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:47.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:48.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:47.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:48.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:47.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:48.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:47.962+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:48.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:48.604+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:48.606689+0000) 2024-04-27T03:16:48.903 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:48.732+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:48.734484+0000) 2024-04-27T03:16:48.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:48.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:48.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:48.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:48.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:48.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:48.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:48.521+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:48.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:48.581+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:48.583660+0000) 2024-04-27T03:16:49.214 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:48.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:49.214 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:48.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:49.214 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:48.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:49.214 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:48.952+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:49.214 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:49.083+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:49.214 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:49.083+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:49.214 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:49.083+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:49.214 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:49.084+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:49.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:49.207+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:49.209353+0000) 2024-04-27T03:16:49.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:49.207+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:49.209070+0000) 2024-04-27T03:16:49.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:49.207+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:49.209705+0000) 2024-04-27T03:16:49.480 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:16:49.209+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57756 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:16:49.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T03:16:49.608+0000 7ff0e4ed9700 0 monclient(hunting): authenticate timed out after 300 2024-04-27T03:16:49.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:49.605+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:49.606930+0000) 2024-04-27T03:16:49.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:49.732+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:49.734709+0000) 2024-04-27T03:16:49.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:49.942+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:49.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:49.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:49.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:49.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:49.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:49.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:49.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:49.520+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:49.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:49.582+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:49.583915+0000) 2024-04-27T03:16:50.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:49.942+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:50.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:49.942+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:50.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:49.942+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:50.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:50.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:50.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:50.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:50.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:50.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:50.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:50.128+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:50.922 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:50.737+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:50.738060+0000) 2024-04-27T03:16:50.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:50.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:50.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:50.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:50.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:50.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:50.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:50.547+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:50.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:50.586+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:50.587750+0000) 2024-04-27T03:16:50.923 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:50.607+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:50.609035+0000) 2024-04-27T03:16:51.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:50.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:51.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:50.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:51.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:50.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:51.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:50.913+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:51.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:51.158+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:51.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:51.158+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:51.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:51.158+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:51.230 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:51.159+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:51.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:51.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:51.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:51.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:51.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:51.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:51.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:51.545+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:51.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:51.591+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:51.591967+0000) 2024-04-27T03:16:51.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:51.609+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:51.610737+0000) 2024-04-27T03:16:51.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:51.740+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:51.741147+0000) 2024-04-27T03:16:51.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:51.902+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:51.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:51.902+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:51.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:51.902+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:52.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:51.903+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:52.168 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:52.120+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:52.169 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:52.120+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:52.169 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:52.120+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:52.169 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:52.120+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:52.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:52.562+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:52.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:52.562+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:52.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:52.562+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:52.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:52.562+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:52.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:52.594+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:52.594999+0000) 2024-04-27T03:16:52.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:52.612+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:52.613503+0000) 2024-04-27T03:16:52.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:52.743+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:52.743888+0000) 2024-04-27T03:16:53.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:52.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:53.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:52.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:53.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:52.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:53.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:52.883+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:53.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:53.153+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:53.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:53.153+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:53.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:53.153+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:53.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:53.154+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:53.799 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:53.743+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:53.744376+0000) 2024-04-27T03:16:53.800 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:53.612+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:53.613778+0000) 2024-04-27T03:16:53.800 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:53.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:53.800 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:53.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:53.800 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:53.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:53.800 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:53.531+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:53.800 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:53.594+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:53.595353+0000) 2024-04-27T03:16:54.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:53.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:54.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:53.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:54.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:53.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:54.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:53.878+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:54.343 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:16:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:16:54.217+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57796 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:16:54.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:54.215+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:54.216531+0000) 2024-04-27T03:16:54.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:54.215+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:54.216564+0000) 2024-04-27T03:16:54.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:54.127+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:54.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:54.127+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:54.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:54.127+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:54.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:54.127+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:54.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:54.214+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:54.215361+0000) 2024-04-27T03:16:54.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:54.522+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:54.601 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:54.522+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:54.601 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:54.522+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:54.601 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:54.523+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:54.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:54.613+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:54.613966+0000) 2024-04-27T03:16:54.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:54.594+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:54.595587+0000) 2024-04-27T03:16:54.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:54.743+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:54.744632+0000) 2024-04-27T03:16:54.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:54.828+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:54.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:54.828+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:54.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:54.828+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:54.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:54.829+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:55.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:55.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:55.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:55.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:55.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:55.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:55.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:55.161+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:55.822 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:55.615+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:55.616071+0000) 2024-04-27T03:16:55.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:55.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:55.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:55.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:55.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:55.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:55.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:55.555+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:55.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:55.596+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:55.597654+0000) 2024-04-27T03:16:55.823 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:55.746+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:55.746900+0000) 2024-04-27T03:16:55.823 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:55.813+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:55.823 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:55.813+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:55.823 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:55.813+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:56.124 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:55.813+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:56.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:56.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:56.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:56.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:56.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:56.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:56.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:56.162+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:56.655 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:56.617+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:56.618213+0000) 2024-04-27T03:16:56.655 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:56.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:56.655 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:56.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:56.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:56.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:56.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:56.566+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:56.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:56.599+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:56.599801+0000) 2024-04-27T03:16:56.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:56.748+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:56.749308+0000) 2024-04-27T03:16:56.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:56.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:56.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:56.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:56.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:56.843+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:56.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:56.843+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:57.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:57.127+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:57.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:57.127+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:57.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:57.127+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:57.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:57.128+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:57.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:57.619+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:57.620342+0000) 2024-04-27T03:16:57.844 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:57.750+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:57.751481+0000) 2024-04-27T03:16:57.844 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:57.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:57.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:57.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:57.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:57.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:57.845 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:57.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:57.845 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:57.527+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:57.845 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:57.603+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:57.604029+0000) 2024-04-27T03:16:58.124 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:57.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:58.124 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:57.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:58.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:57.836+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:58.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:58.155+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:58.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:58.155+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:58.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:58.155+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:58.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:58.157+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:58.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:58.619+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:58.620630+0000) 2024-04-27T03:16:58.861 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:58.751+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:58.751842+0000) 2024-04-27T03:16:58.861 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:58.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:58.861 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:58.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:58.861 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:58.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:58.861 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:58.522+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:58.861 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:58.603+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:58.604422+0000) 2024-04-27T03:16:59.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:58.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:59.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:58.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:59.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:58.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:16:59.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:58.852+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:59.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:59.221+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:59.222387+0000) 2024-04-27T03:16:59.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:59.222+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:59.222867+0000) 2024-04-27T03:16:59.479 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:16:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:16:59.224+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57836 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:16:59.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:59.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:59.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:59.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:59.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:59.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:16:59.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:59.170+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:59.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:59.223+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:59.224050+0000) 2024-04-27T03:16:59.865 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:16:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:16:59.620+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:59.620826+0000) 2024-04-27T03:16:59.865 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:59.751+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:59.752019+0000) 2024-04-27T03:16:59.865 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:59.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:59.865 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:59.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:59.865 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:59.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:16:59.865 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:59.564+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:16:59.865 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:16:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:16:59.603+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:16:59.604600+0000) 2024-04-27T03:17:00.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:59.855+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:00.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:59.855+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:00.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:59.855+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:00.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:16:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:16:59.856+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:00.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:00.205+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:00.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:00.205+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:00.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:00.205+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:00.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:00.206+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:00.845 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:00.622+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:00.623440+0000) 2024-04-27T03:17:00.845 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:00.754+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:00.755082+0000) 2024-04-27T03:17:00.845 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:00.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:00.845 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:00.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:00.845 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:00.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:00.845 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:00.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:00.845 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:00.517+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:00.845 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:00.606+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:00.607408+0000) 2024-04-27T03:17:01.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:00.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:01.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:00.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:01.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:00.836+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:01.419 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:01.189+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:01.419 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:01.189+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:01.419 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:01.189+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:01.419 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:01.190+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:01.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:01.625+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:01.626365+0000) 2024-04-27T03:17:01.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:01.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:01.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:01.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:01.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:01.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:01.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:01.535+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:01.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:01.609+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:01.610141+0000) 2024-04-27T03:17:02.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:01.758+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:01.759324+0000) 2024-04-27T03:17:02.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:01.855+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:02.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:01.855+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:02.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:01.855+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:02.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:01.856+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:02.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:02.154+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:02.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:02.154+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:02.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:02.154+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:02.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:02.155+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:02.876 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:02.760+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:02.761498+0000) 2024-04-27T03:17:02.876 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:02.576+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:02.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:02.576+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:02.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:02.576+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:02.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:02.577+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:02.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:02.611+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:02.612415+0000) 2024-04-27T03:17:02.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:02.627+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:02.628494+0000) 2024-04-27T03:17:03.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:02.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:03.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:02.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:03.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:02.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:03.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:02.867+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:03.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:03.186+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:03.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:03.186+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:03.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:03.186+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:03.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:03.187+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:03.833 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:03.628+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:03.628940+0000) 2024-04-27T03:17:03.834 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:03.761+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:03.761961+0000) 2024-04-27T03:17:03.834 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:03.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:03.834 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:03.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:03.834 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:03.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:03.834 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:03.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:03.834 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:03.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:03.834 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:03.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:03.834 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:03.564+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:03.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:03.612+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:03.612905+0000) 2024-04-27T03:17:04.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:03.825+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:04.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:04.228+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:04.229671+0000) 2024-04-27T03:17:04.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:04.229+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:04.230650+0000) 2024-04-27T03:17:04.479 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:17:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:17:04.231+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57875 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:17:04.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:04.213+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:04.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:04.213+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:04.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:04.213+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:04.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:04.214+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:04.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:04.229+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:04.230561+0000) 2024-04-27T03:17:04.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:04.628+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:04.629118+0000) 2024-04-27T03:17:04.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:04.761+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:04.762170+0000) 2024-04-27T03:17:04.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:04.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:04.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:04.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:04.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:04.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:04.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:04.814+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:04.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:04.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:04.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:04.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:04.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:04.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:04.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:04.575+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:04.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:04.612+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:04.613078+0000) 2024-04-27T03:17:05.601 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:05.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:05.601 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:05.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:05.601 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:05.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:05.601 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:05.265+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:05.857 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:05.631+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:05.632145+0000) 2024-04-27T03:17:05.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:05.763+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:05.764425+0000) 2024-04-27T03:17:05.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:05.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:05.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:05.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:05.858 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:05.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:05.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:05.592+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:05.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:05.592+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:05.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:05.592+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:05.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:05.592+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:05.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:05.614+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:05.615322+0000) 2024-04-27T03:17:06.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:05.848+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:06.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:06.257+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:06.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:06.257+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:06.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:06.257+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:06.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:06.258+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:06.627 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:06.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:06.627 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:06.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:06.627 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:06.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:06.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:06.595+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:06.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:06.618+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:06.619581+0000) 2024-04-27T03:17:06.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:06.634+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:06.635190+0000) 2024-04-27T03:17:06.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:06.765+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:06.766531+0000) 2024-04-27T03:17:06.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:06.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:06.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:06.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:06.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:06.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:07.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:06.879+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:07.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:07.277+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:07.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:07.277+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:07.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:07.277+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:07.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:07.277+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:07.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:07.769+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:07.769981+0000) 2024-04-27T03:17:07.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:07.636+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:07.637486+0000) 2024-04-27T03:17:07.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:07.603+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:07.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:07.603+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:07.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:07.603+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:07.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:07.604+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:07.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:07.623+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:07.623866+0000) 2024-04-27T03:17:08.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:07.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:08.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:07.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:08.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:07.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:08.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:07.888+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:08.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:08.306+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:08.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:08.306+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:08.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:08.306+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:08.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:08.306+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:08.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:08.636+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:08.637780+0000) 2024-04-27T03:17:08.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:08.769+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:08.770530+0000) 2024-04-27T03:17:08.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:08.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:08.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:08.615+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:08.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:08.615+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:08.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:08.615+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:08.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:08.616+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:08.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:08.623+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:08.624171+0000) 2024-04-27T03:17:09.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:08.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:09.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:08.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:09.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:08.911+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:09.478 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:17:09.238+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57915 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:17:09.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:09.237+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:09.238283+0000) 2024-04-27T03:17:09.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:09.235+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:09.236630+0000) 2024-04-27T03:17:09.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:09.236+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:09.237446+0000) 2024-04-27T03:17:09.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:09.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:09.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:09.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:09.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:09.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:09.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:09.314+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:09.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:09.637+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:09.637961+0000) 2024-04-27T03:17:09.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:09.769+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:09.770785+0000) 2024-04-27T03:17:09.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:09.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:09.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:09.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:09.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:09.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:09.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:09.570+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:09.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:09.570+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:09.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:09.570+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:09.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:09.571+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:09.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:09.623+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:09.624436+0000) 2024-04-27T03:17:10.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:09.916+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:10.542 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:10.296+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:10.542 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:10.296+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:10.542 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:10.296+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:10.542 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:10.297+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:10.923 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:10.639+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:10.640082+0000) 2024-04-27T03:17:10.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:10.774+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:10.775003+0000) 2024-04-27T03:17:10.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:10.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:10.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:10.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:10.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:10.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:10.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:10.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:10.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:10.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:10.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:10.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:10.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:10.534+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:10.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:10.627+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:10.628676+0000) 2024-04-27T03:17:11.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:10.914+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:11.602 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:11.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:11.602 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:11.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:11.602 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:11.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:11.602 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:11.492+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:11.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:11.298+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:11.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:11.298+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:11.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:11.298+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:11.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:11.298+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:11.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:11.629+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:11.630740+0000) 2024-04-27T03:17:11.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:11.641+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:11.642735+0000) 2024-04-27T03:17:11.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:11.777+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:11.778106+0000) 2024-04-27T03:17:12.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:11.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:12.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:11.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:12.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:11.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:12.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:11.891+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:12.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:12.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:12.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:12.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:12.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:12.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:12.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:12.314+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:12.788 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:12.645+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:12.645947+0000) 2024-04-27T03:17:12.788 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:12.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:12.789 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:12.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:12.789 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:12.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:12.789 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:12.481+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:12.789 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:12.633+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:12.634102+0000) 2024-04-27T03:17:13.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:12.779+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:12.780315+0000) 2024-04-27T03:17:13.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:12.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:13.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:12.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:13.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:12.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:13.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:12.935+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:13.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:13.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:13.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:13.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:13.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:13.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:13.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:13.492+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:13.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:13.299+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:13.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:13.299+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:13.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:13.299+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:13.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:13.300+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:13.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:13.633+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:13.634417+0000) 2024-04-27T03:17:13.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:13.645+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:13.646238+0000) 2024-04-27T03:17:13.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:13.779+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:13.780709+0000) 2024-04-27T03:17:14.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:13.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:14.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:13.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:14.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:13.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:14.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:13.954+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:14.602 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:17:14.245+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57954 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:17:14.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:14.243+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:14.244014+0000) 2024-04-27T03:17:14.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:14.244+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:14.245165+0000) 2024-04-27T03:17:14.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:14.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:14.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:14.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:14.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:14.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:14.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:14.459+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:14.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:14.243+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:14.243860+0000) 2024-04-27T03:17:14.604 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:14.311+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:14.604 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:14.311+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:14.604 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:14.311+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:14.604 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:14.311+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:14.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:14.780+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:14.780929+0000) 2024-04-27T03:17:14.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:14.633+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:14.634613+0000) 2024-04-27T03:17:14.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:14.645+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:14.646425+0000) 2024-04-27T03:17:15.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:14.984+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:15.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:14.984+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:15.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:14.984+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:15.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:14.985+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:15.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:15.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:15.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:15.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:15.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:15.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:15.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:15.440+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:15.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:15.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:15.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:15.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:15.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:15.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:15.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:15.308+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:15.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:15.649+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:15.650769+0000) 2024-04-27T03:17:15.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:15.782+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:15.783600+0000) 2024-04-27T03:17:15.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:15.636+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:15.637677+0000) 2024-04-27T03:17:16.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:15.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:16.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:15.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:16.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:15.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:16.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:15.973+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:16.550 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:16.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:16.550 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:16.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:16.550 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:16.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:16.550 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:16.417+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:16.550 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:16.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:16.551 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:16.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:16.551 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:16.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:16.551 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:16.274+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:16.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:16.639+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:16.639821+0000) 2024-04-27T03:17:16.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:16.652+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:16.652828+0000) 2024-04-27T03:17:16.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:16.785+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:16.786395+0000) 2024-04-27T03:17:17.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:17.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:17.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:17.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:17.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:17.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:17.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:17.016+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:17.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:17.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:17.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:17.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:17.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:17.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:17.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:17.462+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:17.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:17.253+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:17.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:17.253+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:17.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:17.253+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:17.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:17.254+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:17.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:17.641+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:17.642151+0000) 2024-04-27T03:17:17.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:17.656+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:17.657206+0000) 2024-04-27T03:17:17.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:17.788+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:17.789197+0000) 2024-04-27T03:17:18.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:18.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:18.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:18.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:18.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:18.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:18.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:18.031+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:18.492 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:18.224+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:18.492 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:18.224+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:18.492 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:18.224+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:18.493 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:18.225+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:18.797 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:18.656+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:18.657530+0000) 2024-04-27T03:17:18.797 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:18.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:18.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:18.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:18.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:18.483+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:18.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:18.484+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:18.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:18.641+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:18.642742+0000) 2024-04-27T03:17:19.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:18.788+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:18.789541+0000) 2024-04-27T03:17:19.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:19.250+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:19.250908+0000) 2024-04-27T03:17:19.383 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:17:19.251+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 57994 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:17:19.383 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:19.221+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:19.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:19.221+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:19.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:19.221+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:19.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:19.221+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:19.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:19.249+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:19.250501+0000) 2024-04-27T03:17:19.384 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:19.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:19.384 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:19.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:19.384 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:19.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:19.384 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:19.042+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:19.384 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:19.249+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:19.250770+0000) 2024-04-27T03:17:19.650 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:19.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:19.651 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:19.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:19.651 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:19.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:19.651 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:19.467+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:19.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:19.642+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:19.642932+0000) 2024-04-27T03:17:19.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:19.656+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:19.657707+0000) 2024-04-27T03:17:19.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:19.789+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:19.789932+0000) 2024-04-27T03:17:20.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:20.212+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:20.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:20.212+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:20.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:20.212+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:20.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:20.213+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:20.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:20.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:20.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:20.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:20.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:20.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:20.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:20.022+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:20.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:20.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:20.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:20.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:20.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:20.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:20.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:20.499+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:20.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:20.644+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:20.645218+0000) 2024-04-27T03:17:20.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:20.791+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:20.792064+0000) 2024-04-27T03:17:20.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:20.659+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:20.659879+0000) 2024-04-27T03:17:21.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:21.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:21.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:21.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:21.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:21.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:21.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:21.021+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:21.509 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:21.242+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:21.510 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:21.242+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:21.510 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:21.242+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:21.510 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:21.243+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:21.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:21.661+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:21.662029+0000) 2024-04-27T03:17:21.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:21.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:21.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:21.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:21.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:21.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:21.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:21.501+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:21.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:21.646+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:21.647331+0000) 2024-04-27T03:17:22.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:21.795+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:21.796276+0000) 2024-04-27T03:17:22.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:22.249+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:22.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:22.249+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:22.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:22.249+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:22.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:22.249+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:22.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:22.064+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:22.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:22.064+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:22.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:22.064+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:22.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:22.064+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:22.806 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:22.663+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:22.664228+0000) 2024-04-27T03:17:22.807 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:22.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:22.807 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:22.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:22.807 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:22.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:22.807 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:22.546+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:22.807 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:22.650+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:22.651568+0000) 2024-04-27T03:17:23.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:22.797+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:22.798742+0000) 2024-04-27T03:17:23.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:23.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:23.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:23.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:23.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:23.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:23.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:23.016+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:23.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:23.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:23.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:23.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:23.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:23.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:23.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:23.203+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:23.800 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:23.663+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:23.664535+0000) 2024-04-27T03:17:23.801 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:23.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:23.801 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:23.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:23.801 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:23.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:23.801 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:23.512+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:23.801 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:23.651+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:23.651988+0000) 2024-04-27T03:17:24.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:23.798+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:23.799118+0000) 2024-04-27T03:17:24.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:24.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:24.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:24.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:24.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:24.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:24.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:24.013+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:24.387 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:24.256+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:24.257021+0000) 2024-04-27T03:17:24.387 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:17:24.258+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58033 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:17:24.387 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:24.257+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:24.258482+0000) 2024-04-27T03:17:24.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:24.195+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:24.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:24.195+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:24.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:24.195+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:24.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:24.197+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:24.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:24.257+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:24.257980+0000) 2024-04-27T03:17:24.660 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:24.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:24.660 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:24.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:24.660 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:24.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:24.660 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:24.475+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:24.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:24.651+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:24.652175+0000) 2024-04-27T03:17:24.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:24.663+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:24.664704+0000) 2024-04-27T03:17:24.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:24.798+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:24.799292+0000) 2024-04-27T03:17:25.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:24.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:25.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:24.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:25.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:24.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:25.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:24.966+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:25.514 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:25.238+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:25.514 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:25.238+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:25.514 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:25.238+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:25.514 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:25.239+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:25.765 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:25.666+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:25.666835+0000) 2024-04-27T03:17:25.766 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:25.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:25.766 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:25.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:25.766 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:25.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:25.766 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:25.506+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:25.766 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:25.653+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:25.654622+0000) 2024-04-27T03:17:26.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:25.803+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:25.804005+0000) 2024-04-27T03:17:26.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:25.920+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:26.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:25.920+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:26.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:25.920+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:26.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:25.920+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:26.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:26.262+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:26.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:26.262+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:26.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:26.262+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:26.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:26.263+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:26.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:26.543+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:26.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:26.543+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:26.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:26.543+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:26.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:26.544+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:26.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:26.655+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:26.656696+0000) 2024-04-27T03:17:26.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:26.669+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:26.670139+0000) 2024-04-27T03:17:26.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:26.805+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:26.806782+0000) 2024-04-27T03:17:26.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:26.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:26.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:26.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:26.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:26.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:26.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:26.876+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:27.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:27.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:27.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:27.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:27.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:27.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:27.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:27.216+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:27.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:27.671+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:27.672516+0000) 2024-04-27T03:17:27.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:27.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:27.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:27.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:27.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:27.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:27.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:27.518+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:27.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:27.660+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:27.660849+0000) 2024-04-27T03:17:28.127 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:27.809+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:27.810118+0000) 2024-04-27T03:17:28.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:27.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:28.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:27.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:28.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:27.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:28.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:27.869+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:28.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:28.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:28.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:28.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:28.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:28.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:28.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:28.219+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:28.818 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:28.672+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:28.672990+0000) 2024-04-27T03:17:28.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:28.473+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:28.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:28.473+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:28.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:28.473+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:28.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:28.473+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:28.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:28.660+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:28.661475+0000) 2024-04-27T03:17:29.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:28.809+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:28.810663+0000) 2024-04-27T03:17:29.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:28.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:29.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:28.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:29.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:28.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:29.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:28.857+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:29.394 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:17:29.266+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58073 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:17:29.395 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:29.263+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:29.264207+0000) 2024-04-27T03:17:29.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:29.264+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:29.264926+0000) 2024-04-27T03:17:29.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:29.235+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:29.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:29.235+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:29.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:29.236+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:29.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:29.236+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:29.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:29.264+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:29.265099+0000) 2024-04-27T03:17:29.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:29.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:29.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:29.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:29.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:29.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:29.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:29.430+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:29.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:29.672+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:29.673253+0000) 2024-04-27T03:17:29.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:29.660+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:29.661740+0000) 2024-04-27T03:17:29.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:29.810+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:29.810850+0000) 2024-04-27T03:17:29.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:29.853+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:29.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:29.853+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:29.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:29.853+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:29.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:29.854+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:30.604 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:30.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:30.604 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:30.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:30.604 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:30.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:30.604 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:30.464+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:30.604 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:30.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:30.604 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:30.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:30.604 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:30.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:30.605 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:30.248+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:30.907 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:30.663+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:30.664094+0000) 2024-04-27T03:17:30.907 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:30.812+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:30.813278+0000) 2024-04-27T03:17:30.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:30.675+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:30.676019+0000) 2024-04-27T03:17:31.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:30.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:31.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:30.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:31.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:30.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:31.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:30.899+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:31.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:31.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:31.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:31.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:31.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:31.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:31.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:31.201+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:31.824 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:31.678+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:31.678916+0000) 2024-04-27T03:17:31.824 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:31.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:31.824 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:31.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:31.824 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:31.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:31.824 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:31.512+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:31.824 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:31.665+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:31.666265+0000) 2024-04-27T03:17:32.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:31.815+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:31.816264+0000) 2024-04-27T03:17:32.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:31.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:32.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:31.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:32.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:31.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:32.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:31.865+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:32.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:32.169+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:32.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:32.169+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:32.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:32.169+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:32.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:32.170+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:32.826 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:32.681+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:32.682122+0000) 2024-04-27T03:17:32.827 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:32.498+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:32.827 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:32.498+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:32.827 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:32.498+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:32.827 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:32.499+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:32.827 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:32.669+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:32.670539+0000) 2024-04-27T03:17:33.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:32.818+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:32.818824+0000) 2024-04-27T03:17:33.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:32.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:33.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:32.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:33.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:32.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:33.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:32.868+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:33.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:33.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:33.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:33.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:33.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:33.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:33.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:33.162+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:33.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:33.681+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:33.682594+0000) 2024-04-27T03:17:33.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:33.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:33.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:33.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:33.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:33.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:33.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:33.529+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:33.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:33.670+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:33.671042+0000) 2024-04-27T03:17:34.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:33.818+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:33.819267+0000) 2024-04-27T03:17:34.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:33.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:34.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:33.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:34.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:33.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:34.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:33.900+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:34.400 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:17:34.273+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58112 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:17:34.401 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:34.270+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:34.271669+0000) 2024-04-27T03:17:34.401 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:34.270+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:34.271535+0000) 2024-04-27T03:17:34.401 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:34.148+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:34.401 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:34.148+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:34.401 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:34.148+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:34.401 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:34.149+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:34.401 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:34.271+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:34.272148+0000) 2024-04-27T03:17:34.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:34.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:34.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:34.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:34.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:34.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:34.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:34.541+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:34.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:34.670+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:34.671231+0000) 2024-04-27T03:17:34.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:34.681+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:34.682773+0000) 2024-04-27T03:17:34.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:34.818+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:34.819453+0000) 2024-04-27T03:17:34.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:34.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:34.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:34.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:34.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:34.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:35.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:34.941+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:35.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:35.102+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:35.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:35.102+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:35.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:35.102+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:35.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:35.103+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:35.830 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:35.686+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:35.687048+0000) 2024-04-27T03:17:35.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:35.559+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:35.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:35.559+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:35.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:35.559+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:35.831 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:35.559+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:35.831 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:35.675+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:35.676203+0000) 2024-04-27T03:17:36.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:35.821+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:35.822565+0000) 2024-04-27T03:17:36.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:35.967+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:36.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:35.967+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:36.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:35.967+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:36.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:35.968+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:36.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:36.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:36.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:36.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:36.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:36.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:36.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:36.062+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:36.833 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:36.688+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:36.689725+0000) 2024-04-27T03:17:36.833 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:36.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:36.833 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:36.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:36.833 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:36.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:36.833 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:36.553+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:36.833 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:36.677+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:36.678273+0000) 2024-04-27T03:17:37.128 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:37.053+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:37.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:37.053+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:37.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:37.053+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:37.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:37.054+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:37.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:36.824+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:36.825229+0000) 2024-04-27T03:17:37.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:36.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:37.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:36.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:37.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:36.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:37.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:36.993+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:37.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:37.691+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:37.691848+0000) 2024-04-27T03:17:37.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:37.827+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:37.828327+0000) 2024-04-27T03:17:37.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:37.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:37.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:37.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:37.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:37.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:37.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:37.580+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:37.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:37.680+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:37.681174+0000) 2024-04-27T03:17:38.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:38.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:38.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:38.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:38.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:38.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:38.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:38.028+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:38.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:38.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:38.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:38.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:38.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:38.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:38.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:38.014+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:38.836 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:38.691+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:38.692343+0000) 2024-04-27T03:17:38.836 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:38.551+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:38.837 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:38.551+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:38.837 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:38.551+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:38.837 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:38.552+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:38.837 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:38.680+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:38.681683+0000) 2024-04-27T03:17:39.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:38.827+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:38.828717+0000) 2024-04-27T03:17:39.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:39.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:39.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:39.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:39.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:39.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:39.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:39.005+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:39.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:39.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:39.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:39.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:39.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:39.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:39.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:39.002+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:39.409 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:17:39.280+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58152 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:17:39.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:39.277+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:39.278720+0000) 2024-04-27T03:17:39.410 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:39.278+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:39.279052+0000) 2024-04-27T03:17:39.410 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:39.277+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:39.278701+0000) 2024-04-27T03:17:39.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:39.543+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:39.690 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:39.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:39.690 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:39.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:39.690 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:39.544+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:39.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:39.681+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:39.681846+0000) 2024-04-27T03:17:39.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:39.828+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:39.828955+0000) 2024-04-27T03:17:39.965 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:39.691+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:39.692540+0000) 2024-04-27T03:17:39.965 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:39.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:39.965 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:39.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:40.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:39.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:40.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:39.957+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:40.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:40.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:40.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:40.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:40.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:40.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:40.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:40.022+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:40.841 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:40.693+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:40.694608+0000) 2024-04-27T03:17:40.841 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:40.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:40.841 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:40.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:40.841 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:40.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:40.841 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:40.535+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:40.841 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:40.683+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:40.684422+0000) 2024-04-27T03:17:41.126 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:40.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:41.126 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:40.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:41.126 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:40.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:41.126 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:40.968+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:41.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:40.832+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:40.833145+0000) 2024-04-27T03:17:41.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:41.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:41.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:41.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:41.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:41.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:41.127 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:41.049+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:41.830 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:41.694+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:41.696733+0000) 2024-04-27T03:17:41.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:41.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:41.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:41.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:41.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:41.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:41.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:41.513+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:41.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:41.685+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:41.687434+0000) 2024-04-27T03:17:42.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:41.833+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:41.835681+0000) 2024-04-27T03:17:42.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:42.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:42.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:42.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:42.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:42.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:42.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:42.001+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:42.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:41.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:42.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:41.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:42.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:41.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:42.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:41.990+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:42.848 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:42.699+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:42.701792+0000) 2024-04-27T03:17:42.848 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:42.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:42.849 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:42.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:42.849 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:42.552+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:42.849 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:42.553+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:42.849 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:42.687+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:42.689608+0000) 2024-04-27T03:17:43.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:42.957+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:43.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:42.957+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:43.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:42.957+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:43.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:42.958+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:43.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:42.839+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:42.840820+0000) 2024-04-27T03:17:43.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:43.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:43.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:43.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:43.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:43.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:43.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:43.037+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:43.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:43.839+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:43.841587+0000) 2024-04-27T03:17:43.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:43.700+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:43.702503+0000) 2024-04-27T03:17:43.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:43.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:43.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:43.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:43.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:43.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:43.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:43.598+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:43.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:43.688+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:43.689976+0000) 2024-04-27T03:17:44.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:44.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:44.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:44.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:44.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:44.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:44.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:44.013+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:44.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:43.970+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:44.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:43.970+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:44.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:43.971+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:44.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:43.971+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:44.585 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:44.285+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:44.287364+0000) 2024-04-27T03:17:44.585 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:44.285+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:44.287320+0000) 2024-04-27T03:17:44.586 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:17:44.286+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58192 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:17:44.586 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:44.283+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:44.285675+0000) 2024-04-27T03:17:44.586 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:44.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:44.586 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:44.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:44.586 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:44.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:44.849 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:44.577+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:44.849 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:44.688+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:44.690184+0000) 2024-04-27T03:17:44.850 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:44.700+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:44.702759+0000) 2024-04-27T03:17:45.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:44.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:45.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:44.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:45.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:44.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:45.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:44.926+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:45.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:44.840+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:44.841855+0000) 2024-04-27T03:17:45.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:45.003+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:45.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:45.003+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:45.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:45.003+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:45.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:45.003+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:45.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:45.843+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:45.845001+0000) 2024-04-27T03:17:45.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:45.703+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:45.704818+0000) 2024-04-27T03:17:45.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:45.961+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:45.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:45.961+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:45.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:45.961+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:45.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:45.613+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:45.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:45.613+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:45.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:45.613+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:45.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:45.614+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:45.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:45.693+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:45.695110+0000) 2024-04-27T03:17:46.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:45.962+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:46.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:45.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:46.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:45.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:46.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:45.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:46.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:45.971+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:46.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:46.705+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:46.707053+0000) 2024-04-27T03:17:46.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:46.845+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:46.847147+0000) 2024-04-27T03:17:46.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:46.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:46.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:46.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:46.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:46.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:46.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:46.623+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:46.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:46.623+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:46.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:46.623+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:46.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:46.624+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:46.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:46.698+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:46.700105+0000) 2024-04-27T03:17:47.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:46.935+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:47.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:46.994+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:47.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:46.994+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:47.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:46.994+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:47.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:46.995+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:47.857 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:47.709+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:47.711434+0000) 2024-04-27T03:17:47.857 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:47.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:47.857 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:47.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:47.857 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:47.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:47.857 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:47.575+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:47.857 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:47.703+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:47.705186+0000) 2024-04-27T03:17:48.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:47.981+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:48.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:47.981+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:48.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:47.981+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:48.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:47.983+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:48.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:47.847+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:47.849338+0000) 2024-04-27T03:17:48.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:47.888+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:48.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:47.888+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:48.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:47.888+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:48.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:47.888+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:48.857 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:48.710+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:48.711811+0000) 2024-04-27T03:17:48.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:48.531+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:48.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:48.531+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:48.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:48.531+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:48.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:48.532+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:48.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:48.704+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:48.705912+0000) 2024-04-27T03:17:49.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:48.848+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:48.849826+0000) 2024-04-27T03:17:49.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:48.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:49.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:48.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:49.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:48.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:49.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:48.864+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:49.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:48.943+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:49.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:48.943+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:49.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:48.943+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:49.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:48.943+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:49.421 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:49.290+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:49.292708+0000) 2024-04-27T03:17:49.421 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:49.291+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:49.293482+0000) 2024-04-27T03:17:49.422 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:17:49.293+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58232 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:17:49.422 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:49.291+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:49.293158+0000) 2024-04-27T03:17:49.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:49.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:49.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:49.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:49.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:49.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:49.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:49.492+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:49.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:49.704+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:49.706174+0000) 2024-04-27T03:17:49.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:49.710+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:49.712021+0000) 2024-04-27T03:17:49.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:49.848+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:49.849984+0000) 2024-04-27T03:17:49.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:49.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:49.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:49.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:49.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:49.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:49.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:49.885+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:50.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:49.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:50.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:49.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:50.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:49.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:50.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:49.990+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:50.606 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:50.488+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:50.606 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:50.488+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:50.606 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:50.488+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:50.606 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:50.489+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:50.859 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:50.707+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:50.708938+0000) 2024-04-27T03:17:50.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:50.712+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:50.713938+0000) 2024-04-27T03:17:51.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:50.960+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:51.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:50.960+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:51.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:50.960+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:51.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:50.960+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:51.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:50.850+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:50.852055+0000) 2024-04-27T03:17:51.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:50.872+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:51.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:50.872+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:51.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:50.872+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:51.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:50.872+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:51.862 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:51.714+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:51.716050+0000) 2024-04-27T03:17:51.862 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:51.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:51.862 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:51.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:51.863 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:51.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:51.863 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:51.485+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:51.863 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:51.709+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:51.711185+0000) 2024-04-27T03:17:52.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:51.852+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:51.854784+0000) 2024-04-27T03:17:52.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:51.899+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:52.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:51.899+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:52.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:51.899+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:52.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:51.900+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:52.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:51.992+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:52.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:51.992+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:52.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:51.992+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:52.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:51.992+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:52.866 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:52.717+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:52.718211+0000) 2024-04-27T03:17:52.866 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:52.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:52.866 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:52.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:52.866 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:52.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:52.866 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:52.518+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:52.866 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:52.713+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:52.713908+0000) 2024-04-27T03:17:53.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:52.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:53.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:52.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:53.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:52.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:53.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:52.956+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:53.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:52.857+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:52.858078+0000) 2024-04-27T03:17:53.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:52.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:53.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:52.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:53.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:52.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:53.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:52.947+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:53.806 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:53.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:53.806 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:53.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:53.806 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:53.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:53.806 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:53.471+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:53.806 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:53.713+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:53.714721+0000) 2024-04-27T03:17:53.807 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:53.717+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:53.718655+0000) 2024-04-27T03:17:54.061 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:53.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:54.061 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:53.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:54.061 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:53.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:54.061 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:53.997+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:54.061 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:53.858+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:53.858866+0000) 2024-04-27T03:17:54.062 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:53.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:54.062 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:53.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:54.062 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:53.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:54.062 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:53.978+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:54.321 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:17:54.301+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58271 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:17:54.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:54.300+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:54.301367+0000) 2024-04-27T03:17:54.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:54.299+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:54.299939+0000) 2024-04-27T03:17:54.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:54.300+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:54.301158+0000) 2024-04-27T03:17:54.606 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:54.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:54.606 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:54.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:54.607 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:54.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:54.607 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:54.464+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:54.866 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:54.714+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:54.715014+0000) 2024-04-27T03:17:54.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:54.718+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:54.718923+0000) 2024-04-27T03:17:55.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:54.963+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:55.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:54.963+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:55.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:54.963+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:55.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:54.964+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:55.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:54.858+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:54.859096+0000) 2024-04-27T03:17:55.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:55.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:55.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:55.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:55.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:55.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:55.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:55.002+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:55.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:55.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:55.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:55.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:55.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:55.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:55.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:55.454+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:55.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:55.718+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:55.719227+0000) 2024-04-27T03:17:55.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:55.860+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:55.861417+0000) 2024-04-27T03:17:55.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:55.723+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:55.724362+0000) 2024-04-27T03:17:55.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:55.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:55.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:55.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:55.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:55.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:55.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:55.968+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:56.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:56.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:56.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:56.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:56.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:56.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:56.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:56.043+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:56.485 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:56.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:56.485 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:56.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:56.486 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:56.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:56.486 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:56.427+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:56.871 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:56.721+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:56.722213+0000) 2024-04-27T03:17:56.872 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:56.726+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:56.727183+0000) 2024-04-27T03:17:57.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:56.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:57.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:56.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:57.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:56.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:57.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:56.938+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:57.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:56.863+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:56.864299+0000) 2024-04-27T03:17:57.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:57.092+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:57.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:57.092+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:57.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:57.092+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:57.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:57.092+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:57.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:57.392+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:57.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:57.392+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:57.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:57.392+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:57.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:57.392+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:57.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:57.723+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:57.724357+0000) 2024-04-27T03:17:57.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:57.865+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:57.866663+0000) 2024-04-27T03:17:57.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:57.728+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:57.729417+0000) 2024-04-27T03:17:57.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:57.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:57.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:57.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:57.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:57.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:57.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:57.928+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:58.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:58.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:58.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:58.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:58.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:58.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:58.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:58.127+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:58.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:58.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:58.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:58.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:58.707 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:58.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:58.707 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:58.443+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:58.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:58.723+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:58.724753+0000) 2024-04-27T03:17:58.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:58.866+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:58.867227+0000) 2024-04-27T03:17:58.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:58.729+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:58.730048+0000) 2024-04-27T03:17:58.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:58.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:58.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:58.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:58.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:58.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:17:59.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:58.969+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:59.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:59.153+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:59.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:59.153+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:59.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:59.153+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:17:59.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:59.154+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:59.607 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:17:59.308+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58313 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:17:59.607 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:59.306+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:59.307601+0000) 2024-04-27T03:17:59.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:59.306+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:59.307208+0000) 2024-04-27T03:17:59.608 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:59.307+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:59.307897+0000) 2024-04-27T03:17:59.608 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:59.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:59.608 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:59.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:59.608 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:59.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:17:59.608 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:59.456+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:17:59.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:17:59.724+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:59.724962+0000) 2024-04-27T03:17:59.875 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:17:59.729+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:59.730246+0000) 2024-04-27T03:18:00.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:00.017+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:00.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:00.017+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:00.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:00.017+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:00.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:00.018+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:00.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:17:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:17:59.866+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:17:59.867422+0000) 2024-04-27T03:18:00.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:00.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:00.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:00.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:00.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:00.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:00.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:00.111+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:00.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:00.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:00.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:00.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:00.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:00.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:00.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:00.420+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:00.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:00.726+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:00.727073+0000) 2024-04-27T03:18:00.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:00.868+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:00.869676+0000) 2024-04-27T03:18:00.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:00.733+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:00.734552+0000) 2024-04-27T03:18:00.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:00.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:00.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:00.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:00.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:00.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:01.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:00.969+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:01.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:01.152+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:01.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:01.152+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:01.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:01.152+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:01.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:01.152+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:01.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:01.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:01.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:01.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:01.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:01.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:01.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:01.428+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:01.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:01.873+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:01.873890+0000) 2024-04-27T03:18:01.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:01.728+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:01.729182+0000) 2024-04-27T03:18:01.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:01.738+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:01.738939+0000) 2024-04-27T03:18:02.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:02.181+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:02.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:02.181+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:02.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:02.181+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:02.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:02.182+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:02.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:01.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:02.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:01.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:02.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:01.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:02.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:01.975+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:02.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:02.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:02.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:02.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:02.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:02.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:02.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:02.440+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:02.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:02.730+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:02.731297+0000) 2024-04-27T03:18:02.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:02.741+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:02.742405+0000) 2024-04-27T03:18:02.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:02.875+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:02.876112+0000) 2024-04-27T03:18:03.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:02.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:03.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:02.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:03.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:02.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:03.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:02.997+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:03.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:03.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:03.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:03.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:03.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:03.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:03.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:03.221+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:03.608 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:03.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:03.608 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:03.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:03.608 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:03.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:03.608 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:03.433+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:03.884 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:03.731+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:03.731822+0000) 2024-04-27T03:18:03.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:03.742+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:03.742973+0000) 2024-04-27T03:18:04.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:03.875+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:03.876630+0000) 2024-04-27T03:18:04.175 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:04.014+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:04.175 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:04.014+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:04.175 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:04.014+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:04.175 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:04.015+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:04.442 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:18:04.315+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58352 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:18:04.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:04.313+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:04.314584+0000) 2024-04-27T03:18:04.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:04.313+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:04.314260+0000) 2024-04-27T03:18:04.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:04.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:04.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:04.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:04.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:04.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:04.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:04.219+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:04.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:04.315+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:04.316203+0000) 2024-04-27T03:18:04.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:04.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:04.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:04.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:04.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:04.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:04.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:04.452+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:04.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:04.731+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:04.732003+0000) 2024-04-27T03:18:04.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:04.742+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:04.743170+0000) 2024-04-27T03:18:04.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:04.876+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:04.876957+0000) 2024-04-27T03:18:05.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:05.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:05.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:05.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:05.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:05.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:05.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:05.206+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:05.466 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:05.025+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:05.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:05.025+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:05.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:05.025+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:05.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:05.026+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:05.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:05.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:05.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:05.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:05.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:05.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:05.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:05.458+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:06.010 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:05.744+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:05.745792+0000) 2024-04-27T03:18:06.010 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:05.733+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:05.734207+0000) 2024-04-27T03:18:06.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:05.878+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:05.879104+0000) 2024-04-27T03:18:06.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:06.002+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:06.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:06.002+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:06.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:06.002+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:06.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:06.003+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:06.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:06.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:06.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:06.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:06.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:06.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:06.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:06.175+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:06.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:06.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:06.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:06.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:06.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:06.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:06.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:06.498+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:06.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:06.736+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:06.737232+0000) 2024-04-27T03:18:06.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:06.749+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:06.750128+0000) 2024-04-27T03:18:06.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:06.880+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:06.881358+0000) 2024-04-27T03:18:07.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:07.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:07.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:07.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:07.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:07.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:07.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:07.179+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:07.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:07.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:07.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:07.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:07.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:07.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:07.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:07.042+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:07.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:07.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:07.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:07.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:07.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:07.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:07.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:07.456+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:07.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:07.739+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:07.740648+0000) 2024-04-27T03:18:07.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:07.752+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:07.753176+0000) 2024-04-27T03:18:07.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:07.882+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:07.883481+0000) 2024-04-27T03:18:08.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:08.207+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:08.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:08.207+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:08.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:08.207+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:08.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:08.208+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:08.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:08.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:08.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:08.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:08.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:08.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:08.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:08.011+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:08.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:08.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:08.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:08.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:08.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:08.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:08.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:08.441+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:08.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:08.752+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:08.753567+0000) 2024-04-27T03:18:08.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:08.883+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:08.883922+0000) 2024-04-27T03:18:08.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:08.740+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:08.741316+0000) 2024-04-27T03:18:09.257 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:09.018+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:09.258 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:09.018+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:09.258 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:09.018+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:09.258 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:09.019+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:09.608 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:18:09.322+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58392 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:18:09.608 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:09.319+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:09.320570+0000) 2024-04-27T03:18:09.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:09.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:09.609 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:09.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:09.609 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:09.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:09.609 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:09.250+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:09.609 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:09.320+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:09.321336+0000) 2024-04-27T03:18:09.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:09.320+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:09.321354+0000) 2024-04-27T03:18:09.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:09.438+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:09.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:09.438+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:09.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:09.438+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:09.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:09.439+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:09.891 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:09.740+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:09.741529+0000) 2024-04-27T03:18:09.891 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:09.752+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:09.753752+0000) 2024-04-27T03:18:10.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:09.883+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:09.884092+0000) 2024-04-27T03:18:10.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:10.034+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:10.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:10.034+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:10.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:10.034+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:10.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:10.035+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:10.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:10.287+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:10.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:10.287+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:10.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:10.287+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:10.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:10.288+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:10.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:10.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:10.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:10.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:10.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:10.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:10.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:10.446+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:10.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:10.742+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:10.743691+0000) 2024-04-27T03:18:10.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:10.755+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:10.756581+0000) 2024-04-27T03:18:10.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:10.886+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:10.886941+0000) 2024-04-27T03:18:11.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:11.005+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:11.285 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:11.005+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:11.285 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:11.005+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:11.285 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:11.006+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:11.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:11.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:11.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:11.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:11.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:11.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:11.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:11.276+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:11.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:11.411+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:11.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:11.411+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:11.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:11.411+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:11.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:11.412+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:11.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:11.744+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:11.745682+0000) 2024-04-27T03:18:11.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:11.757+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:11.758777+0000) 2024-04-27T03:18:12.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:11.888+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:11.889022+0000) 2024-04-27T03:18:12.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:12.050+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:12.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:12.050+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:12.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:12.050+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:12.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:12.051+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:12.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:12.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:12.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:12.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:12.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:12.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:12.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:12.309+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:12.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:12.390+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:12.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:12.390+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:12.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:12.390+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:12.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:12.391+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:12.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:12.747+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:12.747898+0000) 2024-04-27T03:18:12.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:12.762+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:12.763143+0000) 2024-04-27T03:18:13.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:12.890+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:12.891360+0000) 2024-04-27T03:18:13.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:13.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:13.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:13.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:13.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:13.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:13.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:13.076+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:13.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:13.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:13.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:13.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:13.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:13.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:13.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:13.404+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:13.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:13.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:13.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:13.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:13.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:13.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:13.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:13.264+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:13.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:13.747+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:13.748332+0000) 2024-04-27T03:18:13.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:13.763+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:13.763828+0000) 2024-04-27T03:18:14.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:13.891+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:13.891942+0000) 2024-04-27T03:18:14.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:14.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:14.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:14.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:14.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:14.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:14.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:14.067+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:14.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:18:14.329+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58431 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:18:14.457 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:14.329+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:14.329990+0000) 2024-04-27T03:18:14.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:14.328+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:14.328996+0000) 2024-04-27T03:18:14.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:14.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:14.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:14.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:14.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:14.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:14.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:14.404+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:14.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:14.247+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:14.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:14.247+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:14.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:14.247+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:14.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:14.248+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:14.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:14.327+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:14.328266+0000) 2024-04-27T03:18:14.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:14.747+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:14.748514+0000) 2024-04-27T03:18:14.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:14.763+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:14.763988+0000) 2024-04-27T03:18:14.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:14.891+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:14.892131+0000) 2024-04-27T03:18:15.303 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:15.086+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:15.303 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:15.086+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:15.303 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:15.086+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:15.303 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:15.087+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:15.608 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:15.387+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:15.608 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:15.387+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:15.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:15.387+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:15.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:15.388+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:15.609 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:15.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:15.609 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:15.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:15.609 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:15.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:15.609 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:15.295+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:15.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:15.750+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:15.751491+0000) 2024-04-27T03:18:15.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:15.767+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:15.768346+0000) 2024-04-27T03:18:16.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:15.896+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:15.897086+0000) 2024-04-27T03:18:16.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:16.060+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:16.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:16.060+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:16.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:16.060+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:16.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:16.061+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:16.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:16.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:16.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:16.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:16.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:16.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:16.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:16.380+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:16.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:16.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:16.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:16.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:16.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:16.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:16.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:16.309+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:16.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:16.771+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:16.772679+0000) 2024-04-27T03:18:16.907 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:16.752+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:16.753602+0000) 2024-04-27T03:18:17.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:16.898+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:16.899404+0000) 2024-04-27T03:18:17.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:17.035+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:17.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:17.035+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:17.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:17.035+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:17.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:17.036+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:17.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:17.396+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:17.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:17.396+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:17.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:17.396+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:17.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:17.397+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:17.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:17.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:17.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:17.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:17.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:17.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:17.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:17.343+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:17.909 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:17.774+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:17.775423+0000) 2024-04-27T03:18:17.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:17.755+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:17.755814+0000) 2024-04-27T03:18:18.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:17.901+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:17.902223+0000) 2024-04-27T03:18:18.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:18.074+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:18.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:18.074+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:18.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:18.074+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:18.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:18.076+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:18.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:18.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:18.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:18.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:18.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:18.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:18.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:18.374+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:18.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:18.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:18.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:18.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:18.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:18.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:18.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:18.316+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:18.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:18.755+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:18.756246+0000) 2024-04-27T03:18:18.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:18.775+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:18.775849+0000) 2024-04-27T03:18:19.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:18.902+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:18.902929+0000) 2024-04-27T03:18:19.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:19.063+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:19.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:19.063+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:19.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:19.063+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:19.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:19.064+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:19.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:19.334+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:19.334950+0000) 2024-04-27T03:18:19.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:19.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:19.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:19.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:19.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:19.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:19.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:19.334+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:19.335722+0000) 2024-04-27T03:18:19.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:19.274+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:19.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:19.274+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:19.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:19.274+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:19.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:19.275+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:19.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:19.334+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:19.335605+0000) 2024-04-27T03:18:19.428 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:18:19.337+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58472 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:18:19.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:19.419+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:19.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:19.775+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:19.776088+0000) 2024-04-27T03:18:19.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:19.755+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:19.756551+0000) 2024-04-27T03:18:19.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:19.902+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:19.903140+0000) 2024-04-27T03:18:20.262 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:20.114+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:20.262 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:20.114+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:20.262 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:20.114+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:20.262 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:20.115+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:20.609 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:20.263+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:20.609 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:20.263+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:20.609 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:20.263+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:20.609 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:20.263+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:20.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:20.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:20.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:20.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:20.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:20.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:20.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:20.455+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:20.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:20.757+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:20.758776+0000) 2024-04-27T03:18:20.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:20.777+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:20.778770+0000) 2024-04-27T03:18:21.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:20.904+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:20.905292+0000) 2024-04-27T03:18:21.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:21.270+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:21.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:21.270+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:21.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:21.270+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:21.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:21.271+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:21.442 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:21.158+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:21.442 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:21.158+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:21.442 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:21.158+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:21.442 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:21.159+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:21.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:21.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:21.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:21.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:21.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:21.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:21.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:21.434+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:21.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:21.760+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:21.760954+0000) 2024-04-27T03:18:21.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:21.780+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:21.781185+0000) 2024-04-27T03:18:21.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:21.907+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:21.907937+0000) 2024-04-27T03:18:22.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:22.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:22.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:22.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:22.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:22.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:22.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:22.119+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:22.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:22.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:22.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:22.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:22.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:22.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:22.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:22.482+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:22.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:22.271+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:22.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:22.271+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:22.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:22.271+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:22.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:22.272+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:22.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:22.783+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:22.784171+0000) 2024-04-27T03:18:22.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:22.763+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:22.764217+0000) 2024-04-27T03:18:23.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:22.911+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:22.912264+0000) 2024-04-27T03:18:23.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:23.123+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:23.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:23.123+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:23.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:23.123+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:23.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:23.124+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:23.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:23.245+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:23.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:23.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:23.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:23.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:23.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:23.246+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:23.772 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:23.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:23.772 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:23.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:23.772 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:23.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:23.772 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:23.491+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:24.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:23.912+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:23.913184+0000) 2024-04-27T03:18:24.187 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:23.764+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:23.764837+0000) 2024-04-27T03:18:24.187 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:23.783+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:23.784653+0000) 2024-04-27T03:18:24.346 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:24.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:24.347 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:24.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:24.347 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:24.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:24.347 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:24.085+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:24.347 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:24.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:24.347 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:24.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:24.347 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:24.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:24.348 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:24.208+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:24.610 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:18:24.344+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58511 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:18:24.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:24.344+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:24.345037+0000) 2024-04-27T03:18:24.611 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:24.341+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:24.342498+0000) 2024-04-27T03:18:24.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:24.342+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:24.343212+0000) 2024-04-27T03:18:24.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:24.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:24.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:24.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:24.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:24.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:24.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:24.443+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:24.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:24.764+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:24.765064+0000) 2024-04-27T03:18:24.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:24.784+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:24.784891+0000) 2024-04-27T03:18:25.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:24.912+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:24.913397+0000) 2024-04-27T03:18:25.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:25.127+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:25.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:25.127+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:25.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:25.127+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:25.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:25.129+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:25.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:25.181+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:25.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:25.181+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:25.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:25.181+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:25.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:25.182+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:25.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:25.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:25.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:25.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:25.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:25.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:25.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:25.437+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:25.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:25.766+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:25.767352+0000) 2024-04-27T03:18:25.981 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:25.916+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:25.917456+0000) 2024-04-27T03:18:25.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:25.786+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:25.787226+0000) 2024-04-27T03:18:26.376 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:26.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:26.376 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:26.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:26.376 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:26.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:26.376 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:26.181+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:26.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:26.119+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:26.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:26.119+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:26.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:26.119+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:26.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:26.120+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:26.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:26.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:26.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:26.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:26.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:26.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:26.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:26.400+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:26.981 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:26.768+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:26.769422+0000) 2024-04-27T03:18:26.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:26.788+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:26.789347+0000) 2024-04-27T03:18:26.981 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:26.920+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:26.921702+0000) 2024-04-27T03:18:27.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:27.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:27.254 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:27.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:27.254 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:27.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:27.254 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:27.221+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:27.254 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:27.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:27.254 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:27.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:27.254 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:27.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:27.254 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:27.098+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:27.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:27.384+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:27.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:27.384+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:27.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:27.384+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:27.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:27.386+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:27.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:27.771+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:27.772356+0000) 2024-04-27T03:18:27.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:27.791+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:27.792336+0000) 2024-04-27T03:18:28.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:27.926+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:27.926821+0000) 2024-04-27T03:18:28.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:28.142+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:28.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:28.142+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:28.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:28.142+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:28.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:28.143+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:28.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:28.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:28.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:28.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:28.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:28.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:28.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:28.241+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:28.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:28.348+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:28.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:28.348+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:28.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:28.348+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:28.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:28.349+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:28.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:28.772+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:28.772802+0000) 2024-04-27T03:18:28.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:28.792+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:28.792917+0000) 2024-04-27T03:18:29.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:28.926+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:28.927555+0000) 2024-04-27T03:18:29.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:29.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:29.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:29.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:29.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:29.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:29.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:29.097+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:29.478 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:18:29.351+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58551 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:18:29.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:29.349+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:29.350088+0000) 2024-04-27T03:18:29.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:29.213+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:29.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:29.213+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:29.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:29.213+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:29.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:29.214+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:29.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:29.349+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:29.350634+0000) 2024-04-27T03:18:29.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:29.348+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:29.349688+0000) 2024-04-27T03:18:29.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:29.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:29.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:29.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:29.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:29.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:29.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:29.354+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:29.981 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:29.772+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:29.773006+0000) 2024-04-27T03:18:29.981 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:29.927+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:29.927829+0000) 2024-04-27T03:18:29.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:29.792+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:29.793122+0000) 2024-04-27T03:18:30.262 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:30.219+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:30.262 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:30.219+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:30.262 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:30.219+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:30.263 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:30.220+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:30.263 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:30.071+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:30.263 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:30.071+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:30.263 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:30.071+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:30.263 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:30.072+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:30.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:30.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:30.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:30.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:30.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:30.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:30.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:30.390+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:30.937 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:30.776+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:30.776984+0000) 2024-04-27T03:18:30.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:30.794+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:30.795342+0000) 2024-04-27T03:18:31.203 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:30.929+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:30.929930+0000) 2024-04-27T03:18:31.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:31.106+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:31.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:31.106+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:31.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:31.106+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:31.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:31.107+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:31.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:31.195+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:31.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:31.195+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:31.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:31.195+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:31.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:31.196+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:31.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:31.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:31.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:31.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:31.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:31.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:31.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:31.397+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:31.939 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:31.778+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:31.779093+0000) 2024-04-27T03:18:31.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:31.796+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:31.797509+0000) 2024-04-27T03:18:32.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:31.931+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:31.932016+0000) 2024-04-27T03:18:32.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:32.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:32.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:32.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:32.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:32.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:32.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:32.130+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:32.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:32.231+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:32.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:32.231+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:32.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:32.231+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:32.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:32.232+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:32.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:32.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:32.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:32.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:32.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:32.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:32.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:32.396+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:32.942 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:32.780+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:32.781287+0000) 2024-04-27T03:18:32.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:32.801+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:32.801841+0000) 2024-04-27T03:18:33.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:32.933+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:32.934757+0000) 2024-04-27T03:18:33.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:33.114+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:33.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:33.114+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:33.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:33.114+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:33.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:33.115+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:33.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:33.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:33.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:33.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:33.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:33.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:33.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:33.280+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:33.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:33.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:33.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:33.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:33.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:33.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:33.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:33.403+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:33.942 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:33.780+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:33.781728+0000) 2024-04-27T03:18:33.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:33.801+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:33.802606+0000) 2024-04-27T03:18:34.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:33.934+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:33.935401+0000) 2024-04-27T03:18:34.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:34.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:34.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:34.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:34.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:34.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:34.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:34.139+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:34.611 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:18:34.358+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58590 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:18:34.611 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:34.356+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:34.357631+0000) 2024-04-27T03:18:34.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:34.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:34.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:34.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:34.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:34.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:34.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:34.258+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:34.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:34.356+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:34.357023+0000) 2024-04-27T03:18:34.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:34.356+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:34.357242+0000) 2024-04-27T03:18:34.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:34.391+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:34.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:34.391+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:34.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:34.391+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:34.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:34.393+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:34.942 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:34.781+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:34.781889+0000) 2024-04-27T03:18:34.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:34.802+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:34.802812+0000) 2024-04-27T03:18:35.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:34.934+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:34.935621+0000) 2024-04-27T03:18:35.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:35.180+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:35.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:35.180+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:35.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:35.180+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:35.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:35.182+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:35.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:35.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:35.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:35.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:35.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:35.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:35.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:35.442+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:35.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:35.263+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:35.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:35.263+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:35.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:35.263+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:35.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:35.264+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:35.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:35.783+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:35.783935+0000) 2024-04-27T03:18:35.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:35.804+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:35.804843+0000) 2024-04-27T03:18:36.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:35.938+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:35.939082+0000) 2024-04-27T03:18:36.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:36.296+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:36.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:36.296+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:36.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:36.296+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:36.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:36.297+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:36.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:36.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:36.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:36.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:36.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:36.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:36.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:36.447+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:36.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:36.224+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:36.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:36.224+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:36.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:36.224+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:36.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:36.225+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:36.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:36.806+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:36.806960+0000) 2024-04-27T03:18:36.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:36.786+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:36.787178+0000) 2024-04-27T03:18:37.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:36.940+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:36.941375+0000) 2024-04-27T03:18:37.611 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:37.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:37.611 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:37.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:37.611 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:37.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:37.611 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:37.227+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:37.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:37.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:37.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:37.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:37.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:37.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:37.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:37.425+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:37.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:37.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:37.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:37.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:37.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:37.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:37.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:37.338+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:37.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:37.789+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:37.790304+0000) 2024-04-27T03:18:37.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:37.809+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:37.810144+0000) 2024-04-27T03:18:38.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:37.942+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:37.943751+0000) 2024-04-27T03:18:38.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:38.205+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:38.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:38.205+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:38.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:38.205+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:38.466 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:38.205+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:38.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:38.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:38.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:38.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:38.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:38.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:38.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:38.313+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:38.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:38.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:38.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:38.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:38.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:38.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:38.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:38.458+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:39.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:38.809+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:38.810735+0000) 2024-04-27T03:18:39.066 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:38.789+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:38.790752+0000) 2024-04-27T03:18:39.066 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:38.943+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:38.944514+0000) 2024-04-27T03:18:39.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:39.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:39.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:39.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:39.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:39.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:39.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:39.283+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:39.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:39.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:39.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:39.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:39.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:39.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:39.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:39.222+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:39.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:18:39.365+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58630 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:18:39.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:39.363+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:39.364429+0000) 2024-04-27T03:18:39.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:39.363+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:39.363877+0000) 2024-04-27T03:18:39.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:39.363+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:39.363890+0000) 2024-04-27T03:18:39.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:39.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:39.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:39.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:39.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:39.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:39.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:39.466+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:40.066 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:39.790+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:39.790965+0000) 2024-04-27T03:18:40.066 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:39.810+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:39.810897+0000) 2024-04-27T03:18:40.066 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:39.943+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:39.944700+0000) 2024-04-27T03:18:40.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:40.229+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:40.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:40.229+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:40.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:40.229+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:40.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:40.230+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:40.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:40.515+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:40.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:40.515+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:40.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:40.515+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:40.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:40.516+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:40.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:40.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:40.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:40.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:40.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:40.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:40.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:40.311+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:40.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:40.792+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:40.793077+0000) 2024-04-27T03:18:40.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:40.812+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:40.813127+0000) 2024-04-27T03:18:41.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:40.946+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:40.946841+0000) 2024-04-27T03:18:41.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:41.205+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:41.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:41.205+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:41.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:41.205+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:41.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:41.205+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:41.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:41.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:41.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:41.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:41.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:41.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:41.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:41.311+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:41.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:41.531+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:41.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:41.531+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:41.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:41.531+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:41.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:41.533+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:42.066 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:41.794+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:41.795179+0000) 2024-04-27T03:18:42.066 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:41.950+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:41.950973+0000) 2024-04-27T03:18:42.066 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:41.814+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:41.815333+0000) 2024-04-27T03:18:42.347 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:42.245+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:42.347 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:42.245+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:42.348 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:42.245+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:42.348 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:42.245+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:42.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:42.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:42.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:42.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:42.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:42.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:42.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:42.339+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:42.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:42.523+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:42.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:42.523+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:42.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:42.523+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:42.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:42.525+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:42.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:42.796+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:42.797627+0000) 2024-04-27T03:18:42.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:42.816+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:42.817563+0000) 2024-04-27T03:18:43.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:42.954+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:42.955186+0000) 2024-04-27T03:18:43.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:43.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:43.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:43.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:43.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:43.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:43.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:43.239+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:43.577 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:43.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:43.577 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:43.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:43.577 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:43.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:43.578 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:43.348+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:43.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:43.816+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:43.818062+0000) 2024-04-27T03:18:43.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:43.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:43.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:43.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:43.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:43.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:43.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:43.570+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:43.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:43.796+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:43.798124+0000) 2024-04-27T03:18:44.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:43.954+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:43.956020+0000) 2024-04-27T03:18:44.590 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:18:44.371+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58669 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:18:44.590 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:44.369+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:44.371510+0000) 2024-04-27T03:18:44.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:44.368+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:44.370778+0000) 2024-04-27T03:18:44.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:44.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:44.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:44.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:44.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:44.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:44.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:44.394+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:44.591 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:44.262+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:44.591 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:44.262+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:44.591 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:44.262+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:44.592 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:44.262+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:44.592 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:44.368+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:44.370711+0000) 2024-04-27T03:18:44.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:44.816+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:44.818250+0000) 2024-04-27T03:18:44.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:44.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:44.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:44.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:44.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:44.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:44.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:44.582+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:44.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:44.796+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:44.798388+0000) 2024-04-27T03:18:45.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:44.954+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:44.956237+0000) 2024-04-27T03:18:45.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:45.427+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:45.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:45.427+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:45.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:45.427+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:45.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:45.427+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:45.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:45.266+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:45.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:45.266+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:45.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:45.266+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:45.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:45.267+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:45.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:45.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:45.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:45.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:45.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:45.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:45.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:45.626+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:45.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:45.798+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:45.800590+0000) 2024-04-27T03:18:45.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:45.819+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:45.820849+0000) 2024-04-27T03:18:46.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:45.956+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:45.958756+0000) 2024-04-27T03:18:46.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:46.292+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:46.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:46.292+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:46.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:46.292+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:46.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:46.292+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:46.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:46.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:46.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:46.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:46.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:46.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:46.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:46.456+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:46.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:46.822+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:46.824076+0000) 2024-04-27T03:18:46.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:46.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:46.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:46.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:46.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:46.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:46.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:46.649+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:46.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:46.803+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:46.804812+0000) 2024-04-27T03:18:47.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:46.960+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:46.962107+0000) 2024-04-27T03:18:47.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:47.431+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:47.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:47.431+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:47.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:47.431+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:47.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:47.431+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:47.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:47.274+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:47.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:47.274+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:47.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:47.274+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:47.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:47.275+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:47.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:47.824+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:47.826738+0000) 2024-04-27T03:18:47.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:47.653+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:47.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:47.653+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:47.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:47.653+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:47.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:47.655+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:47.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:47.805+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:47.807117+0000) 2024-04-27T03:18:48.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:47.964+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:47.966297+0000) 2024-04-27T03:18:48.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:48.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:48.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:48.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:48.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:48.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:48.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:48.389+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:48.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:48.325+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:48.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:48.325+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:48.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:48.325+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:48.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:48.325+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:48.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:48.695+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:48.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:48.695+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:48.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:48.695+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:48.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:48.696+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:48.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:48.806+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:48.807995+0000) 2024-04-27T03:18:48.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:48.825+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:48.827240+0000) 2024-04-27T03:18:49.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:48.965+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:48.967152+0000) 2024-04-27T03:18:49.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:49.376+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:49.377926+0000) 2024-04-27T03:18:49.612 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:18:49.377+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58709 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:18:49.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:49.376+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:49.378413+0000) 2024-04-27T03:18:49.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:49.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:49.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:49.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:49.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:49.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:49.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:49.421+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:49.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:49.302+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:49.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:49.302+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:49.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:49.302+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:49.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:49.303+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:49.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:49.375+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:49.376974+0000) 2024-04-27T03:18:49.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:49.825+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:49.827494+0000) 2024-04-27T03:18:49.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:49.732+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:49.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:49.732+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:49.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:49.732+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:49.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:49.732+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:49.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:49.806+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:49.808164+0000) 2024-04-27T03:18:50.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:49.965+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:49.967421+0000) 2024-04-27T03:18:50.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:50.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:50.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:50.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:50.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:50.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:50.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:50.281+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:50.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:50.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:50.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:50.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:50.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:50.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:50.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:50.387+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:50.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:50.830+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:50.832518+0000) 2024-04-27T03:18:50.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:50.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:50.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:50.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:50.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:50.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:50.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:50.731+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:50.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:50.808+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:50.810305+0000) 2024-04-27T03:18:51.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:50.968+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:50.970409+0000) 2024-04-27T03:18:51.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:51.249+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:51.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:51.249+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:51.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:51.249+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:51.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:51.250+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:51.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:51.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:51.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:51.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:51.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:51.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:51.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:51.373+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:51.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:51.835+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:51.837428+0000) 2024-04-27T03:18:51.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:51.761+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:51.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:51.761+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:51.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:51.761+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:51.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:51.763+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:51.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:51.812+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:51.814497+0000) 2024-04-27T03:18:52.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:51.971+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:51.973309+0000) 2024-04-27T03:18:52.518 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:52.259+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:52.518 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:52.259+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:52.518 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:52.259+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:52.519 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:52.260+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:52.519 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:52.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:52.519 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:52.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:52.519 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:52.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:52.519 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:52.362+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:52.824 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:52.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:52.824 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:52.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:52.824 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:52.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:52.824 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:52.744+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:53.138 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:52.814+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:52.816646+0000) 2024-04-27T03:18:53.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:52.838+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:52.840319+0000) 2024-04-27T03:18:53.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:52.975+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:52.977544+0000) 2024-04-27T03:18:53.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:53.249+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:53.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:53.249+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:53.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:53.249+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:53.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:53.250+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:53.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:53.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:53.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:53.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:53.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:53.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:53.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:53.408+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:54.067 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:53.976+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:53.978098+0000) 2024-04-27T03:18:54.067 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:53.839+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:53.841180+0000) 2024-04-27T03:18:54.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:53.751+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:54.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:53.751+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:54.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:53.751+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:54.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:53.752+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:54.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:53.815+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:53.817278+0000) 2024-04-27T03:18:54.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:54.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:54.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:54.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:54.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:54.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:54.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:54.248+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:54.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:18:54.385+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58748 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:18:54.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:54.383+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:54.384920+0000) 2024-04-27T03:18:54.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:54.383+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:54.385685+0000) 2024-04-27T03:18:54.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:54.382+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:54.384609+0000) 2024-04-27T03:18:54.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:54.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:54.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:54.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:54.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:54.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:54.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:54.414+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:55.067 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:54.839+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:54.841437+0000) 2024-04-27T03:18:55.067 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:54.976+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:54.978288+0000) 2024-04-27T03:18:55.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:54.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:55.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:54.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:55.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:54.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:55.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:54.740+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:55.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:54.815+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:54.817531+0000) 2024-04-27T03:18:55.432 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:55.282+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:55.433 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:55.282+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:55.433 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:55.282+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:55.433 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:55.283+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:55.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:55.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:55.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:55.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:55.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:55.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:55.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:55.424+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:55.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:55.841+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:55.843523+0000) 2024-04-27T03:18:55.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:55.690+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:55.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:55.690+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:55.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:55.690+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:55.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:55.691+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:55.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:55.817+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:55.819578+0000) 2024-04-27T03:18:56.290 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:55.979+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:55.980509+0000) 2024-04-27T03:18:56.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:56.282+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:56.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:56.282+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:56.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:56.282+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:56.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:56.283+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:56.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:56.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:56.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:56.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:56.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:56.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:56.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:56.459+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:56.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:56.845+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:56.845829+0000) 2024-04-27T03:18:56.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:56.689+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:56.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:56.689+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:56.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:56.689+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:56.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:56.690+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:56.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:56.821+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:56.822333+0000) 2024-04-27T03:18:57.332 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:56.981+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:56.982640+0000) 2024-04-27T03:18:57.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:57.324+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:57.333 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:57.324+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:57.333 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:57.324+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:57.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:57.325+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:57.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:57.435+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:57.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:57.435+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:57.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:57.435+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:57.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:57.436+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:57.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:57.847+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:57.848057+0000) 2024-04-27T03:18:57.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:57.726+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:57.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:57.726+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:57.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:57.726+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:57.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:57.727+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:57.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:57.823+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:57.824455+0000) 2024-04-27T03:18:58.313 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:57.984+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:57.984907+0000) 2024-04-27T03:18:58.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:58.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:58.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:58.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:58.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:58.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:58.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:58.306+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:58.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:58.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:58.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:58.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:58.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:58.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:58.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:58.409+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:58.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:58.847+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:58.848755+0000) 2024-04-27T03:18:58.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:58.755+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:58.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:58.755+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:58.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:58.755+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:58.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:58.756+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:58.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:58.824+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:58.824981+0000) 2024-04-27T03:18:59.356 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:58.985+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:58.986000+0000) 2024-04-27T03:18:59.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:59.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:59.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:59.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:59.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:59.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:18:59.613 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:18:59.394+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58788 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:18:59.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:59.392+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:59.393777+0000) 2024-04-27T03:18:59.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:59.349+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:59.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:59.391+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:59.392278+0000) 2024-04-27T03:18:59.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:59.391+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:59.392147+0000) 2024-04-27T03:18:59.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:59.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:59.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:59.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:59.615 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:59.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:18:59.615 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:59.443+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:59.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:18:59.848+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:59.849036+0000) 2024-04-27T03:18:59.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:59.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:59.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:59.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:59.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:59.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:18:59.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:59.801+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:18:59.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:18:59.824+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:59.825237+0000) 2024-04-27T03:19:00.264 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:18:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:18:59.985+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:18:59.986194+0000) 2024-04-27T03:19:00.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:00.331+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:00.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:00.331+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:00.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:00.331+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:00.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:00.331+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:00.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:00.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:00.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:00.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:00.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:00.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:00.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:00.420+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:00.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:00.853+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:00.853820+0000) 2024-04-27T03:19:00.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:00.826+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:00.827333+0000) 2024-04-27T03:19:00.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:00.842+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:00.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:00.842+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:00.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:00.842+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:00.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:00.843+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:01.356 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:00.987+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:00.988512+0000) 2024-04-27T03:19:01.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:01.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:01.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:01.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:01.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:01.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:01.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:01.349+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:01.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:01.468+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:01.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:01.468+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:01.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:01.468+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:01.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:01.469+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:01.863 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:01.798+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:01.864 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:01.798+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:01.864 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:01.798+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:01.864 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:01.799+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:01.864 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:01.828+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:01.829485+0000) 2024-04-27T03:19:02.137 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:01.855+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:01.856544+0000) 2024-04-27T03:19:02.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:01.989+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:01.990749+0000) 2024-04-27T03:19:02.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:02.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:02.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:02.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:02.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:02.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:02.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:02.367+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:02.760 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:02.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:02.760 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:02.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:02.760 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:02.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:02.760 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:02.483+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:03.068 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:02.992+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:02.992993+0000) 2024-04-27T03:19:03.068 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:02.858+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:02.859024+0000) 2024-04-27T03:19:03.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:02.771+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:03.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:02.771+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:03.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:02.771+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:03.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:02.773+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:03.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:02.830+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:02.831571+0000) 2024-04-27T03:19:03.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:03.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:03.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:03.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:03.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:03.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:03.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:03.483+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:03.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:03.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:03.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:03.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:03.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:03.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:03.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:03.366+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:04.068 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:03.859+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:03.860121+0000) 2024-04-27T03:19:04.068 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:03.992+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:03.993529+0000) 2024-04-27T03:19:04.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:03.752+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:04.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:03.752+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:04.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:03.752+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:04.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:03.753+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:04.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:03.831+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:03.832620+0000) 2024-04-27T03:19:04.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:04.398+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:04.399028+0000) 2024-04-27T03:19:04.614 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:19:04.401+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58827 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:19:04.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:04.351+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:04.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:04.351+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:04.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:04.351+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:04.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:04.351+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:04.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:04.398+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:04.399381+0000) 2024-04-27T03:19:04.615 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:04.398+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:04.399742+0000) 2024-04-27T03:19:04.615 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:04.482+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:04.615 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:04.482+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:04.615 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:04.482+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:04.615 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:04.483+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:04.867 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:04.730+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:04.867 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:04.730+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:04.868 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:04.730+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:04.868 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:04.731+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:04.868 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:04.832+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:04.832822+0000) 2024-04-27T03:19:05.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:04.992+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:04.993782+0000) 2024-04-27T03:19:05.137 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:04.859+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:04.860379+0000) 2024-04-27T03:19:05.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:05.320+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:05.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:05.320+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:05.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:05.320+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:05.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:05.320+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:05.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:05.437+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:05.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:05.437+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:05.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:05.437+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:05.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:05.437+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:05.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:05.862+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:05.862916+0000) 2024-04-27T03:19:05.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:05.696+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:05.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:05.696+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:05.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:05.696+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:05.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:05.696+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:05.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:05.836+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:05.837038+0000) 2024-04-27T03:19:06.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:05.995+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:05.995827+0000) 2024-04-27T03:19:06.660 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:06.359+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:06.661 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:06.359+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:06.661 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:06.359+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:06.661 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:06.359+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:06.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:06.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:06.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:06.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:06.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:06.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:06.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:06.407+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:06.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:06.864+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:06.865498+0000) 2024-04-27T03:19:06.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:06.652+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:06.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:06.652+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:06.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:06.652+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:06.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:06.654+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:06.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:06.839+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:06.840139+0000) 2024-04-27T03:19:07.387 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:06.997+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:06.998005+0000) 2024-04-27T03:19:07.650 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:07.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:07.650 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:07.380+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:07.650 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:07.380+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:07.650 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:07.381+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:07.650 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:07.453+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:07.650 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:07.453+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:07.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:07.453+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:07.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:07.453+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:07.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:07.869+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:07.869831+0000) 2024-04-27T03:19:07.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:07.642+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:07.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:07.642+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:07.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:07.642+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:07.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:07.643+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:07.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:07.843+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:07.844389+0000) 2024-04-27T03:19:08.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:08.001+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:08.002314+0000) 2024-04-27T03:19:08.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:08.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:08.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:08.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:08.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:08.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:08.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:08.457+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:08.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:08.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:08.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:08.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:08.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:08.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:08.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:08.419+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:08.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:08.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:08.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:08.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:08.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:08.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:08.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:08.634+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:09.068 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:09.002+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:09.002849+0000) 2024-04-27T03:19:09.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:08.844+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:08.844919+0000) 2024-04-27T03:19:09.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:08.869+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:08.870358+0000) 2024-04-27T03:19:09.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:19:09.408+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58867 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:19:09.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:09.405+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:09.406492+0000) 2024-04-27T03:19:09.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:09.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:09.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:09.466+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:09.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:09.466+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:09.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:09.466+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:09.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:09.406+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:09.406818+0000) 2024-04-27T03:19:09.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:09.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:09.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:09.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:09.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:09.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:09.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:09.598+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:09.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:09.406+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:09.406927+0000) 2024-04-27T03:19:09.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:09.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:09.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:09.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:09.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:09.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:09.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:09.468+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:10.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:09.844+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:09.845100+0000) 2024-04-27T03:19:10.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:10.002+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:10.003004+0000) 2024-04-27T03:19:10.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:09.869+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:09.870540+0000) 2024-04-27T03:19:10.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:10.515+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:10.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:10.515+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:10.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:10.515+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:10.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:10.516+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:10.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:10.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:10.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:10.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:10.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:10.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:10.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:10.562+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:10.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:10.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:10.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:10.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:10.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:10.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:10.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:10.425+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:11.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:10.846+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:10.847432+0000) 2024-04-27T03:19:11.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:11.004+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:11.005180+0000) 2024-04-27T03:19:11.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:10.874+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:10.875174+0000) 2024-04-27T03:19:11.769 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:11.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:11.769 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:11.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:11.769 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:11.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:11.769 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:11.593+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:11.769 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:11.525+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:11.769 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:11.525+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:11.769 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:11.525+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:11.770 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:11.526+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:11.770 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:11.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:11.770 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:11.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:11.770 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:11.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:11.770 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:11.475+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:12.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:11.848+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:11.849543+0000) 2024-04-27T03:19:12.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:11.876+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:11.877466+0000) 2024-04-27T03:19:12.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:12.009+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:12.009927+0000) 2024-04-27T03:19:12.769 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:12.561+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:12.769 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:12.561+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:12.769 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:12.561+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:12.769 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:12.562+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:12.770 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:12.491+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:12.770 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:12.491+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:12.770 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:12.491+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:12.770 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:12.492+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:12.770 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:12.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:12.770 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:12.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:12.770 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:12.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:12.770 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:12.585+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:13.019 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:12.850+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:12.851686+0000) 2024-04-27T03:19:13.021 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:12.879+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:12.880518+0000) 2024-04-27T03:19:13.285 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:13.012+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:13.012819+0000) 2024-04-27T03:19:13.563 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:13.501+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:13.564 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:13.501+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:13.564 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:13.501+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:13.564 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:13.502+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:13.859 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:13.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:13.860 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:13.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:13.860 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:13.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:13.860 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:13.623+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:13.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:13.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:13.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:13.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:13.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:13.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:13.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:13.556+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:14.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:14.012+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:14.013522+0000) 2024-04-27T03:19:14.138 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:13.851+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:13.852775+0000) 2024-04-27T03:19:14.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:13.880+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:13.881632+0000) 2024-04-27T03:19:14.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:19:14.415+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58906 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:19:14.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:14.413+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:14.414300+0000) 2024-04-27T03:19:14.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:14.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:14.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:14.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:14.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:14.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:14.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:14.559+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:14.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:14.413+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:14.414632+0000) 2024-04-27T03:19:14.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:14.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:14.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:14.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:14.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:14.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:14.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:14.480+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:14.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:14.418+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:14.418823+0000) 2024-04-27T03:19:14.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:14.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:14.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:14.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:14.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:14.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:14.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:14.618+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:15.020 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:14.852+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:14.853034+0000) 2024-04-27T03:19:15.021 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:14.881+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:14.881841+0000) 2024-04-27T03:19:15.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:15.012+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:15.013773+0000) 2024-04-27T03:19:15.773 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:15.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:15.774 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:15.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:15.774 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:15.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:15.774 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:15.554+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:15.774 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:15.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:15.774 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:15.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:15.774 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:15.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:15.774 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:15.591+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:15.775 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:15.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:15.775 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:15.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:15.775 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:15.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:15.775 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:15.505+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:16.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:15.883+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:15.883973+0000) 2024-04-27T03:19:16.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:15.854+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:15.855095+0000) 2024-04-27T03:19:16.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:16.015+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:16.016592+0000) 2024-04-27T03:19:16.615 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:16.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:16.615 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:16.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:16.615 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:16.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:16.615 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:16.548+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:16.615 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:16.511+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:16.615 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:16.511+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:16.616 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:16.511+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:16.616 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:16.512+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:16.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:16.608+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:16.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:16.608+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:16.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:16.608+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:16.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:16.609+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:16.894 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:16.856+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:16.857216+0000) 2024-04-27T03:19:17.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:17.018+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:17.018843+0000) 2024-04-27T03:19:17.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:16.885+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:16.886223+0000) 2024-04-27T03:19:17.775 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:17.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:17.775 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:17.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:17.775 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:17.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:17.775 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:17.480+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:17.775 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:17.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:17.775 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:17.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:17.776 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:17.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:17.776 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:17.639+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:17.776 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:17.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:17.776 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:17.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:17.776 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:17.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:17.776 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:17.551+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:18.027 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:17.860+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:17.861420+0000) 2024-04-27T03:19:18.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:17.887+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:17.888571+0000) 2024-04-27T03:19:18.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:18.020+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:18.020980+0000) 2024-04-27T03:19:18.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:18.447+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:18.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:18.447+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:18.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:18.447+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:18.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:18.447+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:18.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:18.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:18.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:18.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:18.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:18.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:18.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:18.508+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:18.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:18.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:18.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:18.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:18.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:18.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:18.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:18.602+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:19.028 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:18.861+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:18.862533+0000) 2024-04-27T03:19:19.029 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:18.888+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:18.889267+0000) 2024-04-27T03:19:19.424 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:19.020+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:19.021726+0000) 2024-04-27T03:19:19.727 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:19:19.423+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58946 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:19:19.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:19.419+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:19.420461+0000) 2024-04-27T03:19:19.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:19.440+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:19.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:19.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:19.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:19.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:19.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:19.441+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:19.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:19.420+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:19.421789+0000) 2024-04-27T03:19:19.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:19.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:19.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:19.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:19.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:19.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:19.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:19.495+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:19.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:19.420+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:19.421322+0000) 2024-04-27T03:19:19.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:19.559+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:19.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:19.559+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:19.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:19.559+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:19.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:19.559+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:20.028 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:19.861+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:19.862788+0000) 2024-04-27T03:19:20.029 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:19.888+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:19.889527+0000) 2024-04-27T03:19:20.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:20.021+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:20.021899+0000) 2024-04-27T03:19:20.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:20.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:20.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:20.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:20.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:20.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:20.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:20.430+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:20.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:20.468+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:20.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:20.468+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:20.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:20.468+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:20.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:20.470+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:20.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:20.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:20.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:20.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:20.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:20.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:20.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:20.549+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:21.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:20.864+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:20.865443+0000) 2024-04-27T03:19:21.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:20.891+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:20.892482+0000) 2024-04-27T03:19:21.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:21.024+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:21.025101+0000) 2024-04-27T03:19:21.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:21.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:21.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:21.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:21.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:21.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:21.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:21.502+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:21.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:21.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:21.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:21.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:21.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:21.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:21.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:21.484+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:21.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:21.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:21.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:21.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:21.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:21.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:21.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:21.405+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:22.034 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:21.866+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:21.867559+0000) 2024-04-27T03:19:22.034 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:21.893+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:21.894582+0000) 2024-04-27T03:19:22.381 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:22.026+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:22.027554+0000) 2024-04-27T03:19:22.381 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:22.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:22.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:22.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:22.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:22.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:22.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:22.374+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:22.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:22.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:22.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:22.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:22.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:22.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:22.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:22.532+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:22.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:22.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:22.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:22.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:22.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:22.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:22.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:22.506+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:23.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:22.870+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:22.870980+0000) 2024-04-27T03:19:23.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:22.896+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:22.897365+0000) 2024-04-27T03:19:23.364 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:23.030+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:23.030916+0000) 2024-04-27T03:19:23.364 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:23.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:23.364 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:23.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:23.615 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:23.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:23.616 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:23.357+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:23.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:23.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:23.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:23.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:23.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:23.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:23.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:23.486+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:23.616 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:23.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:23.617 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:23.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:23.617 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:23.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:23.617 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:23.487+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:24.139 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:23.870+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:23.871771+0000) 2024-04-27T03:19:24.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:24.030+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:24.031710+0000) 2024-04-27T03:19:24.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:23.897+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:23.898269+0000) 2024-04-27T03:19:24.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:24.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:24.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:24.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:24.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:24.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:24.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:24.377+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:24.727 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:19:24.430+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 58985 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:19:24.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:24.428+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:24.429355+0000) 2024-04-27T03:19:24.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:24.427+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:24.428608+0000) 2024-04-27T03:19:24.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:24.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:24.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:24.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:24.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:24.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:24.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:24.526+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:24.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:24.428+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:24.429730+0000) 2024-04-27T03:19:24.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:24.518+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:24.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:24.518+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:24.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:24.518+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:24.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:24.520+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:25.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:24.871+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:24.871943+0000) 2024-04-27T03:19:25.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:24.897+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:24.898531+0000) 2024-04-27T03:19:25.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:25.031+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:25.031980+0000) 2024-04-27T03:19:25.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:25.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:25.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:25.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:25.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:25.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:25.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:25.366+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:25.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:25.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:25.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:25.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:25.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:25.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:25.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:25.524+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:25.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:25.570+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:25.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:25.570+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:25.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:25.570+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:25.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:25.571+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:26.041 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:25.873+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:25.874180+0000) 2024-04-27T03:19:26.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:25.902+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:25.902826+0000) 2024-04-27T03:19:26.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:26.034+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:26.034944+0000) 2024-04-27T03:19:26.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:26.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:26.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:26.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:26.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:26.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:26.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:26.532+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:26.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:26.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:26.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:26.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:26.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:26.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:26.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:26.495+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:26.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:26.398+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:26.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:26.398+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:26.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:26.398+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:26.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:26.399+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:27.045 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:26.875+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:26.876345+0000) 2024-04-27T03:19:27.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:26.904+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:26.905466+0000) 2024-04-27T03:19:27.358 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:27.037+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:27.037899+0000) 2024-04-27T03:19:27.616 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:27.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:27.616 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:27.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:27.616 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:27.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:27.616 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:27.351+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:27.617 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:27.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:27.617 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:27.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:27.617 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:27.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:27.617 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:27.456+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:27.617 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:27.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:27.617 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:27.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:27.617 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:27.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:27.617 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:27.522+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:28.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:27.906+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:27.907586+0000) 2024-04-27T03:19:28.140 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:27.878+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:27.879273+0000) 2024-04-27T03:19:28.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:28.041+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:28.042152+0000) 2024-04-27T03:19:28.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:28.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:28.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:28.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:28.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:28.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:28.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:28.369+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:28.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:28.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:28.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:28.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:28.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:28.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:28.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:28.464+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:28.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:28.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:28.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:28.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:28.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:28.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:28.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:28.511+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:29.049 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:28.879+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:28.879827+0000) 2024-04-27T03:19:29.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:28.908+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:28.908801+0000) 2024-04-27T03:19:29.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:29.041+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:29.042783+0000) 2024-04-27T03:19:29.727 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:19:29.437+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59025 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:19:29.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:29.416+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:29.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:29.416+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:29.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:29.416+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:29.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:29.416+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:29.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:29.435+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:29.436787+0000) 2024-04-27T03:19:29.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:29.436+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:29.436887+0000) 2024-04-27T03:19:29.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:29.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:29.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:29.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:29.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:29.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:29.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:29.507+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:29.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:29.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:29.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:29.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:29.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:29.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:29.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:29.389+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:29.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:29.435+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:29.436679+0000) 2024-04-27T03:19:30.049 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:29.879+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:29.880042+0000) 2024-04-27T03:19:30.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:29.908+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:29.909048+0000) 2024-04-27T03:19:30.384 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:30.042+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:30.042975+0000) 2024-04-27T03:19:30.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:30.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:30.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:30.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:30.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:30.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:30.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:30.491+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:30.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:30.401+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:30.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:30.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:30.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:30.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:30.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:30.402+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:30.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:30.376+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:30.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:30.376+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:30.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:30.376+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:30.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:30.378+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:31.054 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:30.882+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:30.882879+0000) 2024-04-27T03:19:31.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:30.912+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:30.913325+0000) 2024-04-27T03:19:31.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:31.046+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:31.047177+0000) 2024-04-27T03:19:31.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:31.412+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:31.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:31.412+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:31.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:31.412+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:31.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:31.413+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:31.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:31.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:31.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:31.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:31.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:31.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:31.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:31.527+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:31.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:31.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:31.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:31.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:31.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:31.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:31.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:31.385+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:31.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:31.884+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:31.884932+0000) 2024-04-27T03:19:31.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:31.914+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:31.915595+0000) 2024-04-27T03:19:32.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:32.050+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:32.051456+0000) 2024-04-27T03:19:32.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:32.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:32.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:32.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:32.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:32.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:32.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:32.387+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:32.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:32.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:32.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:32.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:32.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:32.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:32.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:32.552+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:32.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:32.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:32.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:32.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:32.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:32.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:32.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:32.419+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:33.060 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:32.887+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:32.888618+0000) 2024-04-27T03:19:33.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:32.917+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:32.917849+0000) 2024-04-27T03:19:33.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:33.052+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:33.053729+0000) 2024-04-27T03:19:33.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:33.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:33.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:33.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:33.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:33.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:33.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:33.383+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:33.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:33.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:33.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:33.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:33.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:33.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:33.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:33.583+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:33.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:33.403+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:33.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:33.403+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:33.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:33.403+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:33.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:33.403+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:34.061 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:33.888+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:33.889285+0000) 2024-04-27T03:19:34.062 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:33.917+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:33.918575+0000) 2024-04-27T03:19:34.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:34.054+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:34.054880+0000) 2024-04-27T03:19:34.611 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:19:34.444+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59064 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:19:34.612 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:34.444+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:34.444844+0000) 2024-04-27T03:19:34.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:34.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:34.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:34.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:34.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:34.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:34.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:34.350+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:34.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:34.442+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:34.442868+0000) 2024-04-27T03:19:34.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:34.443+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:34.444005+0000) 2024-04-27T03:19:34.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:34.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:34.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:34.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:34.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:34.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:34.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:34.446+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:34.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:34.603+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:34.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:34.603+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:34.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:34.603+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:34.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:34.605+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:35.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:34.888+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:34.889449+0000) 2024-04-27T03:19:35.184 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:34.917+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:34.918764+0000) 2024-04-27T03:19:35.184 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:35.054+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:35.055056+0000) 2024-04-27T03:19:35.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:35.361+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:35.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:35.361+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:35.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:35.361+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:35.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:35.362+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:35.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:35.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:35.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:35.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:35.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:35.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:35.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:35.443+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:35.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:35.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:35.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:35.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:35.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:35.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:35.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:35.623+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:36.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:35.893+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:35.894335+0000) 2024-04-27T03:19:36.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:35.920+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:35.921412+0000) 2024-04-27T03:19:36.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:36.056+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:36.057210+0000) 2024-04-27T03:19:36.617 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:36.355+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:36.617 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:36.355+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:36.617 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:36.355+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:36.617 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:36.355+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:36.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:36.458+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:36.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:36.458+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:36.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:36.458+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:36.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:36.459+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:36.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:36.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:36.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:36.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:36.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:36.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:36.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:36.631+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:37.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:36.896+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:36.897290+0000) 2024-04-27T03:19:37.184 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:37.060+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:37.061322+0000) 2024-04-27T03:19:37.184 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:36.922+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:36.923518+0000) 2024-04-27T03:19:37.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:37.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:37.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:37.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:37.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:37.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:37.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:37.377+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:37.794 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:37.482+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:37.794 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:37.482+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:37.794 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:37.482+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:37.794 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:37.483+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:37.794 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:37.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:37.794 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:37.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:37.794 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:37.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:37.794 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:37.610+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:38.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:37.898+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:37.899649+0000) 2024-04-27T03:19:38.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:37.927+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:37.928322+0000) 2024-04-27T03:19:38.430 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:38.065+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:38.065915+0000) 2024-04-27T03:19:38.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:38.605+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:38.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:38.605+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:38.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:38.605+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:38.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:38.605+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:38.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:38.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:38.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:38.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:38.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:38.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:38.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:38.480+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:38.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:38.423+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:38.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:38.423+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:38.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:38.423+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:38.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:38.424+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:39.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:38.899+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:38.900431+0000) 2024-04-27T03:19:39.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:38.928+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:38.929540+0000) 2024-04-27T03:19:39.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:39.065+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:39.066505+0000) 2024-04-27T03:19:39.657 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:19:39.452+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59104 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:19:39.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:39.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:39.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:39.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:39.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:39.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:39.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:39.387+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:39.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:39.450+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:39.451166+0000) 2024-04-27T03:19:39.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:39.449+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:39.449820+0000) 2024-04-27T03:19:39.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:39.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:39.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:39.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:39.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:39.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:39.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:39.509+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:39.659 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:39.449+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:39.450258+0000) 2024-04-27T03:19:39.659 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:39.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:39.659 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:39.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:39.659 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:39.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:39.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:39.899+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:39.900615+0000) 2024-04-27T03:19:39.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:39.650+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:40.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:39.928+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:39.929795+0000) 2024-04-27T03:19:40.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:40.065+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:40.066705+0000) 2024-04-27T03:19:40.554 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:40.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:40.554 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:40.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:40.554 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:40.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:40.554 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:40.374+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:40.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:40.641+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:40.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:40.641+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:40.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:40.641+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:40.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:40.642+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:40.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:40.546+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:40.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:40.546+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:40.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:40.546+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:40.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:40.546+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:41.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:40.902+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:40.903345+0000) 2024-04-27T03:19:41.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:40.933+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:40.934118+0000) 2024-04-27T03:19:41.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:41.070+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:41.070888+0000) 2024-04-27T03:19:41.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:41.355+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:41.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:41.355+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:41.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:41.355+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:41.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:41.356+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:41.799 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:41.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:41.799 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:41.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:41.799 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:41.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:41.799 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:41.649+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:41.800 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:41.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:41.800 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:41.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:41.800 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:41.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:41.800 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:41.500+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:42.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:41.936+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:41.937121+0000) 2024-04-27T03:19:42.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:41.905+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:41.906233+0000) 2024-04-27T03:19:42.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:42.072+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:42.073477+0000) 2024-04-27T03:19:42.617 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:42.314+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:42.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:42.314+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:42.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:42.314+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:42.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:42.316+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:42.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:42.550+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:42.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:42.550+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:42.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:42.550+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:42.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:42.550+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:42.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:42.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:42.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:42.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:42.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:42.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:42.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:42.626+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:43.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:42.908+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:42.909167+0000) 2024-04-27T03:19:43.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:42.939+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:42.939847+0000) 2024-04-27T03:19:43.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:43.076+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:43.077756+0000) 2024-04-27T03:19:43.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:43.293+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:43.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:43.293+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:43.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:43.293+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:43.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:43.294+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:43.799 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:43.514+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:43.799 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:43.514+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:43.800 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:43.514+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:43.800 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:43.515+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:43.800 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:43.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:43.800 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:43.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:43.800 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:43.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:43.800 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:43.641+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:44.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:43.909+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:43.910194+0000) 2024-04-27T03:19:44.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:43.940+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:43.940892+0000) 2024-04-27T03:19:44.461 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:44.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:44.461 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:44.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:44.461 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:44.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:44.461 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:44.280+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:44.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:44.077+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:44.078401+0000) 2024-04-27T03:19:44.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:19:44.460+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59143 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:19:44.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:44.459+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:44.460705+0000) 2024-04-27T03:19:44.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:44.456+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:44.457657+0000) 2024-04-27T03:19:44.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:44.545+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:44.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:44.545+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:44.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:44.545+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:44.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:44.545+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:44.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:44.456+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:44.457468+0000) 2024-04-27T03:19:44.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:44.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:44.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:44.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:44.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:44.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:44.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:44.689+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:45.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:44.909+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:44.910402+0000) 2024-04-27T03:19:45.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:44.940+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:44.941090+0000) 2024-04-27T03:19:45.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:45.077+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:45.078649+0000) 2024-04-27T03:19:45.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:45.241+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:45.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:45.241+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:45.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:45.241+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:45.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:45.242+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:45.803 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:45.533+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:45.803 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:45.533+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:45.803 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:45.533+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:45.803 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:45.533+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:45.803 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:45.681+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:45.803 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:45.681+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:45.803 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:45.681+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:45.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:45.682+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:46.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:45.912+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:45.913217+0000) 2024-04-27T03:19:46.073 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:45.941+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:45.943330+0000) 2024-04-27T03:19:46.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:46.079+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:46.080822+0000) 2024-04-27T03:19:46.376 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:46.225+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:46.376 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:46.225+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:46.376 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:46.225+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:46.376 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:46.226+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:46.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:46.577+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:46.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:46.577+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:46.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:46.577+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:46.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:46.577+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:46.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:46.913+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:46.915491+0000) 2024-04-27T03:19:46.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:46.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:46.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:46.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:46.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:46.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:46.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:46.706+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:46.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:46.943+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:46.945452+0000) 2024-04-27T03:19:47.251 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:47.081+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:47.083495+0000) 2024-04-27T03:19:47.539 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:47.243+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:47.539 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:47.243+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:47.539 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:47.243+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:47.539 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:47.244+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:47.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:47.530+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:47.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:47.530+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:47.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:47.530+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:47.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:47.531+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:47.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:47.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:47.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:47.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:47.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:47.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:47.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:47.750+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:48.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:47.915+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:47.917743+0000) 2024-04-27T03:19:48.073 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:47.945+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:47.947648+0000) 2024-04-27T03:19:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:48.083+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:48.085777+0000) 2024-04-27T03:19:48.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:48.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:48.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:48.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:48.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:48.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:48.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:48.254+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:48.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:48.535+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:48.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:48.535+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:48.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:48.535+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:48.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:48.536+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:48.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:48.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:48.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:48.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:48.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:48.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:48.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:48.707+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:49.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:48.916+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:48.918329+0000) 2024-04-27T03:19:49.073 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:48.946+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:48.948737+0000) 2024-04-27T03:19:49.469 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:49.297+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:49.469 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:49.297+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:49.469 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:49.297+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:49.469 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:49.298+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:49.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:49.084+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:49.086432+0000) 2024-04-27T03:19:49.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:49.464+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:49.466355+0000) 2024-04-27T03:19:49.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:19:49.467+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59183 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:19:49.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:49.464+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:49.466039+0000) 2024-04-27T03:19:49.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:49.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:49.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:49.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:49.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:49.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:49.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:49.677+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:49.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:49.466+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:49.467910+0000) 2024-04-27T03:19:49.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:49.582+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:49.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:49.582+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:49.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:49.582+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:49.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:49.582+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:50.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:49.916+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:49.918526+0000) 2024-04-27T03:19:50.073 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:49.947+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:49.948911+0000) 2024-04-27T03:19:50.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:50.084+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:50.086616+0000) 2024-04-27T03:19:50.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:50.271+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:50.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:50.271+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:50.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:50.271+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:50.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:50.271+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:50.927 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:50.587+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:50.927 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:50.587+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:50.927 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:50.587+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:50.927 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:50.587+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:50.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:50.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:50.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:50.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:50.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:50.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:50.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:50.720+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:51.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:50.918+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:50.920691+0000) 2024-04-27T03:19:51.186 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:50.951+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:50.953202+0000) 2024-04-27T03:19:51.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:51.089+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:51.090845+0000) 2024-04-27T03:19:51.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:51.255+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:51.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:51.255+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:51.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:51.255+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:51.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:51.256+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:51.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:51.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:51.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:51.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:51.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:51.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:51.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:51.675+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:51.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:51.577+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:51.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:51.577+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:51.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:51.577+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:51.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:51.577+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:52.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:51.953+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:51.955479+0000) 2024-04-27T03:19:52.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:52.093+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:52.095054+0000) 2024-04-27T03:19:52.186 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:51.922+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:51.923977+0000) 2024-04-27T03:19:52.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:52.226+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:52.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:52.226+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:52.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:52.226+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:52.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:52.227+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:52.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:52.618+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:52.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:52.618+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:52.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:52.618+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:52.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:52.619+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:52.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:52.704+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:52.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:52.704+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:52.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:52.704+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:52.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:52.705+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:53.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:53.098+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:53.099918+0000) 2024-04-27T03:19:53.186 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:52.924+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:52.926626+0000) 2024-04-27T03:19:53.186 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:52.955+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:52.957515+0000) 2024-04-27T03:19:53.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:53.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:53.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:53.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:53.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:53.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:53.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:53.212+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:53.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:53.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:53.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:53.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:53.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:53.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:53.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:53.748+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:53.935 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:53.640+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:53.935 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:53.640+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:53.935 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:53.640+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:53.935 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:53.641+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:54.186 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:53.925+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:53.927696+0000) 2024-04-27T03:19:54.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:54.099+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:54.101056+0000) 2024-04-27T03:19:54.187 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:53.956+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:53.958131+0000) 2024-04-27T03:19:54.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:54.206+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:54.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:54.206+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:54.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:54.206+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:54.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:54.207+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:54.727 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:19:54.474+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59222 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:19:54.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:54.473+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:54.474914+0000) 2024-04-27T03:19:54.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:54.472+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:54.474065+0000) 2024-04-27T03:19:54.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:54.471+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:54.473466+0000) 2024-04-27T03:19:54.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:54.619+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:54.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:54.619+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:54.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:54.619+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:54.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:54.620+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:55.073 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:54.926+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:54.927932+0000) 2024-04-27T03:19:55.073 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:54.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:55.074 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:54.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:55.074 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:54.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:55.074 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:54.774+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:55.074 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:54.956+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:54.958317+0000) 2024-04-27T03:19:55.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:55.099+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:55.101330+0000) 2024-04-27T03:19:55.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:55.251+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:55.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:55.251+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:55.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:55.251+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:55.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:55.252+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:55.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:55.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:55.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:55.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:55.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:55.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:55.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:55.787+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:55.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:55.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:55.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:55.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:55.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:55.576+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:55.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:55.576+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:56.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:55.928+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:55.930563+0000) 2024-04-27T03:19:56.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:55.958+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:55.960556+0000) 2024-04-27T03:19:56.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:56.103+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:56.105314+0000) 2024-04-27T03:19:56.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:56.257+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:56.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:56.257+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:56.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:56.257+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:56.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:56.258+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:56.940 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:56.621+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:56.940 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:56.621+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:56.940 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:56.621+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:56.940 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:56.621+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:56.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:56.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:56.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:56.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:56.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:56.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:56.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:56.759+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:57.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:56.931+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:56.933047+0000) 2024-04-27T03:19:57.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:56.963+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:56.964840+0000) 2024-04-27T03:19:57.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:57.106+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:57.107506+0000) 2024-04-27T03:19:57.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:57.263+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:57.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:57.263+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:57.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:57.263+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:57.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:57.265+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:57.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:57.780+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:57.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:57.780+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:57.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:57.780+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:57.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:57.780+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:57.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:57.605+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:57.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:57.605+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:57.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:57.605+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:57.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:57.605+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:58.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:57.937+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:57.938027+0000) 2024-04-27T03:19:58.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:58.108+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:58.109740+0000) 2024-04-27T03:19:58.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:57.967+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:57.968021+0000) 2024-04-27T03:19:58.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:58.290+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:58.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:58.290+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:58.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:58.290+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:58.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:58.291+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:58.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:58.642+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:58.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:58.642+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:58.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:58.642+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:58.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:58.642+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:58.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:58.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:58.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:58.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:58.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:58.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:58.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:58.761+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:59.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:58.967+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:58.968634+0000) 2024-04-27T03:19:59.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:58.938+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:58.939160+0000) 2024-04-27T03:19:59.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:59.109+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:59.110567+0000) 2024-04-27T03:19:59.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:59.480+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:59.481113+0000) 2024-04-27T03:19:59.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:59.481+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:59.482133+0000) 2024-04-27T03:19:59.614 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:19:59.483+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59262 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:19:59.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:59.331+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:59.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:59.331+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:59.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:59.331+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:19:59.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:59.333+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:59.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:59.481+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:59.482057+0000) 2024-04-27T03:19:59.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:59.614+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:59.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:59.615+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:59.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:59.615+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:19:59.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:19:59.615+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:19:59.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:59.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:59.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:59.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:59.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:59.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:19:59.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:59.786+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:00.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:19:59.938+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:59.939363+0000) 2024-04-27T03:20:00.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:00.109+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:00.110750+0000) 2024-04-27T03:20:00.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:19:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:19:59.968+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:19:59.968825+0000) 2024-04-27T03:20:00.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:00.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:00.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:00.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:00.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:00.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:00.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:00.379+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:00.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:00.657+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:00.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:00.657+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:00.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:00.657+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:00.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:00.658+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:00.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:00.742+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:00.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:00.742+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:00.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:00.742+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:00.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:00.743+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:01.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:00.943+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:00.944260+0000) 2024-04-27T03:20:01.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:01.113+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:01.113965+0000) 2024-04-27T03:20:01.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:00.970+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:00.970975+0000) 2024-04-27T03:20:01.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:01.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:01.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:01.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:01.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:01.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:01.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:01.389+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:01.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:01.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:01.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:01.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:01.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:01.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:01.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:01.760+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:01.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:01.667+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:01.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:01.667+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:01.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:01.667+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:01.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:01.668+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:02.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:02.115+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:02.116632+0000) 2024-04-27T03:20:02.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:01.974+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:01.975258+0000) 2024-04-27T03:20:02.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:01.948+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:01.949657+0000) 2024-04-27T03:20:02.652 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:02.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:02.652 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:02.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:02.652 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:02.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:02.652 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:02.389+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:02.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:02.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:02.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:02.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:02.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:02.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:02.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:02.712+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:02.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:02.644+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:02.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:02.644+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:02.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:02.644+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:02.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:02.645+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:03.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:03.119+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:03.119934+0000) 2024-04-27T03:20:03.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:02.951+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:02.952570+0000) 2024-04-27T03:20:03.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:02.979+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:02.980417+0000) 2024-04-27T03:20:03.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:03.390+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:03.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:03.390+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:03.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:03.390+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:03.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:03.391+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:03.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:03.685+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:03.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:03.685+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:03.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:03.685+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:03.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:03.686+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:03.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:03.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:03.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:03.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:03.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:03.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:03.961 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:03.672+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:04.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:03.952+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:03.953374+0000) 2024-04-27T03:20:04.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:03.980+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:03.981542+0000) 2024-04-27T03:20:04.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:04.119+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:04.120791+0000) 2024-04-27T03:20:04.688 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:20:04.491+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59301 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:20:04.688 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:04.488+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:04.488966+0000) 2024-04-27T03:20:04.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:04.489+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:04.490067+0000) 2024-04-27T03:20:04.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:04.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:04.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:04.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:04.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:04.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:04.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:04.405+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:04.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:04.489+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:04.490083+0000) 2024-04-27T03:20:04.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:04.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:04.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:04.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:04.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:04.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:04.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:04.718+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:04.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:04.680+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:04.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:04.680+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:04.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:04.680+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:04.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:04.681+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:05.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:04.981+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:04.981818+0000) 2024-04-27T03:20:05.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:05.120+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:05.120999+0000) 2024-04-27T03:20:05.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:04.952+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:04.953647+0000) 2024-04-27T03:20:05.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:05.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:05.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:05.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:05.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:05.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:05.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:05.389+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:05.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:05.955+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:05.956085+0000) 2024-04-27T03:20:05.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:05.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:05.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:05.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:05.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:05.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:05.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:05.730+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:05.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:05.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:05.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:05.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:05.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:05.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:05.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:05.718+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:06.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:06.122+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:06.123200+0000) 2024-04-27T03:20:06.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:05.985+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:05.986171+0000) 2024-04-27T03:20:06.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:06.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:06.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:06.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:06.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:06.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:06.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:06.418+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:07.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:06.957+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:06.958208+0000) 2024-04-27T03:20:07.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:06.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:07.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:06.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:07.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:06.760+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:07.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:06.760+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:07.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:06.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:07.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:06.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:07.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:06.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:07.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:06.739+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:07.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:06.987+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:06.988226+0000) 2024-04-27T03:20:07.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:07.124+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:07.125345+0000) 2024-04-27T03:20:07.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:07.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:07.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:07.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:07.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:07.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:07.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:07.416+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:08.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:07.959+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:07.960367+0000) 2024-04-27T03:20:08.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:07.781+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:08.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:07.781+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:08.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:07.781+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:08.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:07.781+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:08.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:07.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:08.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:07.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:08.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:07.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:08.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:07.786+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:08.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:07.989+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:07.990391+0000) 2024-04-27T03:20:08.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:08.127+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:08.128296+0000) 2024-04-27T03:20:08.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:08.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:08.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:08.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:08.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:08.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:08.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:08.384+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:09.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:08.960+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:08.961024+0000) 2024-04-27T03:20:09.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:08.802+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:09.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:08.802+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:09.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:08.802+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:09.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:08.802+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:09.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:08.808+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:09.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:08.808+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:09.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:08.808+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:09.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:08.808+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:09.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:08.990+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:08.991036+0000) 2024-04-27T03:20:09.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:09.128+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:09.129490+0000) 2024-04-27T03:20:09.621 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:20:09.498+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59341 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:20:09.621 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:09.495+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:09.496529+0000) 2024-04-27T03:20:09.621 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:09.496+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:09.497290+0000) 2024-04-27T03:20:09.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:09.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:09.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:09.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:09.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:09.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:09.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:09.347+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:09.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:09.497+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:09.497930+0000) 2024-04-27T03:20:09.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:09.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:09.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:09.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:09.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:09.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:09.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:09.795+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:09.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:09.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:09.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:09.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:09.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:09.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:09.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:09.791+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:10.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:09.960+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:09.961215+0000) 2024-04-27T03:20:10.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:09.990+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:09.991218+0000) 2024-04-27T03:20:10.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:10.128+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:10.129701+0000) 2024-04-27T03:20:10.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:10.392+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:10.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:10.392+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:10.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:10.392+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:10.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:10.394+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:10.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:10.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:10.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:10.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:10.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:10.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:10.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:10.761+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:10.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:10.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:10.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:10.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:10.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:10.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:10.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:10.756+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:11.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:10.964+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:10.965347+0000) 2024-04-27T03:20:11.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:10.993+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:10.994211+0000) 2024-04-27T03:20:11.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:11.133+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:11.133920+0000) 2024-04-27T03:20:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:11.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:11.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:11.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:11.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:11.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:11.443+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:12.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:11.966+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:11.967685+0000) 2024-04-27T03:20:12.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:11.810+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:12.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:11.810+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:12.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:11.810+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:12.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:11.810+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:12.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:11.995+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:11.996267+0000) 2024-04-27T03:20:12.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:11.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:12.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:11.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:12.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:11.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:12.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:11.785+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:12.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:12.135+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:12.136577+0000) 2024-04-27T03:20:12.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:12.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:12.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:12.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:12.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:12.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:12.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:12.396+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:13.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:12.969+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:12.970485+0000) 2024-04-27T03:20:13.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:12.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:13.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:12.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:13.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:12.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:13.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:12.757+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:13.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:12.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:13.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:12.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:13.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:12.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:13.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:12.818+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:13.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:12.997+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:12.998553+0000) 2024-04-27T03:20:13.379 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:13.138+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:13.139488+0000) 2024-04-27T03:20:13.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:13.371+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:13.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:13.371+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:13.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:13.371+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:13.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:13.373+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:14.005 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:13.970+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:13.971298+0000) 2024-04-27T03:20:14.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:13.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:14.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:13.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:14.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:13.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:14.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:13.742+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:14.006 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:13.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:14.006 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:13.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:14.006 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:13.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:14.006 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:13.778+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:14.337 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:13.998+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:13.999234+0000) 2024-04-27T03:20:14.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:14.139+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:14.140183+0000) 2024-04-27T03:20:14.621 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:20:14.505+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59380 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:20:14.621 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:14.508+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:14.509017+0000) 2024-04-27T03:20:14.622 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:14.503+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:14.503902+0000) 2024-04-27T03:20:14.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:14.330+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:14.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:14.330+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:14.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:14.330+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:14.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:14.331+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:14.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:14.502+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:14.503651+0000) 2024-04-27T03:20:14.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:14.788+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:14.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:14.788+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:14.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:14.788+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:14.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:14.788+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:14.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:14.783+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:14.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:14.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:14.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:14.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:14.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:14.784+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:15.367 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:14.998+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:14.999450+0000) 2024-04-27T03:20:15.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:15.139+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:15.140363+0000) 2024-04-27T03:20:15.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:14.970+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:14.971476+0000) 2024-04-27T03:20:15.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:15.360+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:15.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:15.360+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:15.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:15.360+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:15.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:15.362+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:15.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:15.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:15.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:15.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:15.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:15.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:15.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:15.799+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:15.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:15.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:15.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:15.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:15.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:15.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:15.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:15.829+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:16.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:15.973+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:15.974466+0000) 2024-04-27T03:20:16.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:16.142+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:16.143254+0000) 2024-04-27T03:20:16.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:16.000+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:16.001666+0000) 2024-04-27T03:20:16.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:16.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:16.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:16.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:16.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:16.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:16.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:16.398+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:17.144 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:16.975+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:16.976594+0000) 2024-04-27T03:20:17.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:16.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:17.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:16.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:17.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:16.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:46.082228+0000 (oldest deadline 2024-04-27T03:10:06.082228+0000) 2024-04-27T03:20:17.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:16.851+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:17.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:17.004+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:17.004893+0000) 2024-04-27T03:20:17.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:16.839+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:17.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:16.839+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:17.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:16.839+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:17.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:16.839+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:17.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:17.144+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:17.145370+0000) 2024-04-27T03:20:17.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:17.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:17.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:17.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:17.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:17.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:17.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:17.419+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:18.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:17.978+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:17.979385+0000) 2024-04-27T03:20:18.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:17.812+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:18.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:18.006+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:18.007002+0000) 2024-04-27T03:20:18.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:17.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:18.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:17.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:18.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:17.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:18.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:17.870+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:18.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:18.146+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:18.147487+0000) 2024-04-27T03:20:18.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:18.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:18.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:18.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:18.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:18.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:18.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:18.386+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:19.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:18.979+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:18.980060+0000) 2024-04-27T03:20:19.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:18.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:19.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:18.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:19.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:18.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:45.012023+0000 (oldest deadline 2024-04-27T03:10:05.012023+0000) 2024-04-27T03:20:19.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:18.897+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:19.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:18.858+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:19.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:19.006+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:19.007675+0000) 2024-04-27T03:20:19.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:19.147+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:19.148675+0000) 2024-04-27T03:20:19.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:19.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:19.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:19.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:19.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:19.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:19.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:19.342+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:19.833 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:20:19.512+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59420 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:20:19.833 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:19.509+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:19.510450+0000) 2024-04-27T03:20:19.833 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:19.510+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:19.511634+0000) 2024-04-27T03:20:19.834 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:19.510+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:19.511159+0000) 2024-04-27T03:20:20.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:19.979+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:19.980260+0000) 2024-04-27T03:20:20.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:19.943+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:20.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:19.898+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:20.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:20.007+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:20.007882+0000) 2024-04-27T03:20:20.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:20.148+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:20.148921+0000) 2024-04-27T03:20:20.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:20.341+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:20.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:20.341+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:20.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:20.341+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:20.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:20.342+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:21.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:20.981+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:20.982354+0000) 2024-04-27T03:20:21.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:20.981+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:21.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:21.150+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:21.151248+0000) 2024-04-27T03:20:21.189 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:20.916+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:21.189 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:21.009+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:21.010224+0000) 2024-04-27T03:20:21.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:21.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:21.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:21.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:21.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:21.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:21.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:21.306+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:22.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:21.984+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:21.985261+0000) 2024-04-27T03:20:22.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:21.942+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:22.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:22.011+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:22.012708+0000) 2024-04-27T03:20:22.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:22.021+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:22.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:22.154+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:22.155761+0000) 2024-04-27T03:20:22.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:22.331+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:22.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:22.331+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:22.623 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:22.331+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:22.623 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:22.332+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:23.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:22.987+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:22.988096+0000) 2024-04-27T03:20:23.189 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:22.917+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:23.189 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:23.014+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:23.015396+0000) 2024-04-27T03:20:23.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:22.998+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:23.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:23.158+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:23.158839+0000) 2024-04-27T03:20:23.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:23.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:23.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:23.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:23.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:23.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:23.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:23.368+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:24.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:23.988+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:23.989277+0000) 2024-04-27T03:20:24.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:23.949+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:24.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:24.015+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:24.016078+0000) 2024-04-27T03:20:24.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:23.953+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:24.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:24.158+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:24.159593+0000) 2024-04-27T03:20:24.622 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:20:24.520+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59459 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:20:24.623 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:24.517+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:24.517967+0000) 2024-04-27T03:20:24.623 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:24.518+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:24.518870+0000) 2024-04-27T03:20:24.623 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:24.319+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:24.623 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:24.319+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:24.623 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:24.319+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:09:53.475225+0000 (oldest deadline 2024-04-27T03:10:13.475225+0000) 2024-04-27T03:20:24.623 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:24.320+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:24.623 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:24.518+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:24.519290+0000) 2024-04-27T03:20:25.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:24.988+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:24.989492+0000) 2024-04-27T03:20:25.166 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:24.906+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:25.167 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:25.015+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:25.016280+0000) 2024-04-27T03:20:25.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:24.916+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:25.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:25.344+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:25.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:25.159+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:25.159885+0000) 2024-04-27T03:20:26.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:25.992+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:25.993740+0000) 2024-04-27T03:20:26.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:25.940+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:26.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:25.868+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:26.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:26.018+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:26.019243+0000) 2024-04-27T03:20:26.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:26.349+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:26.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:26.161+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:26.162227+0000) 2024-04-27T03:20:26.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:26.837+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:27.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:26.995+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:26.996440+0000) 2024-04-27T03:20:27.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:27.021+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:27.021886+0000) 2024-04-27T03:20:27.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:26.895+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:27.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:27.366+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:27.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:27.163+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:27.164589+0000) 2024-04-27T03:20:27.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:27.838+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:28.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:27.998+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:27.999384+0000) 2024-04-27T03:20:28.173 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:28.023+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:28.024601+0000) 2024-04-27T03:20:28.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:27.911+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:28.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:28.166+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:28.166998+0000) 2024-04-27T03:20:28.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:28.348+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:28.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:28.810+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:29.146 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:28.999+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:29.000296+0000) 2024-04-27T03:20:29.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:29.024+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:29.025307+0000) 2024-04-27T03:20:29.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:28.881+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:29.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:29.167+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:29.168224+0000) 2024-04-27T03:20:29.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:29.363+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:29.842 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:20:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:20:29.526+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59499 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:20:29.842 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:29.525+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:29.526559+0000) 2024-04-27T03:20:29.843 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:29.525+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:29.525872+0000) 2024-04-27T03:20:29.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:29.524+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:29.525256+0000) 2024-04-27T03:20:29.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:29.768+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:30.146 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:29.999+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:30.000461+0000) 2024-04-27T03:20:30.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:29.842+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:30.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:30.024+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:30.025501+0000) 2024-04-27T03:20:30.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:30.373+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:30.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:30.167+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:30.168384+0000) 2024-04-27T03:20:30.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:30.800+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:30.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:30.795+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:31.422 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:31.027+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:31.028007+0000) 2024-04-27T03:20:31.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:31.004+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:31.005358+0000) 2024-04-27T03:20:31.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:31.170+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:31.171183+0000) 2024-04-27T03:20:31.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:31.415+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:32.014 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:31.814+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:32.014 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:31.748+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:32.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:32.006+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:32.007475+0000) 2024-04-27T03:20:32.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:32.029+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:32.030641+0000) 2024-04-27T03:20:32.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:32.174+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:32.175260+0000) 2024-04-27T03:20:32.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:32.452+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:32.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:32.789+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:32.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:32.718+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:33.406 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:33.009+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:33.010456+0000) 2024-04-27T03:20:33.406 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:33.032+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:33.033777+0000) 2024-04-27T03:20:33.406 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:33.177+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:33.178213+0000) 2024-04-27T03:20:33.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:33.489+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:33.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:33.758+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:33.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:33.703+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:34.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:34.178+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:34.178929+0000) 2024-04-27T03:20:34.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:34.010+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:34.011756+0000) 2024-04-27T03:20:34.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:34.033+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:34.034462+0000) 2024-04-27T03:20:34.799 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:34.531+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:34.532564+0000) 2024-04-27T03:20:34.799 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:20:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:20:34.534+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59538 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:20:34.799 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:34.531+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:34.532442+0000) 2024-04-27T03:20:34.799 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:34.675+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:34.800 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:34.531+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:34.532456+0000) 2024-04-27T03:20:34.800 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:34.540+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:35.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:34.791+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:35.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:35.011+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:35.012074+0000) 2024-04-27T03:20:35.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:35.033+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:35.034729+0000) 2024-04-27T03:20:35.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:35.178+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:35.179119+0000) 2024-04-27T03:20:35.846 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:35.567+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:35.847 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:35.682+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:36.146 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:36.013+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:36.014305+0000) 2024-04-27T03:20:36.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:35.839+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:36.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:36.038+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:36.039025+0000) 2024-04-27T03:20:36.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:36.183+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:36.184193+0000) 2024-04-27T03:20:36.851 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:36.564+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:36.851 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:36.648+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:37.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:37.015+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:37.016382+0000) 2024-04-27T03:20:37.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:37.040+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:37.041150+0000) 2024-04-27T03:20:37.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:36.847+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:37.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:37.185+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:37.186427+0000) 2024-04-27T03:20:37.851 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:37.584+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:37.851 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:37.612+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:37.851 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:37.831+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:38.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:38.019+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:38.020603+0000) 2024-04-27T03:20:38.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:38.042+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:38.043304+0000) 2024-04-27T03:20:38.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:38.188+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:38.189321+0000) 2024-04-27T03:20:38.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:38.598+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:38.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:38.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:38.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:38.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:38.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:38.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:38.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:38.581+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:39.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:39.020+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:39.021416+0000) 2024-04-27T03:20:39.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:39.043+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:39.044101+0000) 2024-04-27T03:20:39.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:38.827+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:39.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:39.189+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:39.190165+0000) 2024-04-27T03:20:39.848 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:20:39.541+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59578 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:20:39.848 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:39.539+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:39.540520+0000) 2024-04-27T03:20:39.848 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:39.584+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:39.848 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:39.539+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:39.540259+0000) 2024-04-27T03:20:39.849 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:39.840+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:39.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:39.539+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:39.540419+0000) 2024-04-27T03:20:39.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:39.616+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:39.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:39.616+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:39.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:39.616+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:39.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:39.616+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:40.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:40.020+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:40.021602+0000) 2024-04-27T03:20:40.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:40.043+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:40.044316+0000) 2024-04-27T03:20:40.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:40.189+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:40.190377+0000) 2024-04-27T03:20:40.840 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:40.556+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:40.841 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:40.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:40.841 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:40.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:40.841 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:40.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:40.841 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:40.648+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:41.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:41.022+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:41.023713+0000) 2024-04-27T03:20:41.148 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:40.833+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:41.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:41.045+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:41.046444+0000) 2024-04-27T03:20:41.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:41.191+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:41.192612+0000) 2024-04-27T03:20:41.834 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:41.517+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:41.835 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:41.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:41.835 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:41.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:41.835 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:41.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:41.835 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:41.670+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:42.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:42.025+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:42.025910+0000) 2024-04-27T03:20:42.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:41.826+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:42.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:42.047+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:42.048602+0000) 2024-04-27T03:20:42.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:42.194+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:42.195290+0000) 2024-04-27T03:20:42.807 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:42.502+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:42.807 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:42.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:42.808 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:42.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:42.808 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:42.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:42.808 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:42.656+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:43.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:43.029+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:43.030130+0000) 2024-04-27T03:20:43.079 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:43.049+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:43.050728+0000) 2024-04-27T03:20:43.079 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:42.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:43.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:42.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:43.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:42.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:43.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:42.800+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:43.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:43.196+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:43.197556+0000) 2024-04-27T03:20:43.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:43.462+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:43.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:43.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:43.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:43.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:43.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:43.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:43.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:43.671+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:44.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:44.050+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:44.051474+0000) 2024-04-27T03:20:44.079 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:44.030+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:44.031214+0000) 2024-04-27T03:20:44.079 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:43.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:44.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:43.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:44.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:43.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:44.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:43.844+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:44.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:44.197+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:44.198465+0000) 2024-04-27T03:20:44.727 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:20:44.550+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59617 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:20:44.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:44.546+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:44.547200+0000) 2024-04-27T03:20:44.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:44.443+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:44.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:44.546+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:44.547619+0000) 2024-04-27T03:20:44.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:44.546+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:44.546800+0000) 2024-04-27T03:20:44.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:44.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:44.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:44.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:44.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:44.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:44.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:44.638+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:45.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:45.050+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:45.051668+0000) 2024-04-27T03:20:45.148 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:45.030+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:45.031412+0000) 2024-04-27T03:20:45.148 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:44.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:45.148 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:44.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:45.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:44.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:45.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:44.895+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:45.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:45.197+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:45.198643+0000) 2024-04-27T03:20:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:45.438+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:45.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:45.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:45.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:45.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:45.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:45.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:45.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:45.631+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:46.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:46.053+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:46.053861+0000) 2024-04-27T03:20:46.148 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:46.032+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:46.033529+0000) 2024-04-27T03:20:46.148 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:45.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:46.148 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:45.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:46.148 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:45.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:46.148 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:45.932+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:46.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:46.202+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:46.203675+0000) 2024-04-27T03:20:46.860 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:46.475+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:46.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:46.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:46.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:46.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:46.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:46.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:46.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:46.599+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:47.148 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:47.034+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:47.035601+0000) 2024-04-27T03:20:47.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:47.055+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:47.056148+0000) 2024-04-27T03:20:47.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:46.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:47.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:46.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:47.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:46.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:47.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:46.917+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:47.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:47.207+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:47.208757+0000) 2024-04-27T03:20:47.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:47.434+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:47.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:47.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:47.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:47.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:47.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:47.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:47.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:47.559+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:48.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:48.035+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:48.037702+0000) 2024-04-27T03:20:48.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:48.056+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:48.058489+0000) 2024-04-27T03:20:48.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:47.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:48.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:47.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:48.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:47.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:48.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:47.956+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:48.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:48.212+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:48.213825+0000) 2024-04-27T03:20:48.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:48.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:48.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:48.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:48.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:48.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:48.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:48.463+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:48.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:48.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:48.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:48.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:48.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:48.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:48.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:48.565+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:49.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:49.057+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:49.059235+0000) 2024-04-27T03:20:49.148 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:49.036+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:49.038411+0000) 2024-04-27T03:20:49.148 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:48.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:49.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:48.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:49.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:48.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:49.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:48.953+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:49.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:49.213+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:49.215474+0000) 2024-04-27T03:20:49.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:20:49.557+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59657 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:20:49.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:49.538+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:49.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:49.538+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:49.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:49.538+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:49.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:49.539+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:49.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:49.555+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:49.557119+0000) 2024-04-27T03:20:49.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:49.553+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:49.554934+0000) 2024-04-27T03:20:49.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:49.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:49.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:49.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:49.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:49.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:49.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:49.434+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:49.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:49.554+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:49.556188+0000) 2024-04-27T03:20:50.148 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:50.036+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:50.038604+0000) 2024-04-27T03:20:50.148 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:49.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:50.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:49.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:50.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:49.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:50.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:49.979+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:50.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:50.057+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:50.059421+0000) 2024-04-27T03:20:50.408 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:50.213+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:50.215760+0000) 2024-04-27T03:20:50.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:50.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:50.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:50.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:50.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:50.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:50.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:50.495+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:50.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:50.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:50.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:50.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:50.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:50.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:50.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:50.401+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:51.148 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:51.039+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:51.041352+0000) 2024-04-27T03:20:51.148 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:51.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:51.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:51.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:51.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:51.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:51.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:51.007+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:51.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:51.061+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:51.063054+0000) 2024-04-27T03:20:51.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:51.216+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:51.217848+0000) 2024-04-27T03:20:51.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:51.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:51.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:51.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:51.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:51.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:51.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:51.453+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:51.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:51.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:51.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:51.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:51.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:51.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:51.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:51.430+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:52.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:52.063+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:52.065756+0000) 2024-04-27T03:20:52.148 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:52.041+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:52.043644+0000) 2024-04-27T03:20:52.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:52.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:52.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:52.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:52.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:52.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:52.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:52.010+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:52.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:52.219+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:52.221041+0000) 2024-04-27T03:20:52.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:52.408+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:52.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:52.408+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:52.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:52.408+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:52.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:52.409+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:52.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:52.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:52.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:52.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:52.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:52.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:52.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:52.498+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:53.148 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:53.044+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:53.045824+0000) 2024-04-27T03:20:53.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:53.067+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:53.069110+0000) 2024-04-27T03:20:53.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:52.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:53.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:52.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:53.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:52.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:53.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:52.992+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:53.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:53.221+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:53.223637+0000) 2024-04-27T03:20:53.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:53.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:53.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:53.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:53.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:53.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:53.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:53.434+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:53.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:53.525+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:53.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:53.525+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:53.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:53.525+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:53.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:53.525+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:54.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:54.045+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:54.047098+0000) 2024-04-27T03:20:54.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:53.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:54.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:53.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:54.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:53.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:54.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:53.982+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:54.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:54.223+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:54.225001+0000) 2024-04-27T03:20:54.413 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:54.068+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:54.069979+0000) 2024-04-27T03:20:54.690 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:20:54.565+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59696 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:20:54.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:54.562+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:54.564504+0000) 2024-04-27T03:20:54.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:54.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:54.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:54.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:54.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:54.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:54.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:54.406+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:54.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:54.562+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:54.564072+0000) 2024-04-27T03:20:54.691 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:54.564+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:54.566415+0000) 2024-04-27T03:20:54.691 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:54.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:54.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:54.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:54.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:54.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:54.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:54.575+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:55.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:55.068+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:55.070247+0000) 2024-04-27T03:20:55.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:55.045+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:55.047298+0000) 2024-04-27T03:20:55.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:54.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:55.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:54.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:55.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:54.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:55.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:54.957+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:55.554 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:55.223+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:55.225201+0000) 2024-04-27T03:20:55.554 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:55.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:55.554 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:55.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:55.554 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:55.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:55.555 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:55.432+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:55.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:55.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:55.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:55.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:55.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:55.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:55.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:55.547+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:56.149 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:56.047+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:56.049444+0000) 2024-04-27T03:20:56.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:55.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:56.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:55.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:56.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:55.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:56.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:55.972+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:56.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:56.070+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:56.072402+0000) 2024-04-27T03:20:56.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:56.227+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:56.229399+0000) 2024-04-27T03:20:56.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:56.407+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:56.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:56.407+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:56.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:56.407+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:56.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:56.408+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:56.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:56.505+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:56.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:56.505+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:56.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:56.505+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:56.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:56.506+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:57.149 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:57.050+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:57.052637+0000) 2024-04-27T03:20:57.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:57.074+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:57.076713+0000) 2024-04-27T03:20:57.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:56.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:57.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:56.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:57.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:56.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:57.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:56.931+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:57.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:57.231+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:57.233161+0000) 2024-04-27T03:20:57.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:57.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:57.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:57.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:57.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:57.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:57.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:57.436+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:57.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:57.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:57.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:57.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:57.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:57.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:57.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:57.467+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:57.999 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:57.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:57.999 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:57.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:57.999 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:57.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:57.999 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:57.970+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:58.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:58.053+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:58.055407+0000) 2024-04-27T03:20:58.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:58.233+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:58.235398+0000) 2024-04-27T03:20:58.451 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:58.077+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:58.078845+0000) 2024-04-27T03:20:58.451 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:58.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:58.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:58.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:58.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:58.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:58.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:58.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:58.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:58.450+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:58.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:58.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:58.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:58.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:58.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:58.443+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:59.149 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:59.055+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:59.057055+0000) 2024-04-27T03:20:59.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:59.078+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:59.080213+0000) 2024-04-27T03:20:59.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:58.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:59.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:58.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:59.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:58.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:20:59.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:58.964+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:59.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:59.234+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:59.236793+0000) 2024-04-27T03:20:59.868 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:20:59.571+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59737 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:20:59.869 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:59.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:59.869 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:59.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:59.869 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:59.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:20:59.869 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:59.491+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:59.869 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:20:59.569+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:59.571182+0000) 2024-04-27T03:20:59.869 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:59.571+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:59.572992+0000) 2024-04-27T03:20:59.870 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:59.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:59.870 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:59.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:59.870 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:59.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:20:59.870 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:59.473+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:20:59.870 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:20:59.569+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:20:59.571366+0000) 2024-04-27T03:21:00.149 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:00.055+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:00.057317+0000) 2024-04-27T03:21:00.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:00.078+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:00.080423+0000) 2024-04-27T03:21:00.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:59.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:00.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:59.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:00.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:59.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:00.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:20:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:20:59.918+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:00.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:00.236+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:00.236980+0000) 2024-04-27T03:21:00.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:00.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:00.805 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:00.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:00.805 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:00.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:00.805 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:00.478+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:00.805 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:00.476+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:00.805 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:00.476+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:00.805 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:00.476+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:00.805 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:00.478+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:01.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:00.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:01.066 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:00.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:01.066 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:00.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:01.066 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:00.946+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:01.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:01.082+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:01.083296+0000) 2024-04-27T03:21:01.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:01.238+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:01.239635+0000) 2024-04-27T03:21:01.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:01.058+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:01.059483+0000) 2024-04-27T03:21:01.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:01.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:01.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:01.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:01.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:01.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:01.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:01.461+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:01.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:01.477+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:01.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:01.477+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:01.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:01.477+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:01.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:01.478+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:02.149 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:02.061+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:02.062220+0000) 2024-04-27T03:21:02.150 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:02.085+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:02.085942+0000) 2024-04-27T03:21:02.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:01.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:02.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:01.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:02.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:01.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:02.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:01.992+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:02.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:02.241+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:02.242777+0000) 2024-04-27T03:21:02.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:02.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:02.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:02.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:02.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:02.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:02.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:02.468+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:02.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:02.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:02.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:02.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:02.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:02.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:02.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:02.455+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:03.149 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:03.063+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:03.064471+0000) 2024-04-27T03:21:03.150 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:03.088+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:03.089096+0000) 2024-04-27T03:21:03.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:03.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:03.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:03.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:03.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:03.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:03.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:03.030+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:03.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:03.244+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:03.245529+0000) 2024-04-27T03:21:03.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:03.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:03.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:03.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:03.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:03.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:03.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:03.466+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:03.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:03.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:03.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:03.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:03.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:03.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:03.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:03.441+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:04.253 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:04.064+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:04.065378+0000) 2024-04-27T03:21:04.253 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:04.089+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:04.089880+0000) 2024-04-27T03:21:04.254 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:04.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:04.254 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:04.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:04.254 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:04.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:04.254 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:04.025+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:04.582 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:04.246+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:04.247022+0000) 2024-04-27T03:21:04.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:04.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:04.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:04.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:04.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:04.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:04.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:04.490+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:04.583 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:04.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:04.583 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:04.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:04.583 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:04.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:04.583 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:04.456+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:04.873 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:21:04.579+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59776 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:21:04.873 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:04.577+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:04.578050+0000) 2024-04-27T03:21:04.874 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:04.578+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:04.579629+0000) 2024-04-27T03:21:04.874 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:04.577+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:04.578718+0000) 2024-04-27T03:21:05.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:05.064+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:05.065660+0000) 2024-04-27T03:21:05.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:04.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:05.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:04.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:05.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:04.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:05.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:04.995+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:05.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:05.089+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:05.090099+0000) 2024-04-27T03:21:05.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:05.246+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:05.247278+0000) 2024-04-27T03:21:05.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:05.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:05.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:05.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:05.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:05.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:05.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:05.502+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:05.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:05.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:05.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:05.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:05.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:05.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:05.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:05.515+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:06.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:06.068+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:06.068825+0000) 2024-04-27T03:21:06.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:05.990+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:06.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:05.990+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:06.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:05.990+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:06.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:05.990+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:06.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:06.091+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:06.092177+0000) 2024-04-27T03:21:06.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:06.248+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:06.249547+0000) 2024-04-27T03:21:06.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:06.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:06.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:06.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:06.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:06.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:06.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:06.528+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:06.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:06.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:06.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:06.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:06.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:06.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:06.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:06.498+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:07.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:07.070+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:07.071490+0000) 2024-04-27T03:21:07.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:06.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:07.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:06.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:07.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:06.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:07.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:06.944+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:07.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:07.094+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:07.095246+0000) 2024-04-27T03:21:07.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:07.251+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:07.252360+0000) 2024-04-27T03:21:07.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:07.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:07.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:07.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:07.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:07.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:07.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:07.524+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:07.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:07.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:07.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:07.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:07.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:07.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:07.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:07.546+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:08.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:08.072+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:08.073708+0000) 2024-04-27T03:21:08.150 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:08.097+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:08.098071+0000) 2024-04-27T03:21:08.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:07.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:08.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:07.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:08.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:07.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:08.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:07.989+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:08.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:08.254+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:08.255326+0000) 2024-04-27T03:21:08.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:08.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:08.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:08.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:08.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:08.593+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:08.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:08.594+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:08.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:08.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:08.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:08.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:08.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:08.513+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:08.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:08.513+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:09.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:09.074+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:09.075160+0000) 2024-04-27T03:21:09.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:08.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:09.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:08.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:09.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:08.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:09.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:08.982+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:09.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:09.098+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:09.099658+0000) 2024-04-27T03:21:09.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:09.256+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:09.256830+0000) 2024-04-27T03:21:09.877 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:21:09.587+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59816 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:21:09.877 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:09.586+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:09.586827+0000) 2024-04-27T03:21:09.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:09.584+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:09.585233+0000) 2024-04-27T03:21:09.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:09.621+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:09.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:09.621+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:09.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:09.621+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:09.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:09.622+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:09.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:09.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:09.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:09.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:09.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:09.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:09.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:09.557+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:09.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:09.584+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:09.585463+0000) 2024-04-27T03:21:10.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:10.099+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:10.099823+0000) 2024-04-27T03:21:10.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:10.074+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:10.075355+0000) 2024-04-27T03:21:10.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:10.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:10.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:10.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:10.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:10.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:10.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:10.031+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:10.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:10.256+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:10.257018+0000) 2024-04-27T03:21:10.880 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:10.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:10.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:10.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:10.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:10.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:10.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:10.595+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:10.881 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:10.599+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:10.881 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:10.599+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:10.881 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:10.599+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:10.881 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:10.599+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:11.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:11.079+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:11.080382+0000) 2024-04-27T03:21:11.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:11.102+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:11.103016+0000) 2024-04-27T03:21:11.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:11.002+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:11.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:11.002+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:11.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:11.002+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:11.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:11.002+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:11.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:11.261+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:11.261980+0000) 2024-04-27T03:21:11.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:11.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:11.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:11.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:11.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:11.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:11.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:11.551+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:11.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:11.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:11.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:11.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:11.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:11.549+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:11.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:11.551+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:12.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:12.084+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:12.085462+0000) 2024-04-27T03:21:12.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:12.106+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:12.107633+0000) 2024-04-27T03:21:12.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:12.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:12.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:12.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:12.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:12.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:12.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:12.037+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:12.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:12.266+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:12.267076+0000) 2024-04-27T03:21:12.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:12.588+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:12.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:12.588+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:12.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:12.588+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:12.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:12.590+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:12.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:12.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:12.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:12.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:12.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:12.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:12.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:12.503+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:13.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:13.089+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:13.090600+0000) 2024-04-27T03:21:13.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:13.111+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:13.111972+0000) 2024-04-27T03:21:13.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:13.059+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:13.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:13.059+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:13.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:13.059+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:13.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:13.059+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:13.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:13.271+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:13.272274+0000) 2024-04-27T03:21:13.793 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:13.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:13.793 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:13.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:13.793 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:13.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:13.793 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:13.633+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:13.793 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:13.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:13.794 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:13.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:13.794 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:13.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:13.794 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:13.500+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:14.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:14.090+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:14.091494+0000) 2024-04-27T03:21:14.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:14.112+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:14.113690+0000) 2024-04-27T03:21:14.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:14.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:14.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:14.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:14.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:14.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:14.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:14.073+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:14.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:14.272+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:14.273250+0000) 2024-04-27T03:21:14.882 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:21:14.594+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59855 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:21:14.883 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:14.591+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:14.592424+0000) 2024-04-27T03:21:14.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:14.593+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:14.594599+0000) 2024-04-27T03:21:14.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:14.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:14.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:14.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:14.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:14.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:14.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:14.630+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:14.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:14.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:14.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:14.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:14.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:14.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:14.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:14.546+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:14.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:14.592+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:14.593208+0000) 2024-04-27T03:21:15.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:15.090+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:15.091763+0000) 2024-04-27T03:21:15.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:15.113+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:15.113954+0000) 2024-04-27T03:21:15.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:15.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:15.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:15.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:15.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:15.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:15.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:15.083+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:15.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:15.272+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:15.273434+0000) 2024-04-27T03:21:15.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:15.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:15.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:15.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:15.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:15.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:15.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:15.585+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:15.886 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:15.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:15.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:15.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:15.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:15.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:15.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:15.593+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:16.139 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:16.095+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:16.095998+0000) 2024-04-27T03:21:16.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:16.117+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:16.118197+0000) 2024-04-27T03:21:16.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:16.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:16.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:16.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:16.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:16.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:16.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:16.133+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:16.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:16.275+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:16.276607+0000) 2024-04-27T03:21:16.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:16.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:16.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:16.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:16.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:16.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:16.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:16.573+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:16.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:16.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:16.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:16.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:16.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:16.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:16.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:16.580+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:17.139 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:17.097+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:17.098263+0000) 2024-04-27T03:21:17.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:17.120+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:17.120810+0000) 2024-04-27T03:21:17.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:17.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:17.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:17.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:17.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:17.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:17.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:17.133+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:17.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:17.278+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:17.278802+0000) 2024-04-27T03:21:17.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:17.603+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:17.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:17.603+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:17.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:17.603+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:17.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:17.605+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:17.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:17.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:17.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:17.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:17.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:17.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:17.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:17.607+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:18.138 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:18.099+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:18.100596+0000) 2024-04-27T03:21:18.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:18.122+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:18.123543+0000) 2024-04-27T03:21:18.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:18.131+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:18.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:18.131+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:18.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:18.131+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:18.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:18.131+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:18.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:18.281+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:18.282000+0000) 2024-04-27T03:21:18.842 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:18.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:18.842 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:18.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:18.842 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:18.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:18.842 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:18.625+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:18.842 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:18.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:18.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:18.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:18.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:18.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:18.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:18.565+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:19.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:19.100+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:19.101789+0000) 2024-04-27T03:21:19.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:19.124+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:19.125001+0000) 2024-04-27T03:21:19.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:19.101+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:19.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:19.101+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:19.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:19.101+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:19.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:19.101+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:19.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:19.282+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:19.282936+0000) 2024-04-27T03:21:19.887 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:21:19.601+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59895 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:21:19.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:19.598+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:19.599560+0000) 2024-04-27T03:21:19.888 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:19.596+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:19.888 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:19.596+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:19.888 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:19.596+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:19.888 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:19.598+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:19.889 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:19.600+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:19.601459+0000) 2024-04-27T03:21:19.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:19.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:19.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:19.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:19.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:19.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:19.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:19.532+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:19.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:19.599+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:19.600169+0000) 2024-04-27T03:21:20.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:20.101+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:20.102015+0000) 2024-04-27T03:21:20.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:20.124+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:20.125185+0000) 2024-04-27T03:21:20.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:20.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:20.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:20.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:20.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:20.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:20.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:20.070+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:20.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:20.282+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:20.283184+0000) 2024-04-27T03:21:20.837 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:20.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:20.837 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:20.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:20.837 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:20.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:20.837 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:20.555+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:20.837 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:20.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:20.837 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:20.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:20.838 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:20.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:20.838 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:20.580+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:21.113 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:21.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:21.113 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:21.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:21.113 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:21.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:21.113 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:21.031+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:21.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:21.105+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:21.106778+0000) 2024-04-27T03:21:21.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:21.126+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:21.127338+0000) 2024-04-27T03:21:21.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:21.285+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:21.286247+0000) 2024-04-27T03:21:21.892 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:21.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:21.892 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:21.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:21.892 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:21.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:21.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:21.519+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:21.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:21.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:21.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:21.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:21.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:21.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:21.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:21.538+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:22.152 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:22.108+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:22.109111+0000) 2024-04-27T03:21:22.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:22.130+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:22.131640+0000) 2024-04-27T03:21:22.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:22.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:22.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:22.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:22.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:22.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:22.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:22.018+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:22.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:22.287+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:22.288516+0000) 2024-04-27T03:21:22.892 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:22.476+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:22.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:22.476+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:22.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:22.476+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:22.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:22.477+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:22.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:22.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:22.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:22.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:22.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:22.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:22.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:22.538+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:23.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:23.132+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:23.133789+0000) 2024-04-27T03:21:23.152 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:23.111+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:23.112677+0000) 2024-04-27T03:21:23.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:23.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:23.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:23.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:23.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:23.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:23.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:23.033+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:23.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:23.289+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:23.290708+0000) 2024-04-27T03:21:23.815 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:23.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:23.815 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:23.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:23.815 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:23.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:23.815 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:23.501+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:23.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:23.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:23.816 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:23.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:23.816 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:23.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:23.816 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:23.559+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:24.082 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:23.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:24.083 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:23.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:24.083 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:23.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:24.083 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:23.986+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:24.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:24.112+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:24.113612+0000) 2024-04-27T03:21:24.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:24.290+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:24.291536+0000) 2024-04-27T03:21:24.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:24.134+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:24.135255+0000) 2024-04-27T03:21:24.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:24.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:24.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:24.605+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:24.606571+0000) 2024-04-27T03:21:24.727 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:21:24.608+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59933 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:21:24.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:24.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:24.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:24.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:24.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:24.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:24.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:24.558+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:24.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:24.606+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:24.607460+0000) 2024-04-27T03:21:24.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:24.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:24.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:24.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:24.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:24.466+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:24.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:24.605+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:24.606444+0000) 2024-04-27T03:21:25.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:25.113+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:25.113799+0000) 2024-04-27T03:21:25.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:25.134+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:25.135463+0000) 2024-04-27T03:21:25.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:24.982+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:25.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:24.982+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:25.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:24.982+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:25.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:24.983+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:25.588 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:25.290+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:25.291742+0000) 2024-04-27T03:21:25.588 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:25.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:25.589 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:25.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:25.589 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:25.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:25.589 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:25.494+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:25.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:25.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:25.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:25.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:25.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:25.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:25.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:25.582+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:26.146 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:26.115+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:26.115841+0000) 2024-04-27T03:21:26.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:25.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:26.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:25.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:26.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:25.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:26.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:25.961+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:26.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:26.138+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:26.139752+0000) 2024-04-27T03:21:26.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:26.294+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:26.295711+0000) 2024-04-27T03:21:26.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:26.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:26.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:26.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:26.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:26.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:26.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:26.562+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:26.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:26.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:26.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:26.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:26.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:26.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:26.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:26.473+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:27.149 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:27.119+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:27.120647+0000) 2024-04-27T03:21:27.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:26.977+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:27.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:26.977+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:27.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:26.977+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:27.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:26.977+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:27.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:27.142+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:27.142974+0000) 2024-04-27T03:21:27.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:27.298+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:27.298920+0000) 2024-04-27T03:21:27.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:27.516+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:27.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:27.516+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:27.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:27.516+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:27.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:27.518+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:27.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:27.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:27.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:27.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:27.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:27.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:27.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:27.531+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:28.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:28.124+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:28.125361+0000) 2024-04-27T03:21:28.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:27.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:28.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:27.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:28.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:27.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:28.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:27.992+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:28.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:28.144+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:28.145261+0000) 2024-04-27T03:21:28.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:28.300+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:28.301605+0000) 2024-04-27T03:21:28.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:28.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:28.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:28.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:28.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:28.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:28.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:28.554+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:28.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:28.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:28.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:28.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:28.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:28.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:28.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:28.546+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:29.152 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:29.125+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:29.126491+0000) 2024-04-27T03:21:29.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:28.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:29.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:28.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:29.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:28.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:29.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:28.952+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:29.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:29.145+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:29.146609+0000) 2024-04-27T03:21:29.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:29.302+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:29.302980+0000) 2024-04-27T03:21:29.896 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:21:29.615+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 59974 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:21:29.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:29.614+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:29.615266+0000) 2024-04-27T03:21:29.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:29.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:29.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:29.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:29.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:29.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:29.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:29.544+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:29.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:29.613+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:29.613810+0000) 2024-04-27T03:21:29.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:29.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:29.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:29.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:29.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:29.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:29.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:29.528+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:29.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:29.612+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:29.613789+0000) 2024-04-27T03:21:30.153 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:30.125+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:30.126690+0000) 2024-04-27T03:21:30.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:29.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:30.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:29.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:30.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:29.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:30.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:29.925+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:30.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:30.146+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:30.146819+0000) 2024-04-27T03:21:30.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:30.302+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:30.303158+0000) 2024-04-27T03:21:30.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:30.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:30.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:30.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:30.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:30.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:30.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:30.579+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:30.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:30.555+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:30.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:30.555+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:30.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:30.555+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:30.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:30.556+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:31.153 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:31.128+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:31.128840+0000) 2024-04-27T03:21:31.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:30.974+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:31.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:30.974+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:31.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:30.974+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:31.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:30.975+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:31.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:31.150+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:31.151074+0000) 2024-04-27T03:21:31.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:31.306+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:31.307376+0000) 2024-04-27T03:21:31.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:31.539+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:31.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:31.539+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:31.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:31.539+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:31.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:31.540+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:31.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:31.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:31.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:31.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:31.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:31.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:31.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:31.603+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:32.153 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:32.131+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:32.132051+0000) 2024-04-27T03:21:32.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:31.945+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:32.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:31.945+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:32.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:31.945+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:32.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:31.946+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:32.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:32.152+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:32.153274+0000) 2024-04-27T03:21:32.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:32.310+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:32.311616+0000) 2024-04-27T03:21:32.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:32.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:32.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:32.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:32.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:32.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:32.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:32.504+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:32.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:32.612+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:32.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:32.612+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:32.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:32.612+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:32.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:32.612+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:33.153 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:33.134+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:33.135192+0000) 2024-04-27T03:21:33.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:32.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:33.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:32.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:33.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:32.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:33.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:32.929+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:33.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:33.154+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:33.155404+0000) 2024-04-27T03:21:33.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:33.315+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:33.315845+0000) 2024-04-27T03:21:33.816 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:33.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:33.816 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:33.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:33.816 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:33.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:33.816 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:33.504+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:33.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:33.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:33.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:33.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:33.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:33.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:33.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:33.620+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:34.083 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:33.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:34.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:33.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:34.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:33.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:34.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:33.921+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:34.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:34.155+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:34.156357+0000) 2024-04-27T03:21:34.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:34.316+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:34.317231+0000) 2024-04-27T03:21:34.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:34.135+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:34.136062+0000) 2024-04-27T03:21:34.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:34.620+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:34.621220+0000) 2024-04-27T03:21:34.901 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:21:34.622+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60012 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:21:34.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:34.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:34.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:34.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:34.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:34.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:34.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:34.608+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:34.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:34.621+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:34.622765+0000) 2024-04-27T03:21:34.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:34.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:34.902 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:34.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:34.902 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:34.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:34.902 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:34.526+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:34.902 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:34.621+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:34.621895+0000) 2024-04-27T03:21:35.153 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:35.135+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:35.136260+0000) 2024-04-27T03:21:35.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:34.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:35.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:34.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:35.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:34.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:35.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:34.922+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:35.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:35.316+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:35.317407+0000) 2024-04-27T03:21:35.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:35.155+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:35.156545+0000) 2024-04-27T03:21:35.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:35.523+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:35.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:35.523+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:35.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:35.523+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:35.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:35.524+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:35.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:35.581+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:35.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:35.581+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:35.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:35.581+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:35.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:35.581+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:36.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:36.137+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:36.138354+0000) 2024-04-27T03:21:36.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:35.904+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:36.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:35.904+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:36.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:35.904+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:36.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:35.905+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:36.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:36.318+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:36.319751+0000) 2024-04-27T03:21:36.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:36.160+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:36.161265+0000) 2024-04-27T03:21:36.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:36.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:36.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:36.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:36.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:36.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:36.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:36.507+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:36.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:36.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:36.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:36.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:36.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:36.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:36.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:36.578+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:37.148 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:36.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:37.148 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:36.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:37.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:36.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:37.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:36.871+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:37.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:37.141+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:37.142633+0000) 2024-04-27T03:21:37.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:37.322+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:37.323405+0000) 2024-04-27T03:21:37.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:37.165+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:37.166055+0000) 2024-04-27T03:21:37.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:37.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:37.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:37.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:37.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:37.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:37.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:37.492+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:37.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:37.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:37.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:37.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:37.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:37.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:37.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:37.531+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:38.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:38.144+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:38.144925+0000) 2024-04-27T03:21:38.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:37.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:38.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:37.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:38.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:37.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:38.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:37.908+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:38.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:38.324+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:38.325672+0000) 2024-04-27T03:21:38.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:38.170+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:38.170891+0000) 2024-04-27T03:21:38.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:38.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:38.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:38.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:38.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:38.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:38.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:38.480+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:38.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:38.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:38.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:38.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:38.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:38.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:38.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:38.513+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:39.178 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:39.144+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:39.145719+0000) 2024-04-27T03:21:39.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:38.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:39.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:38.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:39.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:38.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:39.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:38.913+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:39.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:39.326+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:39.327224+0000) 2024-04-27T03:21:39.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:39.170+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:39.171767+0000) 2024-04-27T03:21:39.757 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:21:39.630+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60053 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:21:39.758 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:39.628+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:39.628923+0000) 2024-04-27T03:21:39.758 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:39.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:39.758 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:39.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:39.758 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:39.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:39.758 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:39.513+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:39.758 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:39.627+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:39.628700+0000) 2024-04-27T03:21:39.759 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:39.493+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:39.759 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:39.493+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:39.759 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:39.493+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:39.759 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:39.493+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:39.759 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:39.627+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:39.628245+0000) 2024-04-27T03:21:40.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:39.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:40.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:39.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:40.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:39.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:40.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:39.915+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:40.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:40.326+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:40.327401+0000) 2024-04-27T03:21:40.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:40.145+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:40.145899+0000) 2024-04-27T03:21:40.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:40.171+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:40.171975+0000) 2024-04-27T03:21:40.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:40.515+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:40.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:40.515+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:40.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:40.515+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:40.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:40.516+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:40.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:40.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:40.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:40.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:40.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:40.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:40.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:40.533+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:41.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:41.147+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:41.148081+0000) 2024-04-27T03:21:41.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:40.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:41.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:40.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:41.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:40.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:41.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:40.928+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:41.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:41.173+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:41.174164+0000) 2024-04-27T03:21:41.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:41.328+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:41.329510+0000) 2024-04-27T03:21:41.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:41.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:41.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:41.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:41.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:41.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:41.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:41.544+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:41.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:41.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:41.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:41.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:41.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:41.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:41.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:41.509+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:42.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:42.149+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:42.150755+0000) 2024-04-27T03:21:42.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:41.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:42.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:41.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:42.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:41.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:42.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:41.954+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:42.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:42.330+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:42.331595+0000) 2024-04-27T03:21:42.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:42.175+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:42.176423+0000) 2024-04-27T03:21:42.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:42.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:42.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:42.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:42.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:42.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:42.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:42.522+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:42.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:42.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:42.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:42.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:42.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:42.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:42.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:42.509+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:43.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:43.152+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:43.153032+0000) 2024-04-27T03:21:43.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:42.989+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:43.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:42.989+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:43.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:42.989+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:43.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:42.990+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:43.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:43.334+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:43.335673+0000) 2024-04-27T03:21:43.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:43.178+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:43.179490+0000) 2024-04-27T03:21:43.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:43.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:43.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:43.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:43.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:43.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:43.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:43.461+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:43.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:43.562+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:43.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:43.562+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:43.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:43.562+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:43.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:43.564+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:44.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:44.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:44.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:44.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:44.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:44.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:44.085 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:44.038+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:44.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:44.153+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:44.154002+0000) 2024-04-27T03:21:44.342 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:44.179+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:44.180497+0000) 2024-04-27T03:21:44.638 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:44.335+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:44.336540+0000) 2024-04-27T03:21:44.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:44.585+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:44.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:44.585+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:44.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:44.585+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:44.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:44.585+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:44.639 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:44.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:44.639 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:44.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:44.639 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:44.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:44.639 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:44.465+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:44.911 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:21:44.638+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60091 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:21:44.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:44.635+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:44.636165+0000) 2024-04-27T03:21:44.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:44.635+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:44.636096+0000) 2024-04-27T03:21:44.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:44.635+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:44.636291+0000) 2024-04-27T03:21:45.187 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:45.153+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:45.154201+0000) 2024-04-27T03:21:45.187 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:45.068+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:45.187 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:45.068+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:45.187 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:45.068+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:45.187 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:45.069+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:45.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:45.336+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:45.336819+0000) 2024-04-27T03:21:45.453 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:45.179+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:45.180696+0000) 2024-04-27T03:21:45.453 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:45.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:45.453 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:45.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:45.453 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:45.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:45.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:45.447+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:45.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:45.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:45.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:45.608+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:46.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:46.155+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:46.156535+0000) 2024-04-27T03:21:46.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:46.051+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:46.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:46.051+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:46.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:46.051+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:46.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:46.052+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:46.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:46.338+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:46.339482+0000) 2024-04-27T03:21:46.452 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:46.182+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:46.182822+0000) 2024-04-27T03:21:46.452 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:46.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:46.452 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:46.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:46.452 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:46.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:46.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:46.446+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:46.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:46.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:46.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:46.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:46.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:46.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:46.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:46.568+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:47.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:47.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:47.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:47.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:47.155 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:47.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:47.155 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:47.038+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:47.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:47.158+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:47.159347+0000) 2024-04-27T03:21:47.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:47.341+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:47.342448+0000) 2024-04-27T03:21:47.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:47.184+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:47.185744+0000) 2024-04-27T03:21:47.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:47.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:47.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:47.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:47.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:47.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:47.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:47.529+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:47.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:47.481+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:47.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:47.481+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:47.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:47.481+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:47.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:47.481+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:48.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:48.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:48.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:48.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:48.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:48.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:48.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:48.023+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:48.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:48.188+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:48.188965+0000) 2024-04-27T03:21:48.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:48.161+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:48.162562+0000) 2024-04-27T03:21:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:48.345+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:48.346777+0000) 2024-04-27T03:21:48.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:48.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:48.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:48.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:48.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:48.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:48.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:48.533+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:48.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:48.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:48.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:48.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:48.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:48.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:48.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:48.565+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:49.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:49.044+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:49.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:49.044+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:49.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:49.044+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:49.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:49.044+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:49.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:49.347+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:49.348166+0000) 2024-04-27T03:21:49.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:49.163+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:49.164008+0000) 2024-04-27T03:21:49.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:49.189+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:49.189818+0000) 2024-04-27T03:21:49.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T03:21:49.609+0000 7ff0e4ed9700 0 monclient(hunting): authenticate timed out after 300 2024-04-27T03:21:49.774 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:21:49.645+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60132 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:21:49.774 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:49.642+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:49.643268+0000) 2024-04-27T03:21:49.775 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:49.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:49.775 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:49.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:49.775 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:49.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:49.775 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:49.567+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:49.775 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:49.642+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:49.643748+0000) 2024-04-27T03:21:49.776 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:49.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:49.776 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:49.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:49.776 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:49.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:49.776 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:49.512+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:49.776 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:49.643+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:49.644358+0000) 2024-04-27T03:21:50.354 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:50.163+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:50.164227+0000) 2024-04-27T03:21:50.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:50.189+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:50.190008+0000) 2024-04-27T03:21:50.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:50.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:50.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:50.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:50.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:50.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:50.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:50.032+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:50.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:50.346+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:50.348334+0000) 2024-04-27T03:21:50.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:50.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:50.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:50.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:50.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:50.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:50.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:50.580+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:50.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:50.462+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:50.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:50.462+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:50.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:50.462+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:50.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:50.464+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:51.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:51.011+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:51.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:51.011+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:51.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:51.011+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:51.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:51.011+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:51.454 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:51.166+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:51.168447+0000) 2024-04-27T03:21:51.454 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:51.192+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:51.194331+0000) 2024-04-27T03:21:51.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:51.351+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:51.353029+0000) 2024-04-27T03:21:51.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:51.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:51.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:51.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:51.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:51.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:51.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:51.569+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:51.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:51.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:51.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:51.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:51.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:51.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:51.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:51.447+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:52.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:51.983+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:52.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:51.983+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:52.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:51.983+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:52.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:51.983+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:52.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:52.168+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:52.170738+0000) 2024-04-27T03:21:52.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:52.354+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:52.356222+0000) 2024-04-27T03:21:52.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:52.194+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:52.196398+0000) 2024-04-27T03:21:52.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:52.413+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:52.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:52.413+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:52.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:52.413+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:52.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:52.415+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:52.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:52.555+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:52.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:52.555+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:52.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:52.555+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:52.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:52.556+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:53.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:53.000+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:53.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:53.000+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:53.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:53.000+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:53.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:53.000+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:53.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:53.359+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:53.361021+0000) 2024-04-27T03:21:53.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:53.172+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:53.173994+0000) 2024-04-27T03:21:53.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:53.196+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:53.198525+0000) 2024-04-27T03:21:53.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:53.385+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:53.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:53.385+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:53.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:53.385+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:53.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:53.386+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:53.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:53.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:53.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:53.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:53.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:53.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:53.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:53.591+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:54.181 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:53.977+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:54.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:53.977+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:54.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:53.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:54.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:53.978+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:54.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:54.173+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:54.175552+0000) 2024-04-27T03:21:54.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:54.360+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:54.361825+0000) 2024-04-27T03:21:54.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:54.197+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:54.199409+0000) 2024-04-27T03:21:54.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:54.355+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:54.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:54.355+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:54.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:54.355+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:54.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:54.355+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:54.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:21:54.651+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60170 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:21:54.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:54.649+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:54.651647+0000) 2024-04-27T03:21:54.922 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:54.649+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:54.651723+0000) 2024-04-27T03:21:54.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:54.614+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:54.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:54.614+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:54.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:54.614+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:54.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:54.615+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:54.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:54.649+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:54.651042+0000) 2024-04-27T03:21:55.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:55.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:55.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:55.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:55.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:55.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:55.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:55.011+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:55.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:55.360+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:55.362061+0000) 2024-04-27T03:21:55.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:55.173+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:55.175741+0000) 2024-04-27T03:21:55.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:55.197+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:55.199598+0000) 2024-04-27T03:21:55.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:55.374+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:55.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:55.374+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:55.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:55.374+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:55.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:55.375+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:55.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:55.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:55.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:55.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:55.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:55.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:55.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:55.570+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:56.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:56.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:56.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:56.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:56.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:56.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:56.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:56.020+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:56.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:56.362+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:56.364749+0000) 2024-04-27T03:21:56.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:56.177+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:56.178976+0000) 2024-04-27T03:21:56.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:56.201+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:56.202979+0000) 2024-04-27T03:21:56.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:56.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:56.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:56.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:56.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:56.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:56.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:56.364+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:56.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:56.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:56.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:56.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:56.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:56.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:56.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:56.525+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:57.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:57.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:57.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:57.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:57.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:57.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:57.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:57.005+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:57.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:57.365+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:57.367587+0000) 2024-04-27T03:21:57.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:57.181+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:57.183016+0000) 2024-04-27T03:21:57.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:57.205+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:57.207296+0000) 2024-04-27T03:21:57.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:57.389+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:57.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:57.389+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:57.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:57.389+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:57.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:57.390+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:57.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:57.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:57.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:57.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:57.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:57.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:57.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:57.536+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:58.191 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:58.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:58.191 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:58.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:58.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:58.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:58.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:58.009+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:58.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:58.183+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:58.185578+0000) 2024-04-27T03:21:58.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:58.369+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:58.370930+0000) 2024-04-27T03:21:58.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:58.209+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:58.211643+0000) 2024-04-27T03:21:58.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:58.364+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:58.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:58.364+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:58.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:58.364+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:58.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:58.365+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:58.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:58.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:58.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:58.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:58.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:58.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:58.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:58.513+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:59.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:58.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:59.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:58.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:59.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:58.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:21:59.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:58.976+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:59.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:59.185+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:59.187121+0000) 2024-04-27T03:21:59.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:59.370+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:59.372316+0000) 2024-04-27T03:21:59.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:59.210+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:59.212637+0000) 2024-04-27T03:21:59.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:59.325+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:59.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:59.325+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:59.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:59.325+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:21:59.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:59.326+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:59.786 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:21:59.658+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60211 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:21:59.787 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:21:59.655+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:59.657600+0000) 2024-04-27T03:21:59.787 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:59.656+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:59.658257+0000) 2024-04-27T03:21:59.787 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:59.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:59.787 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:59.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:59.787 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:59.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:21:59.787 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:59.479+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:21:59.788 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:21:59.657+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:21:59.659131+0000) 2024-04-27T03:22:00.086 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:59.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:00.086 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:59.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:00.086 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:59.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:00.086 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:21:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:21:59.949+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:00.363 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:00.185+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:00.187329+0000) 2024-04-27T03:22:00.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:00.211+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:00.212826+0000) 2024-04-27T03:22:00.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:00.355+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:00.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:00.355+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:00.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:00.355+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:00.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:00.370+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:00.372494+0000) 2024-04-27T03:22:00.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:00.357+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:00.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:00.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:00.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:00.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:00.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:00.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:00.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:00.449+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:01.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:00.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:01.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:00.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:01.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:00.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:01.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:00.909+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:01.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:01.188+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:01.190276+0000) 2024-04-27T03:22:01.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:01.372+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:01.374638+0000) 2024-04-27T03:22:01.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:01.213+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:01.215657+0000) 2024-04-27T03:22:01.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:01.331+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:01.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:01.331+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:01.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:01.331+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:01.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:01.333+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:01.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:01.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:01.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:01.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:01.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:01.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:01.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:01.442+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:02.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:01.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:02.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:01.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:02.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:01.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:02.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:01.898+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:02.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:02.375+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:02.377014+0000) 2024-04-27T03:22:02.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:02.216+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:02.217945+0000) 2024-04-27T03:22:02.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:02.284+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:02.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:02.284+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:02.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:02.284+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:02.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:02.285+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:02.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:02.190+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:02.192583+0000) 2024-04-27T03:22:02.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:02.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:02.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:02.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:02.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:02.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:02.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:02.420+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:03.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:02.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:03.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:02.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:03.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:02.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:03.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:02.888+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:03.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:03.380+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:03.381245+0000) 2024-04-27T03:22:03.426 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:03.195+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:03.195938+0000) 2024-04-27T03:22:03.426 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:03.219+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:03.220660+0000) 2024-04-27T03:22:03.426 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:03.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:03.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:03.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:03.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:03.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:03.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:03.257+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:03.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:03.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:03.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:03.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:03.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:03.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:03.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:03.420+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:04.086 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:03.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:04.086 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:03.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:04.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:03.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:04.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:03.870+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:04.382 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:04.196+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:04.197676+0000) 2024-04-27T03:22:04.383 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:04.216+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:04.383 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:04.216+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:04.383 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:04.216+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:04.383 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:04.217+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:04.383 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:04.220+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:04.221730+0000) 2024-04-27T03:22:04.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:04.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:04.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:04.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:04.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:04.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:04.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:04.378+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:04.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:04.381+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:04.382403+0000) 2024-04-27T03:22:04.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:22:04.666+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60249 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:22:04.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:04.665+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:04.666724+0000) 2024-04-27T03:22:04.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:04.663+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:04.664660+0000) 2024-04-27T03:22:04.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:04.665+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:04.666400+0000) 2024-04-27T03:22:04.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:04.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:04.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:04.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:04.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:04.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:04.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:04.906+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:05.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:05.381+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:05.382624+0000) 2024-04-27T03:22:05.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:05.197+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:05.197984+0000) 2024-04-27T03:22:05.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:05.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:05.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:05.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:05.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:05.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:05.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:05.401+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:05.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:05.195+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:05.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:05.195+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:05.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:05.195+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:05.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:05.195+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:05.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:05.221+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:05.222011+0000) 2024-04-27T03:22:06.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:05.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:06.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:05.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:06.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:05.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:06.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:05.880+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:06.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:06.384+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:06.384824+0000) 2024-04-27T03:22:06.453 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:06.199+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:06.200044+0000) 2024-04-27T03:22:06.453 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:06.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:06.453 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:06.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:06.453 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:06.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:06.453 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:06.163+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:06.453 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:06.163+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:06.454 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:06.163+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:06.454 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:06.164+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:06.454 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:06.223+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:06.224136+0000) 2024-04-27T03:22:06.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:06.447+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:07.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:06.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:07.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:06.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:07.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:06.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:07.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:06.845+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:07.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:07.202+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:07.203116+0000) 2024-04-27T03:22:07.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:07.386+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:07.387511+0000) 2024-04-27T03:22:07.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:07.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:07.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:07.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:07.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:07.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:07.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:07.200+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:07.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:07.225+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:07.226352+0000) 2024-04-27T03:22:07.810 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:07.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:07.810 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:07.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:07.810 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:07.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:07.810 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:07.482+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:08.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:07.802+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:08.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:07.802+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:08.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:07.802+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:08.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:07.803+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:08.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:08.204+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:08.205303+0000) 2024-04-27T03:22:08.383 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:08.190+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:08.383 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:08.190+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:08.383 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:08.190+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:08.383 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:08.192+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:08.383 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:08.230+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:08.231331+0000) 2024-04-27T03:22:08.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:08.389+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:08.389886+0000) 2024-04-27T03:22:08.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:08.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:08.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:08.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:08.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:08.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:08.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:08.453+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:09.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:08.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:09.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:08.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:09.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:08.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:09.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:08.756+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:09.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:09.205+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:09.206184+0000) 2024-04-27T03:22:09.383 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:09.145+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:09.383 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:09.145+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:09.383 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:09.145+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:09.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:09.146+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:09.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:09.231+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:09.232234+0000) 2024-04-27T03:22:09.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:09.391+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:09.391919+0000) 2024-04-27T03:22:09.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:09.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:09.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:09.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:09.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:09.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:09.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:09.412+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:09.937 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:09.672+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:09.673115+0000) 2024-04-27T03:22:09.937 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:22:09.675+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60290 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:22:09.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:09.671+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:09.671829+0000) 2024-04-27T03:22:09.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:09.750+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:09.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:09.750+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:09.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:09.750+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:09.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:09.750+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:09.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:09.671+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:09.672570+0000) 2024-04-27T03:22:10.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:10.391+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:10.392189+0000) 2024-04-27T03:22:10.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:10.205+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:10.206376+0000) 2024-04-27T03:22:10.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:10.423+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:10.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:10.423+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:10.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:10.423+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:10.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:10.425+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:10.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:10.181+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:10.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:10.181+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:10.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:10.181+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:10.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:10.182+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:10.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:10.231+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:10.232424+0000) 2024-04-27T03:22:11.005 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:10.732+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:11.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:10.732+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:11.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:10.732+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:11.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:10.732+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:11.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:11.208+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:11.209039+0000) 2024-04-27T03:22:11.383 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:11.193+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:11.383 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:11.193+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:11.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:11.193+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:11.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:11.193+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:11.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:11.233+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:11.234689+0000) 2024-04-27T03:22:11.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:11.394+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:11.394825+0000) 2024-04-27T03:22:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:11.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:11.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:11.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:11.450+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:12.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:11.737+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:12.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:11.737+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:12.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:11.737+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:12.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:11.738+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:12.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:12.210+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:12.211298+0000) 2024-04-27T03:22:12.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:12.183+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:12.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:12.183+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:12.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:12.183+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:12.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:12.184+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:12.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:12.237+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:12.237926+0000) 2024-04-27T03:22:12.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:12.399+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:12.399852+0000) 2024-04-27T03:22:12.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:12.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:12.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:12.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:12.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:12.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:12.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:12.448+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:13.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:12.766+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:13.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:12.766+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:13.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:12.766+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:13.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:12.766+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:13.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:13.213+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:13.214014+0000) 2024-04-27T03:22:13.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:13.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:13.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:13.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:13.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:13.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:13.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:13.144+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:13.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:13.242+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:13.243025+0000) 2024-04-27T03:22:13.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:13.403+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:13.403938+0000) 2024-04-27T03:22:13.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:13.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:13.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:13.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:13.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:13.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:13.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:13.404+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:14.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:13.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:14.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:13.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:14.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:13.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:14.006 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:13.725+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:14.356 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:14.214+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:14.215346+0000) 2024-04-27T03:22:14.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:14.144+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:14.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:14.144+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:14.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:14.144+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:14.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:14.144+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:14.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:14.244+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:14.244972+0000) 2024-04-27T03:22:14.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:14.404+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:14.405283+0000) 2024-04-27T03:22:14.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:14.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:14.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:14.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:14.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:14.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:14.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:14.364+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:14.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:22:14.682+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60328 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:22:14.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:14.680+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:14.681234+0000) 2024-04-27T03:22:14.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:14.679+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:14.680122+0000) 2024-04-27T03:22:14.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:14.679+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:14.680341+0000) 2024-04-27T03:22:14.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:14.722+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:14.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:14.722+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:14.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:14.722+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:14.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:14.722+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:15.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:15.149+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:15.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:15.149+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:15.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:15.149+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:15.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:15.149+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:15.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:15.244+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:15.245176+0000) 2024-04-27T03:22:15.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:15.404+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:15.405483+0000) 2024-04-27T03:22:15.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:15.214+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:15.215508+0000) 2024-04-27T03:22:15.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:15.348+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:15.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:15.348+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:15.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:15.348+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:15.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:15.349+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:15.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:15.682+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:15.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:15.682+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:15.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:15.682+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:15.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:15.682+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:16.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:16.154+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:16.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:16.154+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:16.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:16.154+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:16.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:16.155+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:16.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:16.246+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:16.247387+0000) 2024-04-27T03:22:16.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:16.407+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:16.408404+0000) 2024-04-27T03:22:16.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:16.219+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:16.220203+0000) 2024-04-27T03:22:16.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:16.391+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:16.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:16.391+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:16.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:16.391+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:16.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:16.392+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:16.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:16.646+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:16.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:16.646+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:16.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:16.646+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:16.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:16.647+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:17.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:17.127+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:17.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:17.127+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:17.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:17.127+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:17.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:17.128+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:17.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:17.250+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:17.251526+0000) 2024-04-27T03:22:17.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:17.411+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:17.412663+0000) 2024-04-27T03:22:17.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:17.224+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:17.224954+0000) 2024-04-27T03:22:17.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:17.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:17.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:17.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:17.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:17.390+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:17.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:17.391+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:17.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:17.623+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:17.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:17.623+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:17.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:17.623+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:17.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:17.623+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:18.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:18.108+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:18.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:18.108+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:18.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:18.108+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:18.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:18.110+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:18.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:18.415+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:18.415927+0000) 2024-04-27T03:22:18.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:18.229+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:18.229811+0000) 2024-04-27T03:22:18.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:18.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:18.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:18.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:18.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:18.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:18.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:18.432+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:18.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:18.253+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:18.254362+0000) 2024-04-27T03:22:18.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:18.636+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:18.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:18.636+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:18.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:18.636+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:18.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:18.636+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:19.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:19.103+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:19.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:19.103+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:19.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:19.103+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:19.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:19.104+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:19.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:19.230+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:19.231470+0000) 2024-04-27T03:22:19.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:19.417+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:19.417924+0000) 2024-04-27T03:22:19.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:19.255+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:19.256287+0000) 2024-04-27T03:22:19.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:22:19.689+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60369 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:22:19.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:19.687+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:19.688711+0000) 2024-04-27T03:22:19.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:19.677+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:19.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:19.677+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:19.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:19.677+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:19.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:19.677+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:19.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:19.686+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:19.687081+0000) 2024-04-27T03:22:19.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:19.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:19.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:19.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:19.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:19.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:19.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:19.465+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:19.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:19.688+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:19.688863+0000) 2024-04-27T03:22:20.384 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:20.230+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:20.231669+0000) 2024-04-27T03:22:20.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:20.119+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:20.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:20.119+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:20.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:20.119+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:20.385 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:20.119+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:20.385 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:20.255+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:20.256468+0000) 2024-04-27T03:22:20.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:20.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:20.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:20.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:20.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:20.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:20.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:20.453+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:20.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:20.417+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:20.418085+0000) 2024-04-27T03:22:20.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:20.705+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:20.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:20.705+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:20.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:20.705+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:20.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:20.705+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:21.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:21.087+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:21.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:21.087+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:21.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:21.087+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:21.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:21.087+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:21.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:21.419+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:21.420380+0000) 2024-04-27T03:22:21.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:21.257+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:21.258655+0000) 2024-04-27T03:22:21.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:21.233+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:21.233862+0000) 2024-04-27T03:22:21.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:21.404+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:21.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:21.404+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:21.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:21.404+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:21.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:21.405+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:21.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:21.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:21.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:21.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:21.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:21.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:21.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:21.710+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:22.266 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:22.237+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:22.238067+0000) 2024-04-27T03:22:22.266 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:22.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:22.267 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:22.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:22.267 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:22.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:22.267 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:22.099+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:22.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:22.259+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:22.260731+0000) 2024-04-27T03:22:22.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:22.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:22.672 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:22.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:22.672 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:22.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:22.672 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:22.419+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:22.672 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:22.422+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:22.423395+0000) 2024-04-27T03:22:22.672 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:22.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:22.672 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:22.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:22.672 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:22.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:22.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:22.665+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:23.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:23.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:23.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:23.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:23.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:23.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:23.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:23.094+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:23.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:23.262+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:23.263450+0000) 2024-04-27T03:22:23.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:23.424+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:23.425662+0000) 2024-04-27T03:22:23.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:23.239+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:23.240223+0000) 2024-04-27T03:22:23.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:23.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:23.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:23.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:23.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:23.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:23.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:23.447+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:23.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:23.707+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:23.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:23.707+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:23.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:23.707+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:23.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:23.707+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:24.270 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:24.240+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:24.241127+0000) 2024-04-27T03:22:24.270 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:24.058+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:24.270 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:24.058+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:24.271 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:24.058+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:24.271 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:24.059+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:24.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:24.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:24.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:24.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:24.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:24.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:24.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:24.478+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:24.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:24.426+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:24.427668+0000) 2024-04-27T03:22:24.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:24.263+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:24.264386+0000) 2024-04-27T03:22:24.951 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:22:24.697+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60407 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:22:24.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:24.695+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:24.696111+0000) 2024-04-27T03:22:24.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:24.695+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:24.696379+0000) 2024-04-27T03:22:24.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:24.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:24.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:24.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:24.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:24.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:24.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:24.710+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:24.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:24.695+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:24.696063+0000) 2024-04-27T03:22:25.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:25.029+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:25.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:25.029+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:25.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:25.029+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:25.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:25.029+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:25.456 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:25.427+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:25.427929+0000) 2024-04-27T03:22:25.456 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:25.263+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:25.264601+0000) 2024-04-27T03:22:25.456 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:25.240+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:25.241323+0000) 2024-04-27T03:22:25.457 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:25.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:25.457 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:25.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:25.457 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:25.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:25.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:25.450+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:26.007 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:25.728+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:26.007 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:25.728+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:26.007 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:25.728+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:26.007 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:25.728+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:26.272 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:26.244+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:26.245575+0000) 2024-04-27T03:22:26.272 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:26.004+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:26.273 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:26.004+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:26.273 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:26.004+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:26.273 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:26.004+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:26.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:26.265+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:26.266710+0000) 2024-04-27T03:22:26.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:26.431+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:26.432705+0000) 2024-04-27T03:22:26.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:26.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:26.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:26.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:26.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:26.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:26.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:26.495+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:27.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:26.768+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:27.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:26.768+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:27.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:26.768+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:27.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:26.769+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:27.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:27.049+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:27.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:27.049+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:27.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:27.049+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:27.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:27.247+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:27.247820+0000) 2024-04-27T03:22:27.385 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:27.050+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:27.385 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:27.268+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:27.269274+0000) 2024-04-27T03:22:27.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:27.435+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:27.435968+0000) 2024-04-27T03:22:27.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:27.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:27.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:27.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:27.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:27.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:27.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:27.487+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:28.086 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:27.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:28.086 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:27.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:28.086 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:27.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:28.086 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:27.794+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:28.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:28.249+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:28.249895+0000) 2024-04-27T03:22:28.385 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:28.079+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:28.385 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:28.079+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:28.385 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:28.079+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:28.385 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:28.080+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:28.385 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:28.273+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:28.274047+0000) 2024-04-27T03:22:28.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:28.437+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:28.438073+0000) 2024-04-27T03:22:28.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:28.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:28.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:28.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:28.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:28.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:28.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:28.446+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:29.100 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:28.833+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:29.101 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:28.833+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:29.101 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:28.833+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:29.101 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:28.833+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:29.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:29.249+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:29.250786+0000) 2024-04-27T03:22:29.385 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:29.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:29.385 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:29.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:29.386 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:29.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:29.386 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:29.094+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:29.386 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:29.274+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:29.274950+0000) 2024-04-27T03:22:29.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:29.438+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:29.439433+0000) 2024-04-27T03:22:29.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:29.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:29.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:29.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:29.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:29.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:29.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:29.486+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:29.955 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:22:29.704+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60448 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:22:29.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:29.703+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:29.703850+0000) 2024-04-27T03:22:29.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:29.703+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:29.703855+0000) 2024-04-27T03:22:29.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:29.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:29.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:29.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:29.957 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:29.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:29.957 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:29.863+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:29.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:29.703+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:29.704316+0000) 2024-04-27T03:22:30.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:30.053+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:30.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:30.053+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:30.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:30.053+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:30.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:30.054+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:30.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:30.438+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:30.439626+0000) 2024-04-27T03:22:30.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:30.274+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:30.275127+0000) 2024-04-27T03:22:30.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:30.250+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:30.250974+0000) 2024-04-27T03:22:30.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:30.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:30.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:30.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:30.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:30.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:30.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:30.464+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:31.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:31.058+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:31.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:31.058+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:31.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:31.058+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:31.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:31.060+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:31.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:30.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:31.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:30.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:31.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:30.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:31.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:30.845+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:31.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:31.252+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:31.253120+0000) 2024-04-27T03:22:31.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:31.278+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:31.279411+0000) 2024-04-27T03:22:31.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:31.441+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:31.442669+0000) 2024-04-27T03:22:31.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:31.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:31.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:31.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:31.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:31.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:31.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:31.450+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:32.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:31.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:32.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:31.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:32.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:31.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:32.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:31.889+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:32.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:32.027+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:32.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:32.027+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:32.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:32.027+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:32.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:32.028+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:32.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:32.255+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:32.255831+0000) 2024-04-27T03:22:32.434 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:32.281+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:32.282244+0000) 2024-04-27T03:22:32.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:32.444+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:32.445592+0000) 2024-04-27T03:22:32.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:32.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:32.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:32.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:32.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:32.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:32.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:32.428+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:33.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:32.992+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:33.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:32.992+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:33.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:32.992+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:33.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:32.993+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:33.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:32.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:33.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:32.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:33.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:32.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:33.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:32.867+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:33.432 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:33.285+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:33.286592+0000) 2024-04-27T03:22:33.433 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:33.259+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:33.260053+0000) 2024-04-27T03:22:33.433 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:33.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:33.433 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:33.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:33.433 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:33.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:33.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:33.447+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:33.447801+0000) 2024-04-27T03:22:33.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:33.426+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:34.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:34.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:34.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:34.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:34.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:34.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:34.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:34.027+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:34.090 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:33.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:34.090 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:33.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:34.090 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:33.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:34.090 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:33.866+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:34.386 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:34.260+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:34.261012+0000) 2024-04-27T03:22:34.386 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:34.286+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:34.287478+0000) 2024-04-27T03:22:34.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:34.448+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:34.449297+0000) 2024-04-27T03:22:34.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:34.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:34.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:34.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:34.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:34.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:34.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:34.455+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:34.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:22:34.712+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60486 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:22:34.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:34.709+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:34.709999+0000) 2024-04-27T03:22:34.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:34.709+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:34.710688+0000) 2024-04-27T03:22:34.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:34.709+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:34.710455+0000) 2024-04-27T03:22:34.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:34.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:34.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:34.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:34.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:34.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:34.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:34.868+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:35.267 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:35.008+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:35.267 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:35.009+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:35.267 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:35.009+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:35.267 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:35.010+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:35.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:35.448+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:35.449493+0000) 2024-04-27T03:22:35.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:35.286+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:35.287666+0000) 2024-04-27T03:22:35.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:35.260+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:35.261171+0000) 2024-04-27T03:22:35.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:35.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:35.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:35.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:35.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:35.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:35.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:35.435+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:36.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:35.965+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:36.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:35.965+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:36.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:35.965+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:36.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:35.965+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:36.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:35.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:36.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:35.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:36.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:35.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:36.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:35.889+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:36.457 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:36.288+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:36.289739+0000) 2024-04-27T03:22:36.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:36.262+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:36.263515+0000) 2024-04-27T03:22:36.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:36.451+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:36.451842+0000) 2024-04-27T03:22:36.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:36.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:36.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:36.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:36.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:36.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:36.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:36.462+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:37.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:36.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:37.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:36.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:37.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:36.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:37.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:36.901+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:37.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:36.997+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:37.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:36.997+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:37.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:36.997+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:37.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:36.999+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:37.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:37.265+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:37.266500+0000) 2024-04-27T03:22:37.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:37.292+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:37.292936+0000) 2024-04-27T03:22:37.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:37.453+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:37.454024+0000) 2024-04-27T03:22:37.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:37.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:37.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:37.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:37.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:37.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:37.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:37.472+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:38.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:37.892+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:38.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:37.892+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:38.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:37.892+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:38.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:37.892+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:38.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:37.976+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:38.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:37.976+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:38.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:37.976+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:38.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:37.978+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:38.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:38.267+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:38.268714+0000) 2024-04-27T03:22:38.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:38.294+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:38.295590+0000) 2024-04-27T03:22:38.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:38.457+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:38.458311+0000) 2024-04-27T03:22:38.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:38.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:38.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:38.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:38.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:38.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:38.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:38.444+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:39.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:38.896+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:39.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:38.896+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:39.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:38.896+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:39.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:38.896+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:39.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:38.960+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:39.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:38.960+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:39.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:38.960+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:39.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:38.961+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:39.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:39.268+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:39.269681+0000) 2024-04-27T03:22:39.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:39.296+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:39.297090+0000) 2024-04-27T03:22:39.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:39.458+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:39.459296+0000) 2024-04-27T03:22:39.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:39.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:39.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:39.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:39.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:39.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:39.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:39.494+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:39.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:22:39.719+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60527 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:22:39.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:39.717+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:39.717921+0000) 2024-04-27T03:22:39.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:39.717+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:39.718275+0000) 2024-04-27T03:22:39.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:39.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:39.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:39.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:39.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:39.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:39.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:39.912+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:39.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:39.716+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:39.717764+0000) 2024-04-27T03:22:39.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:39.919+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:39.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:39.919+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:39.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:39.919+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:39.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:39.920+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:40.546 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:40.269+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:40.269889+0000) 2024-04-27T03:22:40.546 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:40.458+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:40.459505+0000) 2024-04-27T03:22:40.546 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:40.296+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:40.297288+0000) 2024-04-27T03:22:40.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:40.539+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:40.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:40.539+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:40.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:40.539+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:40.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:40.539+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:41.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:40.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:41.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:40.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:41.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:40.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:41.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:40.958+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:41.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:40.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:41.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:40.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:41.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:40.953+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:41.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:40.954+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:41.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:41.271+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:41.272319+0000) 2024-04-27T03:22:41.537 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:41.460+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:41.461561+0000) 2024-04-27T03:22:41.537 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:41.300+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:41.301558+0000) 2024-04-27T03:22:41.957 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:41.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:41.957 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:41.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:41.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:41.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:41.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:41.531+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:42.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:41.950+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:42.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:41.950+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:42.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:41.950+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:42.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:41.951+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:42.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:41.959+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:42.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:41.959+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:42.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:41.959+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:42.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:41.960+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:42.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:42.273+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:42.274410+0000) 2024-04-27T03:22:42.581 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:42.302+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:42.303726+0000) 2024-04-27T03:22:42.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:42.463+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:42.464460+0000) 2024-04-27T03:22:42.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:42.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:42.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:42.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:42.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:42.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:42.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:42.575+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:43.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:42.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:43.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:42.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:43.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:42.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:43.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:42.957+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:43.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:42.953+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:43.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:42.953+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:43.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:42.953+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:43.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:42.954+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:43.591 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:43.305+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:43.305998+0000) 2024-04-27T03:22:43.592 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:43.465+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:43.466573+0000) 2024-04-27T03:22:43.592 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:43.275+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:43.276582+0000) 2024-04-27T03:22:43.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:43.585+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:43.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:43.585+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:43.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:43.585+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:43.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:43.586+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:44.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:43.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:44.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:43.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:44.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:43.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:44.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:43.994+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:44.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:43.980+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:44.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:43.980+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:44.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:43.980+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:44.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:43.981+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:44.601 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:44.277+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:44.278751+0000) 2024-04-27T03:22:44.602 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:44.467+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:44.468770+0000) 2024-04-27T03:22:44.602 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:44.306+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:44.306940+0000) 2024-04-27T03:22:44.853 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:22:44.726+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60565 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:22:44.853 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:44.723+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:44.724259+0000) 2024-04-27T03:22:44.853 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:44.724+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:44.725239+0000) 2024-04-27T03:22:44.853 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:44.594+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:44.854 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:44.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:44.854 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:44.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:44.854 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:44.595+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:44.854 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:44.724+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:44.725031+0000) 2024-04-27T03:22:45.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:44.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:45.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:44.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:45.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:44.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:45.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:44.953+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:45.161 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:45.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:45.161 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:45.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:45.161 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:45.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:45.161 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:45.018+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:45.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:45.278+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:45.278946+0000) 2024-04-27T03:22:45.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:45.306+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:45.307210+0000) 2024-04-27T03:22:45.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:45.468+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:45.468950+0000) 2024-04-27T03:22:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:45.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:45.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:45.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:45.575+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:46.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:46.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:46.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:46.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:46.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:46.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:46.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:46.039+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:46.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:45.931+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:46.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:45.931+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:46.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:45.931+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:46.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:45.932+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:46.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:46.280+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:46.281219+0000) 2024-04-27T03:22:46.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:46.308+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:46.309346+0000) 2024-04-27T03:22:46.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:46.470+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:46.471326+0000) 2024-04-27T03:22:46.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:46.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:46.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:46.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:46.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:46.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:46.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:46.625+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:47.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:46.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:47.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:46.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:47.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:46.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:47.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:46.995+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:47.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:46.918+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:47.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:46.918+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:47.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:46.918+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:47.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:46.919+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:47.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:47.310+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:47.311651+0000) 2024-04-27T03:22:47.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:47.283+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:47.283828+0000) 2024-04-27T03:22:47.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:47.472+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:47.473430+0000) 2024-04-27T03:22:47.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:47.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:47.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:47.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:47.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:47.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:47.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:47.584+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:48.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:48.034+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:48.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:48.034+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:48.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:48.034+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:48.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:48.035+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:48.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:47.917+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:48.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:47.917+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:48.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:47.917+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:48.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:47.918+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:48.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:48.286+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:48.286987+0000) 2024-04-27T03:22:48.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:48.313+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:48.313935+0000) 2024-04-27T03:22:48.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:48.475+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:48.476622+0000) 2024-04-27T03:22:48.939 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:48.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:48.939 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:48.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:48.939 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:48.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:48.939 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:48.606+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:49.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:48.931+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:49.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:48.931+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:49.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:48.931+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:49.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:48.932+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:49.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:49.071+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:49.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:49.071+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:49.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:49.071+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:49.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:49.072+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:49.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:49.314+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:49.315047+0000) 2024-04-27T03:22:49.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:49.287+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:49.287884+0000) 2024-04-27T03:22:49.734 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:49.478+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:49.478821+0000) 2024-04-27T03:22:49.735 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:49.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:49.735 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:49.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:49.735 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:49.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:49.735 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:49.635+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:50.010 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:22:49.734+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60606 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:22:50.010 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:49.731+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:49.732065+0000) 2024-04-27T03:22:50.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:49.731+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:49.731949+0000) 2024-04-27T03:22:50.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:49.731+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:49.732169+0000) 2024-04-27T03:22:50.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:49.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:50.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:49.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:50.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:49.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:50.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:49.890+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:50.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:50.095+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:50.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:50.095+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:50.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:50.095+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:50.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:50.095+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:50.611 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:50.314+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:50.315213+0000) 2024-04-27T03:22:50.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:50.478+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:50.479064+0000) 2024-04-27T03:22:50.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:50.287+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:50.288075+0000) 2024-04-27T03:22:50.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:50.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:50.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:50.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:50.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:50.604+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:50.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:50.606+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:51.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:51.116+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:51.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:51.116+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:51.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:51.116+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:51.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:51.117+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:51.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:50.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:51.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:50.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:51.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:50.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:51.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:50.926+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:51.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:51.291+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:51.292626+0000) 2024-04-27T03:22:51.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:51.319+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:51.319919+0000) 2024-04-27T03:22:51.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:51.482+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:51.483265+0000) 2024-04-27T03:22:51.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:51.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:51.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:51.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:51.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:51.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:51.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:51.626+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:52.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:52.123+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:52.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:52.123+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:52.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:52.123+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:52.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:52.124+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:52.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:51.905+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:52.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:51.905+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:52.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:51.905+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:52.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:51.907+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:52.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:52.294+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:52.294922+0000) 2024-04-27T03:22:52.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:52.321+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:52.322035+0000) 2024-04-27T03:22:52.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:52.669+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:52.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:52.669+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:52.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:52.669+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:52.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:52.670+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:52.940 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:52.483+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:52.485383+0000) 2024-04-27T03:22:53.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:52.932+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:53.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:52.932+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:53.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:52.932+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:53.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:52.934+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:53.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:53.138+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:53.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:53.138+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:53.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:53.138+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:53.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:53.138+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:53.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:53.296+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:53.298196+0000) 2024-04-27T03:22:53.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:53.322+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:53.324777+0000) 2024-04-27T03:22:53.821 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:53.485+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:53.487535+0000) 2024-04-27T03:22:53.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:53.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:53.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:53.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:53.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:53.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:53.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:53.634+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:54.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:53.922+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:54.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:53.922+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:54.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:53.922+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:54.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:53.923+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:54.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:54.297+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:54.299319+0000) 2024-04-27T03:22:54.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:54.324+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:54.326614+0000) 2024-04-27T03:22:54.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:54.164+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:54.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:54.164+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:54.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:54.164+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:54.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:54.165+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:54.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:54.487+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:54.489212+0000) 2024-04-27T03:22:54.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:54.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:54.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:54.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:54.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:54.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:54.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:54.596+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:55.010 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:22:54.741+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60644 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:22:55.011 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:54.739+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:54.741256+0000) 2024-04-27T03:22:55.011 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:54.738+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:54.739955+0000) 2024-04-27T03:22:55.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:54.738+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:54.740273+0000) 2024-04-27T03:22:55.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:54.921+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:55.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:54.921+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:55.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:54.921+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:55.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:54.921+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:55.305 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:55.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:55.305 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:55.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:55.305 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:55.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:55.305 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:55.161+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:55.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:55.297+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:55.299522+0000) 2024-04-27T03:22:55.637 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:55.325+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:55.326825+0000) 2024-04-27T03:22:55.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:55.487+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:55.489465+0000) 2024-04-27T03:22:55.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:55.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:55.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:55.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:55.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:55.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:55.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:55.629+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:56.161 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:56.122+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:56.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:56.122+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:56.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:56.122+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:56.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:56.122+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:56.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:55.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:56.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:55.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:56.162 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:55.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:56.163 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:55.893+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:56.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:56.300+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:56.302474+0000) 2024-04-27T03:22:56.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:56.327+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:56.329524+0000) 2024-04-27T03:22:56.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:56.490+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:56.492269+0000) 2024-04-27T03:22:56.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:56.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:56.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:56.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:56.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:56.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:56.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:56.591+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:57.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:56.916+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:57.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:56.916+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:57.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:56.916+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:57.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:56.917+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:57.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:57.130+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:57.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:57.130+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:57.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:57.130+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:57.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:57.130+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:57.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:57.330+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:57.332569+0000) 2024-04-27T03:22:57.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:57.302+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:57.304755+0000) 2024-04-27T03:22:57.886 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:57.493+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:57.495225+0000) 2024-04-27T03:22:57.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:57.603+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:57.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:57.603+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:57.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:57.603+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:57.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:57.604+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:57.886 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:57.878+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:57.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:57.878+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:58.161 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:57.878+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:58.161 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:57.879+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:58.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:58.305+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:58.307440+0000) 2024-04-27T03:22:58.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:58.333+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:58.335462+0000) 2024-04-27T03:22:58.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:58.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:58.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:58.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:58.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:58.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:58.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:58.168+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:58.936 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:58.496+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:58.497892+0000) 2024-04-27T03:22:58.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:58.645+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:58.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:58.645+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:58.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:58.645+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:58.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:58.647+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:59.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:58.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:59.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:58.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:59.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:58.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:22:59.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:58.929+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:59.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:59.307+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:59.309103+0000) 2024-04-27T03:22:59.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:59.335+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:59.336995+0000) 2024-04-27T03:22:59.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:59.213+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:59.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:59.213+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:59.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:59.213+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:22:59.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:59.213+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:59.755 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:59.497+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:59.499022+0000) 2024-04-27T03:22:59.755 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:22:59.745+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:59.746956+0000) 2024-04-27T03:22:59.755 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:59.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:59.755 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:59.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:59.755 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:59.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:22:59.755 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:59.609+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:22:59.755 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:22:59.744+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:59.746751+0000) 2024-04-27T03:22:59.756 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:59.744+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:22:59.746751+0000) 2024-04-27T03:23:00.011 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:22:59.749+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60685 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:23:00.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:59.881+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:00.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:59.881+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:00.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:59.881+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:00.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:22:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:22:59.881+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:00.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:00.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:00.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:00.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:00.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:00.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:00.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:00.202+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:00.611 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:00.335+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:00.337190+0000) 2024-04-27T03:23:00.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:00.497+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:00.499221+0000) 2024-04-27T03:23:00.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:00.307+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:00.309280+0000) 2024-04-27T03:23:00.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:00.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:00.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:00.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:00.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:00.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:00.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:00.635+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:00.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:00.910+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:00.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:00.910+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:00.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:00.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:01.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:00.911+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:01.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:01.309+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:01.311322+0000) 2024-04-27T03:23:01.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:01.338+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:01.340726+0000) 2024-04-27T03:23:01.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:01.197+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:01.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:01.197+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:01.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:01.197+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:01.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:01.198+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:01.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:01.500+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:01.502210+0000) 2024-04-27T03:23:01.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:01.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:01.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:01.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:01.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:01.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:01.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:01.635+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:02.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:01.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:02.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:01.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:02.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:01.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:02.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:01.939+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:02.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:02.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:02.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:02.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:02.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:02.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:02.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:02.156+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:02.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:02.311+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:02.313402+0000) 2024-04-27T03:23:02.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:02.343+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:02.345532+0000) 2024-04-27T03:23:02.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:02.502+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:02.504493+0000) 2024-04-27T03:23:02.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:02.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:02.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:02.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:02.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:02.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:02.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:02.633+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:03.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:02.964+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:03.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:02.964+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:03.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:02.964+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:03.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:02.965+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:03.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:03.176+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:03.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:03.176+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:03.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:03.176+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:03.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:03.177+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:03.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:03.507+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:03.509275+0000) 2024-04-27T03:23:03.637 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:03.316+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:03.317984+0000) 2024-04-27T03:23:03.637 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:03.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:03.637 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:03.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:03.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:03.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:03.638 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:03.348+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:03.350646+0000) 2024-04-27T03:23:03.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:03.630+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:04.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:03.932+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:04.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:03.932+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:04.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:03.932+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:04.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:03.934+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:04.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:04.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:04.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:04.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:04.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:04.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:04.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:04.147+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:04.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:04.317+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:04.319358+0000) 2024-04-27T03:23:04.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:04.351+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:04.352927+0000) 2024-04-27T03:23:04.771 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:23:04.757+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60723 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:23:04.772 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:04.755+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:04.755968+0000) 2024-04-27T03:23:04.772 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:04.509+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:04.511138+0000) 2024-04-27T03:23:04.772 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:04.755+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:04.756678+0000) 2024-04-27T03:23:04.772 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:04.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:04.773 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:04.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:04.773 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:04.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:04.773 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:04.598+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:04.773 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:04.755+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:04.756692+0000) 2024-04-27T03:23:05.092 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:04.948+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:05.092 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:04.948+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:05.092 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:04.948+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:05.092 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:04.949+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:05.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:05.318+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:05.319557+0000) 2024-04-27T03:23:05.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:05.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:05.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:05.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:05.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:05.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:05.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:05.179+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:05.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:05.510+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:05.511331+0000) 2024-04-27T03:23:05.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:05.352+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:05.353170+0000) 2024-04-27T03:23:05.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:05.580+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:05.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:05.580+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:05.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:05.580+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:05.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:05.581+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:06.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:05.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:06.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:05.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:06.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:05.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:06.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:05.953+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:06.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:06.320+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:06.321772+0000) 2024-04-27T03:23:06.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:06.354+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:06.355518+0000) 2024-04-27T03:23:06.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:06.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:06.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:06.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:06.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:06.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:06.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:06.216+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:06.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:06.513+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:06.514308+0000) 2024-04-27T03:23:06.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:06.615+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:06.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:06.615+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:06.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:06.615+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:06.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:06.617+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:07.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:06.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:07.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:06.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:07.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:06.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:07.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:06.927+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:07.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:07.323+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:07.324010+0000) 2024-04-27T03:23:07.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:07.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:07.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:07.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:07.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:07.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:07.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:07.185+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:07.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:07.356+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:07.357750+0000) 2024-04-27T03:23:07.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:07.516+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:07.517165+0000) 2024-04-27T03:23:07.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:07.599+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:07.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:07.599+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:07.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:07.600+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:07.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:07.600+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:08.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:07.976+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:08.333 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:07.976+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:08.333 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:07.976+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:08.333 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:07.977+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:08.333 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:08.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:08.333 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:08.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:08.333 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:08.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:08.333 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:08.202+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:08.597 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:08.326+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:08.326848+0000) 2024-04-27T03:23:08.598 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:08.362+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:08.362823+0000) 2024-04-27T03:23:08.598 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:08.519+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:08.520164+0000) 2024-04-27T03:23:08.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:08.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:08.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:08.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:08.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:08.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:08.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:08.591+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:09.334 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:09.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:09.334 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:09.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:09.334 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:09.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:09.334 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:09.023+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:09.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:09.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:09.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:09.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:09.335 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:09.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:09.335 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:09.159+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:09.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:09.364+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:09.365134+0000) 2024-04-27T03:23:09.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:09.520+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:09.521552+0000) 2024-04-27T03:23:09.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:09.327+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:09.328095+0000) 2024-04-27T03:23:09.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:09.555+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:09.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:09.555+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:09.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:09.555+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:09.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:09.557+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:10.030 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:23:09.767+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60764 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:23:10.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:09.762+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:09.763516+0000) 2024-04-27T03:23:10.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:09.762+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:09.762834+0000) 2024-04-27T03:23:10.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:10.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:10.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:10.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:10.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:09.763+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:09.763839+0000) 2024-04-27T03:23:10.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:10.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:10.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:10.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:10.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:10.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:10.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:10.193+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:10.334 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:10.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:10.334 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:10.025+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:10.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:10.520+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:10.521745+0000) 2024-04-27T03:23:10.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:10.364+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:10.365402+0000) 2024-04-27T03:23:10.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:10.327+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:10.328307+0000) 2024-04-27T03:23:10.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:10.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:10.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:10.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:10.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:10.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:10.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:10.529+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:11.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:11.210+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:11.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:11.210+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:11.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:11.210+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:11.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:11.211+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:11.337 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:10.993+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:11.337 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:10.993+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:11.337 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:10.993+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:11.337 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:10.994+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:11.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:11.523+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:11.524613+0000) 2024-04-27T03:23:11.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:11.366+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:11.367477+0000) 2024-04-27T03:23:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:11.330+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:11.330844+0000) 2024-04-27T03:23:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:11.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:11.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:11.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:11.491+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:12.258 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:12.015+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:12.258 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:12.015+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:12.258 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:12.015+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:12.258 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:12.016+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:12.512 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:12.368+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:12.369713+0000) 2024-04-27T03:23:12.512 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:12.332+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:12.333599+0000) 2024-04-27T03:23:12.512 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:12.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:12.512 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:12.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:12.512 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:12.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:12.513 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:12.251+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:12.513 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:12.251+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:12.513 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:12.251+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:12.513 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:12.251+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:12.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:12.506+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:12.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:12.527+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:12.527887+0000) 2024-04-27T03:23:13.163 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:12.973+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:13.163 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:12.973+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:13.163 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:12.973+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:13.163 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:12.974+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:13.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:13.335+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:13.336408+0000) 2024-04-27T03:23:13.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:13.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:13.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:13.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:13.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:13.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:13.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:13.256+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:13.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:13.256+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:13.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:13.256+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:13.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:13.256+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:13.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:13.374+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:13.374825+0000) 2024-04-27T03:23:13.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:13.529+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:13.530189+0000) 2024-04-27T03:23:13.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:13.458+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:14.093 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:13.943+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:14.093 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:13.943+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:14.093 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:13.943+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:14.093 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:13.944+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:14.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:14.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:14.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:14.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:14.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:14.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:14.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:14.218+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:14.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:14.376+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:14.377108+0000) 2024-04-27T03:23:14.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:14.531+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:14.532077+0000) 2024-04-27T03:23:14.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:14.337+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:14.338306+0000) 2024-04-27T03:23:14.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:14.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:14.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:14.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:14.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:14.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:14.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:14.447+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:15.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:14.771+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:14.771899+0000) 2024-04-27T03:23:15.093 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:23:14.774+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60802 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:23:15.094 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:14.772+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:14.773509+0000) 2024-04-27T03:23:15.094 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:14.772+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:14.772903+0000) 2024-04-27T03:23:15.094 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:14.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:15.094 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:14.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:15.094 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:14.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:15.094 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:14.975+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:15.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:15.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:15.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:15.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:15.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:15.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:15.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:15.173+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:15.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:15.531+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:15.532289+0000) 2024-04-27T03:23:15.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:15.376+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:15.377383+0000) 2024-04-27T03:23:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:15.337+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:15.338540+0000) 2024-04-27T03:23:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:15.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:15.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:15.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:15.433+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:16.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:16.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:16.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:16.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:16.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:16.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:16.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:16.017+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:16.540 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:16.378+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:16.379607+0000) 2024-04-27T03:23:16.540 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:16.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:16.540 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:16.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:16.540 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:16.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:16.540 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:16.221+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:16.541 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:16.340+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:16.340820+0000) 2024-04-27T03:23:16.541 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:16.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:16.541 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:16.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:16.541 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:16.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:16.541 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:16.402+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:16.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:16.533+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:16.534423+0000) 2024-04-27T03:23:17.349 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:16.972+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:17.350 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:16.972+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:17.350 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:16.972+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:17.350 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:16.973+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:17.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:17.197+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:17.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:17.197+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:17.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:17.197+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:17.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:17.198+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:17.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:17.535+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:17.536700+0000) 2024-04-27T03:23:17.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:17.381+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:17.382374+0000) 2024-04-27T03:23:17.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:17.343+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:17.344045+0000) 2024-04-27T03:23:17.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:17.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:17.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:17.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:17.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:17.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:17.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:17.354+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:18.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:18.007+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:18.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:18.007+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:18.333 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:18.007+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:18.333 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:18.008+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:18.333 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:18.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:18.333 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:18.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:18.333 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:18.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:18.333 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:18.203+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:18.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:18.538+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:18.538858+0000) 2024-04-27T03:23:18.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:18.383+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:18.384761+0000) 2024-04-27T03:23:18.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:18.325+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:18.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:18.325+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:18.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:18.325+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:18.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:18.327+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:18.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:18.346+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:18.346839+0000) 2024-04-27T03:23:19.354 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:19.012+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:19.354 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:19.012+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:19.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:19.012+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:19.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:19.013+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:19.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:19.223+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:19.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:19.223+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:19.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:19.223+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:19.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:19.223+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:19.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:19.385+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:19.386487+0000) 2024-04-27T03:23:19.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:19.539+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:19.539862+0000) 2024-04-27T03:23:19.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:19.347+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:19.348773+0000) 2024-04-27T03:23:19.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:19.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:19.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:19.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:19.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:19.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:19.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:19.364+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:20.094 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:23:19.782+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60843 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:23:20.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:19.778+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:19.779789+0000) 2024-04-27T03:23:20.094 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:19.778+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:19.779504+0000) 2024-04-27T03:23:20.095 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:19.779+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:19.779900+0000) 2024-04-27T03:23:20.095 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:19.992+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:20.095 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:19.992+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:20.095 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:19.992+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:20.095 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:19.993+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:20.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:20.226+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:20.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:20.226+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:20.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:20.226+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:20.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:20.226+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:20.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:20.385+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:20.386709+0000) 2024-04-27T03:23:20.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:20.539+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:20.540042+0000) 2024-04-27T03:23:20.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:20.348+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:20.348842+0000) 2024-04-27T03:23:20.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:20.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:20.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:20.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:20.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:20.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:20.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:20.356+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:21.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:21.185+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:21.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:21.185+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:21.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:21.185+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:21.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:21.186+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:21.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:21.031+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:21.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:21.031+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:21.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:21.031+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:21.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:21.031+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:21.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:21.387+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:21.388761+0000) 2024-04-27T03:23:21.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:21.544+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:21.544811+0000) 2024-04-27T03:23:21.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:21.350+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:21.351064+0000) 2024-04-27T03:23:21.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:21.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:21.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:21.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:21.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:21.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:21.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:21.380+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:22.338 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:22.172+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:22.339 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:22.172+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:22.339 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:22.172+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:22.339 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:22.173+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:22.339 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:22.018+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:22.339 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:22.018+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:22.339 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:22.018+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:22.339 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:22.019+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:22.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:22.390+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:22.390840+0000) 2024-04-27T03:23:22.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:22.548+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:22.549205+0000) 2024-04-27T03:23:22.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:22.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:22.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:22.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:22.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:22.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:22.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:22.333+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:22.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:22.352+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:22.353707+0000) 2024-04-27T03:23:23.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:23.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:23.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:23.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:23.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:23.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:23.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:23.214+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:23.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:23.006+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:23.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:23.006+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:23.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:23.006+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:23.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:23.006+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:23.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:23.553+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:23.553992+0000) 2024-04-27T03:23:23.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:23.392+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:23.392936+0000) 2024-04-27T03:23:23.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:23.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:23.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:23.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:23.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:23.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:23.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:23.346+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:23.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:23.354+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:23.355792+0000) 2024-04-27T03:23:24.094 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:23.962+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:24.094 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:23.962+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:24.094 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:23.962+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:24.094 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:23.963+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:24.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:24.244+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:24.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:24.244+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:24.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:24.244+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:24.364 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:24.245+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:24.364 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:24.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:24.364 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:24.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:24.364 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:24.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:24.364 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:24.313+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:24.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:24.393+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:24.393947+0000) 2024-04-27T03:23:24.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:24.356+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:24.357743+0000) 2024-04-27T03:23:24.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:24.554+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:24.555194+0000) 2024-04-27T03:23:25.004 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:23:24.789+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60881 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:23:25.004 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:24.787+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:24.788531+0000) 2024-04-27T03:23:25.005 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:24.787+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:24.787851+0000) 2024-04-27T03:23:25.005 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:24.786+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:24.787496+0000) 2024-04-27T03:23:25.005 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:24.970+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:25.005 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:24.970+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:25.005 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:24.970+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:25.005 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:24.971+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:25.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:25.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:25.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:25.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:25.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:25.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:25.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:25.206+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:25.561 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:25.393+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:25.394115+0000) 2024-04-27T03:23:25.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:25.273+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:25.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:25.273+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:25.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:25.273+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:25.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:25.274+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:25.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:25.357+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:25.357976+0000) 2024-04-27T03:23:25.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:25.554+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:25.555371+0000) 2024-04-27T03:23:26.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:26.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:26.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:26.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:26.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:26.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:26.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:26.183+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:26.212 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:25.951+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:26.212 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:25.951+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:26.212 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:25.951+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:26.212 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:25.952+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:26.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:26.398+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:26.399018+0000) 2024-04-27T03:23:26.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:26.315+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:26.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:26.315+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:26.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:26.315+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:26.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:26.315+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:26.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:26.362+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:26.362989+0000) 2024-04-27T03:23:26.941 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:26.557+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:26.558265+0000) 2024-04-27T03:23:27.211 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:26.934+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:27.212 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:26.934+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:27.212 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:26.934+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:27.212 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:26.935+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:27.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:27.403+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:27.404072+0000) 2024-04-27T03:23:27.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:27.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:27.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:27.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:27.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:27.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:27.480 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:27.216+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:27.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:27.285+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:27.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:27.285+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:27.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:27.285+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:27.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:27.286+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:27.480 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:27.367+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:27.367974+0000) 2024-04-27T03:23:27.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:27.559+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:27.560607+0000) 2024-04-27T03:23:28.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:28.175+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:28.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:28.175+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:28.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:28.175+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:28.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:28.175+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:28.261 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:27.976+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:28.261 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:27.976+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:28.261 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:27.976+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:28.261 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:27.977+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:28.568 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:28.408+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:28.408942+0000) 2024-04-27T03:23:28.568 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:28.284+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:28.568 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:28.284+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:28.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:28.284+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:28.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:28.284+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:28.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:28.372+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:28.373092+0000) 2024-04-27T03:23:28.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:28.562+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:28.562834+0000) 2024-04-27T03:23:29.318 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:29.147+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:29.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:29.147+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:29.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:29.147+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:29.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:29.148+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:29.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:29.006+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:29.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:29.006+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:29.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:29.006+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:29.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:29.007+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:29.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:29.410+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:29.411326+0000) 2024-04-27T03:23:29.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:29.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:29.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:29.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:29.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:29.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:29.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:29.313+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:29.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:29.373+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:29.374135+0000) 2024-04-27T03:23:29.822 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:23:29.797+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60922 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:23:29.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:29.794+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:29.795586+0000) 2024-04-27T03:23:29.823 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:29.563+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:29.564267+0000) 2024-04-27T03:23:29.823 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:29.794+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:29.795035+0000) 2024-04-27T03:23:29.823 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:29.793+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:29.793992+0000) 2024-04-27T03:23:30.094 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:30.003+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:30.095 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:30.003+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:30.095 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:30.003+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:30.095 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:30.004+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:30.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:30.326+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:30.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:30.326+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:30.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:30.326+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:30.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:30.327+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:30.380 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:30.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:30.380 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:30.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:30.381 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:30.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:30.381 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:30.185+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:30.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:30.563+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:30.564465+0000) 2024-04-27T03:23:30.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:30.373+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:30.374402+0000) 2024-04-27T03:23:30.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:30.410+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:30.411587+0000) 2024-04-27T03:23:31.331 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:31.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:31.332 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:31.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:31.332 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:31.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:31.332 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:31.155+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:31.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:30.970+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:31.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:30.970+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:31.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:30.970+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:31.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:30.970+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:31.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:31.413+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:31.413842+0000) 2024-04-27T03:23:31.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:31.568+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:31.569486+0000) 2024-04-27T03:23:31.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:31.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:31.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:31.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:31.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:31.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:31.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:31.340+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:31.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:31.377+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:31.378576+0000) 2024-04-27T03:23:32.313 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:32.112+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:32.314 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:32.112+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:32.314 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:32.112+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:32.314 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:32.112+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:32.314 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:32.009+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:32.314 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:32.009+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:32.314 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:32.009+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:32.314 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:32.010+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:32.580 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:32.415+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:32.416056+0000) 2024-04-27T03:23:32.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:32.307+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:32.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:32.307+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:32.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:32.307+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:32.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:32.308+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:32.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:32.382+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:32.382799+0000) 2024-04-27T03:23:32.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:32.573+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:32.574425+0000) 2024-04-27T03:23:33.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:33.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:33.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:33.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:33.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:33.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:33.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:33.083+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:33.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:32.982+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:33.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:32.982+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:33.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:32.982+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:33.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:32.983+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:33.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:33.578+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:33.579527+0000) 2024-04-27T03:23:33.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:33.417+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:33.418284+0000) 2024-04-27T03:23:33.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:33.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:33.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:33.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:33.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:33.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:33.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:33.336+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:33.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:33.384+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:33.385032+0000) 2024-04-27T03:23:34.095 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:34.047+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:34.095 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:34.047+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:34.095 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:34.047+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:34.095 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:34.047+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:34.096 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:33.953+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:34.096 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:33.953+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:34.096 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:33.953+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:34.096 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:33.954+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:34.391 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:34.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:34.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:34.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:34.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:34.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:34.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:34.301+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:34.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:34.579+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:34.580632+0000) 2024-04-27T03:23:34.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:34.419+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:34.420328+0000) 2024-04-27T03:23:34.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:34.385+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:34.386076+0000) 2024-04-27T03:23:35.062 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:23:34.804+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 60961 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:23:35.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:34.801+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:34.802226+0000) 2024-04-27T03:23:35.062 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:34.802+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:34.803074+0000) 2024-04-27T03:23:35.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:35.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:35.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:35.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:35.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:35.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:35.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:34.802+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:34.803191+0000) 2024-04-27T03:23:35.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:34.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:35.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:34.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:35.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:34.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:35.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:34.939+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:35.391 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:35.056+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:35.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:35.260+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:35.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:35.260+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:35.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:35.260+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:35.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:35.261+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:35.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:35.385+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:35.386267+0000) 2024-04-27T03:23:35.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:35.580+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:35.580840+0000) 2024-04-27T03:23:35.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:35.419+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:35.420516+0000) 2024-04-27T03:23:36.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:36.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:36.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:36.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:36.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:36.021+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:36.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:36.021+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:36.214 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:35.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:36.214 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:35.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:36.214 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:35.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:36.214 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:35.939+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:36.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:36.424+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:36.425499+0000) 2024-04-27T03:23:36.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:36.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:36.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:36.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:36.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:36.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:36.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:36.244+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:36.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:36.390+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:36.391020+0000) 2024-04-27T03:23:36.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:36.583+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:36.584096+0000) 2024-04-27T03:23:37.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:36.983+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:37.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:36.983+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:37.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:36.983+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:37.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:36.983+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:37.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:36.913+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:37.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:36.913+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:37.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:36.913+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:37.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:36.914+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:37.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:37.426+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:37.427747+0000) 2024-04-27T03:23:37.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:37.245+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:37.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:37.245+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:37.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:37.245+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:37.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:37.246+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:37.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:37.393+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:37.394135+0000) 2024-04-27T03:23:37.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:37.586+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:37.586843+0000) 2024-04-27T03:23:38.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:37.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:38.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:37.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:38.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:37.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:38.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:37.957+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:38.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:38.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:38.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:38.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:38.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:38.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:38.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:38.025+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:38.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:38.429+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:38.429982+0000) 2024-04-27T03:23:38.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:38.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:38.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:38.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:38.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:38.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:38.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:38.243+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:38.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:38.395+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:38.396321+0000) 2024-04-27T03:23:38.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:38.590+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:38.591113+0000) 2024-04-27T03:23:39.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:38.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:39.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:38.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:39.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:38.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:39.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:38.939+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:39.203 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:38.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:39.203 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:38.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:39.203 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:38.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:39.203 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:38.995+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:39.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:39.430+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:39.431124+0000) 2024-04-27T03:23:39.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:39.196+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:39.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:39.196+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:39.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:39.196+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:39.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:39.197+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:39.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:39.397+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:39.398323+0000) 2024-04-27T03:23:39.941 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:23:39.811+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61001 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:23:39.942 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:39.809+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:39.810583+0000) 2024-04-27T03:23:39.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:39.812+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:39.813166+0000) 2024-04-27T03:23:39.942 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:39.592+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:39.593499+0000) 2024-04-27T03:23:39.942 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:39.810+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:39.811058+0000) 2024-04-27T03:23:40.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:39.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:40.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:39.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:40.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:39.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:40.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:39.978+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:40.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:40.157+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:40.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:40.157+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:40.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:40.157+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:40.214 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:40.158+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:40.214 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:39.986+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:40.214 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:39.986+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:40.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:39.986+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:40.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:39.987+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:40.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:40.397+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:40.398505+0000) 2024-04-27T03:23:40.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:40.430+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:40.431391+0000) 2024-04-27T03:23:40.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:40.592+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:40.593748+0000) 2024-04-27T03:23:41.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:41.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:41.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:41.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:41.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:41.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:41.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:41.025+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:41.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:41.033+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:41.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:41.033+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:41.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:41.034+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:41.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:41.034+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:41.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:41.133+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:41.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:41.133+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:41.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:41.133+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:41.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:41.135+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:41.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:41.399+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:41.400600+0000) 2024-04-27T03:23:41.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:41.434+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:41.435685+0000) 2024-04-27T03:23:41.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:41.596+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:41.596932+0000) 2024-04-27T03:23:42.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:41.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:42.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:41.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:42.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:41.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:42.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:41.980+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:42.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:42.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:42.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:42.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:42.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:42.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:42.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:42.092+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:42.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:42.050+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:42.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:42.050+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:42.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:42.050+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:42.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:42.051+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:42.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:42.403+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:42.404467+0000) 2024-04-27T03:23:42.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:42.600+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:42.601144+0000) 2024-04-27T03:23:42.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:42.436+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:42.437779+0000) 2024-04-27T03:23:43.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:43.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:43.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:43.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:43.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:43.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:43.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:43.047+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:43.394 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:43.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:43.394 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:43.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:43.394 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:43.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:43.394 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:43.011+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:43.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:43.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:43.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:43.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:43.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:43.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:43.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:43.043+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:43.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:43.406+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:43.407398+0000) 2024-04-27T03:23:43.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:43.439+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:43.440438+0000) 2024-04-27T03:23:43.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:43.603+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:43.604194+0000) 2024-04-27T03:23:44.096 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:44.045+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:44.096 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:44.045+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:44.096 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:44.045+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:44.096 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:44.046+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:44.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:43.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:44.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:43.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:44.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:43.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:44.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:43.992+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:44.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:44.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:44.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:44.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:44.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:44.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:44.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:44.094+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:44.407+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:44.408471+0000) 2024-04-27T03:23:44.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:44.441+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:44.442060+0000) 2024-04-27T03:23:44.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:44.604+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:44.605706+0000) 2024-04-27T03:23:45.078 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:23:44.819+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61036 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:23:45.079 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:44.816+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:44.817152+0000) 2024-04-27T03:23:45.079 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:44.816+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:44.817452+0000) 2024-04-27T03:23:45.079 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:45.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:45.079 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:45.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:45.079 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:45.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:45.079 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:45.033+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:45.080 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:44.818+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:44.819014+0000) 2024-04-27T03:23:45.080 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:45.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:45.080 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:45.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:45.080 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:45.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:45.080 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:45.043+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:45.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:45.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:45.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:45.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:45.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:45.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:45.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:45.073+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:45.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:45.441+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:45.442250+0000) 2024-04-27T03:23:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:45.407+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:45.408662+0000) 2024-04-27T03:23:45.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:45.605+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:45.605892+0000) 2024-04-27T03:23:46.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:46.121+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:46.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:46.121+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:46.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:46.121+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:46.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:46.122+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:46.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:46.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:46.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:46.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:46.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:46.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:46.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:46.006+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:46.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:46.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:46.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:46.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:46.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:46.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:46.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:46.016+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:46.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:46.409+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:46.410760+0000) 2024-04-27T03:23:46.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:46.444+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:46.445247+0000) 2024-04-27T03:23:46.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:46.607+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:46.608523+0000) 2024-04-27T03:23:46.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:46.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:46.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:46.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:46.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:46.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:46.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:46.966+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:46.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:46.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:46.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:46.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:46.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:46.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:46.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:46.969+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:47.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:47.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:47.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:47.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:47.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:47.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:47.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:47.133+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:47.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:47.412+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:47.412893+0000) 2024-04-27T03:23:47.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:47.611+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:47.612753+0000) 2024-04-27T03:23:47.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:47.446+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:47.447337+0000) 2024-04-27T03:23:48.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:47.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:48.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:47.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:48.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:47.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:48.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:47.959+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:48.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:48.169+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:48.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:48.169+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:48.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:48.169+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:48.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:48.169+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:48.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:48.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:48.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:48.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:48.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:48.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:48.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:48.001+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:48.414+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:48.415415+0000) 2024-04-27T03:23:48.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:48.615+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:48.616058+0000) 2024-04-27T03:23:48.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:48.450+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:48.451696+0000) 2024-04-27T03:23:49.214 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:48.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:49.214 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:48.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:49.214 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:48.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:49.214 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:48.957+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:49.214 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:48.958+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:49.214 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:48.958+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:49.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:48.958+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:49.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:48.959+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:49.215 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:49.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:49.215 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:49.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:49.215 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:49.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:49.215 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:49.165+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:49.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:49.415+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:49.416721+0000) 2024-04-27T03:23:49.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:49.453+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:49.453843+0000) 2024-04-27T03:23:49.925 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:23:49.827+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61076 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:23:49.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:49.825+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:49.826146+0000) 2024-04-27T03:23:49.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:49.825+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:49.826347+0000) 2024-04-27T03:23:49.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:49.616+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:49.617145+0000) 2024-04-27T03:23:49.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:49.825+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:49.826309+0000) 2024-04-27T03:23:49.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:49.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:49.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:49.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:49.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:49.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:50.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:49.919+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:50.183 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:49.962+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:50.183 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:49.962+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:50.183 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:49.962+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:50.183 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:49.963+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:50.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:50.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:50.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:50.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:50.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:50.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:50.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:50.177+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:50.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:50.416+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:50.416897+0000) 2024-04-27T03:23:50.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:50.453+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:50.454022+0000) 2024-04-27T03:23:50.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:50.616+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:50.617406+0000) 2024-04-27T03:23:51.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:51.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:51.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:51.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:51.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:51.161+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:51.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:51.162+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:51.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:50.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:51.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:50.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:51.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:50.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:51.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:50.978+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:51.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:50.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:51.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:50.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:51.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:50.958+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:51.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:50.959+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:51.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:51.456+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:51.456876+0000) 2024-04-27T03:23:51.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:51.620+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:51.621792+0000) 2024-04-27T03:23:51.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:51.418+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:51.419505+0000) 2024-04-27T03:23:52.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:52.019+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:52.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:52.019+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:52.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:52.019+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:52.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:52.020+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:52.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:51.919+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:52.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:51.919+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:52.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:51.919+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:52.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:51.920+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:52.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:52.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:52.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:52.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:52.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:52.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:52.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:52.186+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:52.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:52.420+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:52.421588+0000) 2024-04-27T03:23:52.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:52.623+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:52.624524+0000) 2024-04-27T03:23:52.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:52.458+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:52.459258+0000) 2024-04-27T03:23:53.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:52.896+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:53.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:52.896+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:53.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:52.896+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:53.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:52.897+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:53.098 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:53.034+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:53.098 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:53.034+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:53.098 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:53.034+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:53.098 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:53.035+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:53.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:53.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:53.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:53.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:53.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:53.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:53.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:53.186+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:53.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:53.423+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:53.423827+0000) 2024-04-27T03:23:53.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:53.625+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:53.626698+0000) 2024-04-27T03:23:53.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:53.462+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:53.463604+0000) 2024-04-27T03:23:54.090 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:53.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:54.091 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:53.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:54.091 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:53.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:54.091 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:53.945+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:54.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:54.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:54.395 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:54.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:54.395 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:54.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:54.395 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:54.137+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:54.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:54.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:54.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:54.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:54.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:54.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:54.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:54.085+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:54.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:54.423+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:54.424884+0000) 2024-04-27T03:23:54.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:54.462+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:54.464760+0000) 2024-04-27T03:23:54.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:54.626+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:54.628226+0000) 2024-04-27T03:23:55.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:54.831+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:54.833769+0000) 2024-04-27T03:23:55.097 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:54.831+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:54.833467+0000) 2024-04-27T03:23:55.098 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:23:54.834+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61115 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:23:55.098 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:54.831+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:54.833296+0000) 2024-04-27T03:23:55.098 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:54.903+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:55.098 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:54.903+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:55.098 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:54.903+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:55.098 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:54.903+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:55.350 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:55.140+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:55.350 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:55.140+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:55.350 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:55.140+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:55.350 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:55.141+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:55.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:55.130+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:55.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:55.130+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:55.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:55.130+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:55.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:55.131+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:55.633 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:55.423+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:55.425134+0000) 2024-04-27T03:23:55.634 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:55.463+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:55.464934+0000) 2024-04-27T03:23:55.907 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:55.626+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:55.628389+0000) 2024-04-27T03:23:56.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:56.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:56.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:56.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:56.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:56.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:56.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:56.142+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:56.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:55.925+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:56.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:55.925+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:56.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:55.925+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:56.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:55.925+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:56.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:56.154+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:56.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:56.154+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:56.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:56.154+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:56.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:56.155+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:56.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:56.631+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:56.633314+0000) 2024-04-27T03:23:56.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:56.428+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:56.430204+0000) 2024-04-27T03:23:56.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:56.466+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:56.468178+0000) 2024-04-27T03:23:57.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:56.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:57.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:56.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:57.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:56.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:57.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:56.900+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:57.164 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:57.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:57.164 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:57.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:57.164 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:57.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:57.164 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:57.136+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:57.440 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:57.155+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:57.440 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:57.155+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:57.440 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:57.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:57.441 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:57.157+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:57.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:57.433+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:57.435118+0000) 2024-04-27T03:23:57.730 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:57.470+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:57.472188+0000) 2024-04-27T03:23:57.730 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:57.636+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:57.638336+0000) 2024-04-27T03:23:58.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:58.089+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:58.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:58.089+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:58.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:58.089+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:58.153 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:58.090+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:58.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:57.892+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:58.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:57.892+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:58.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:57.892+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:58.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:57.892+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:58.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:58.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:58.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:58.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:58.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:58.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:58.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:58.146+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:58.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:58.438+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:58.440143+0000) 2024-04-27T03:23:58.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:58.641+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:58.643502+0000) 2024-04-27T03:23:58.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:58.474+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:58.476568+0000) 2024-04-27T03:23:59.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:58.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:59.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:58.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:59.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:58.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:23:59.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:58.930+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:59.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:59.127+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:59.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:59.127+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:59.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:59.127+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:23:59.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:59.127+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:59.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:59.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:59.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:59.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:59.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:59.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:23:59.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:59.185+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:23:59.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:59.439+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:59.441306+0000) 2024-04-27T03:23:59.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:59.476+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:59.477820+0000) 2024-04-27T03:23:59.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:59.643+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:59.645367+0000) 2024-04-27T03:24:00.098 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:23:59.841+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61155 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:24:00.098 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:23:59.840+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:59.842166+0000) 2024-04-27T03:24:00.098 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:23:59.837+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:59.839130+0000) 2024-04-27T03:24:00.099 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:59.838+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:23:59.840366+0000) 2024-04-27T03:24:00.099 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:59.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:00.099 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:59.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:00.099 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:59.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:00.099 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:23:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:23:59.884+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:00.395 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:00.202+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:00.395 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:00.202+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:00.395 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:00.202+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:00.395 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:00.202+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:00.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:00.121+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:00.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:00.121+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:00.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:00.121+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:00.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:00.122+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:00.651 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:00.439+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:00.441505+0000) 2024-04-27T03:24:00.651 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:00.476+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:00.478030+0000) 2024-04-27T03:24:00.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:00.643+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:00.645568+0000) 2024-04-27T03:24:00.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:00.922+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:00.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:00.922+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:00.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:00.922+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:01.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:00.923+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:01.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:01.082+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:01.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:01.082+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:01.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:01.082+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:01.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:01.083+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:01.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:01.237+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:01.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:01.237+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:01.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:01.237+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:01.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:01.238+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:01.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:01.443+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:01.445678+0000) 2024-04-27T03:24:01.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:01.478+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:01.480307+0000) 2024-04-27T03:24:01.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:01.646+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:01.647846+0000) 2024-04-27T03:24:02.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:01.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:02.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:01.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:02.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:01.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:02.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:01.951+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:02.215 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:02.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:02.216 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:02.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:02.216 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:02.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:02.216 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:02.194+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:02.216 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:02.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:02.216 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:02.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:02.216 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:02.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:02.216 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:02.093+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:02.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:02.448+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:02.450380+0000) 2024-04-27T03:24:02.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:02.480+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:02.482569+0000) 2024-04-27T03:24:02.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:02.649+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:02.651052+0000) 2024-04-27T03:24:03.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:02.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:03.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:02.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:03.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:02.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:03.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:02.995+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:03.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:03.200+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:03.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:03.200+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:03.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:03.200+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:03.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:03.201+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:03.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:03.051+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:03.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:03.051+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:03.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:03.051+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:03.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:03.053+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:03.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:03.451+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:03.453327+0000) 2024-04-27T03:24:03.659 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:03.483+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:03.484836+0000) 2024-04-27T03:24:03.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:03.652+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:03.653850+0000) 2024-04-27T03:24:04.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:04.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:04.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:04.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:04.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:04.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:04.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:04.025+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:04.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:04.218+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:04.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:04.218+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:04.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:04.218+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:04.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:04.218+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:04.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:04.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:04.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:04.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:04.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:04.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:04.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:04.025+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:04.660 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:04.452+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:04.454730+0000) 2024-04-27T03:24:04.660 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:04.484+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:04.486056+0000) 2024-04-27T03:24:04.975 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:24:04.848+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61194 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:24:04.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:04.846+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:04.848601+0000) 2024-04-27T03:24:04.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:04.846+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:04.848467+0000) 2024-04-27T03:24:04.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:04.653+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:04.654965+0000) 2024-04-27T03:24:04.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:04.845+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:04.847497+0000) 2024-04-27T03:24:05.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:05.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:05.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:05.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:05.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:05.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:05.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:05.055+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:05.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:04.993+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:05.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:04.993+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:05.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:04.993+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:05.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:04.995+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:05.660 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:05.484+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:05.486239+0000) 2024-04-27T03:24:05.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:05.249+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:05.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:05.249+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:05.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:05.249+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:05.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:05.249+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:05.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:05.453+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:05.454932+0000) 2024-04-27T03:24:05.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:05.653+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:05.655173+0000) 2024-04-27T03:24:06.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:06.039+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:06.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:06.039+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:06.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:06.039+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:06.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:06.039+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:06.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:06.208+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:06.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:06.208+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:06.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:06.208+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:06.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:06.210+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:06.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:06.076+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:06.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:06.076+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:06.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:06.076+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:06.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:06.076+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:06.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:06.456+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:06.457643+0000) 2024-04-27T03:24:06.663 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:06.487+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:06.488951+0000) 2024-04-27T03:24:06.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:06.656+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:06.657595+0000) 2024-04-27T03:24:07.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:07.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:07.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:07.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:07.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:07.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:07.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:07.165+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:07.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:07.007+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:07.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:07.007+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:07.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:07.007+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:07.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:07.008+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:07.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:07.123+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:07.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:07.123+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:07.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:07.123+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:07.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:07.124+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:07.666 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:07.460+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:07.461188+0000) 2024-04-27T03:24:07.666 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:07.491+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:07.492091+0000) 2024-04-27T03:24:07.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:07.660+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:07.660883+0000) 2024-04-27T03:24:08.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:08.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:08.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:08.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:08.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:08.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:08.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:08.180+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:08.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:08.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:08.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:08.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:08.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:08.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:08.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:08.170+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:08.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:08.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:08.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:08.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:08.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:08.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:08.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:08.042+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:08.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:08.463+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:08.464049+0000) 2024-04-27T03:24:08.669 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:08.495+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:08.496426+0000) 2024-04-27T03:24:08.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:08.662+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:08.663654+0000) 2024-04-27T03:24:09.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:09.174+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:09.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:09.174+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:09.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:09.174+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:09.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:09.175+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:09.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:09.219+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:09.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:09.219+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:09.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:09.219+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:09.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:09.220+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:09.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:09.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:09.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:09.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:09.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:09.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:09.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:09.060+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:09.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:09.464+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:09.465661+0000) 2024-04-27T03:24:09.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:09.496+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:09.497757+0000) 2024-04-27T03:24:09.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:09.854+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:09.855442+0000) 2024-04-27T03:24:09.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:24:09.856+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61234 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:24:09.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:09.854+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:09.855152+0000) 2024-04-27T03:24:09.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:09.664+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:09.665432+0000) 2024-04-27T03:24:09.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:09.854+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:09.855140+0000) 2024-04-27T03:24:10.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:10.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:10.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:10.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:10.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:10.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:10.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:10.221+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:10.283 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:10.171+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:10.283 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:10.171+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:10.283 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:10.171+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:10.283 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:10.172+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:10.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:10.085+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:10.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:10.085+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:10.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:10.085+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:10.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:10.086+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:10.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:10.465+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:10.465922+0000) 2024-04-27T03:24:10.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:10.497+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:10.497939+0000) 2024-04-27T03:24:10.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:10.664+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:10.665629+0000) 2024-04-27T03:24:11.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:11.213+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:11.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:11.213+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:11.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:11.213+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:11.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:11.215+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:11.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:11.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:11.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:11.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:11.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:11.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:11.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:11.188+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:11.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:11.089+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:11.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:11.089+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:11.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:11.089+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:11.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:11.090+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:11.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:11.467+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:11.468091+0000) 2024-04-27T03:24:11.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:11.499+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:11.499955+0000) 2024-04-27T03:24:11.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:11.667+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:11.667906+0000) 2024-04-27T03:24:12.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:12.229+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:12.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:12.229+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:12.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:12.229+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:12.397 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:12.230+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:12.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:12.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:12.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:12.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:12.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:12.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:12.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:12.082+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:12.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:12.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:12.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:12.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:12.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:12.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:12.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:12.179+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:12.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:12.469+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:12.470152+0000) 2024-04-27T03:24:12.676 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:12.501+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:12.502034+0000) 2024-04-27T03:24:12.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:12.669+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:12.670082+0000) 2024-04-27T03:24:13.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:13.119+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:13.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:13.119+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:13.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:13.119+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:13.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:13.120+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:13.397 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:13.246+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:13.397 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:13.246+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:13.397 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:13.246+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:13.397 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:13.248+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:13.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:13.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:13.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:13.204+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:13.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:13.204+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:13.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:13.204+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:13.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:13.472+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:13.473208+0000) 2024-04-27T03:24:13.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:13.503+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:13.504110+0000) 2024-04-27T03:24:13.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:13.672+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:13.673127+0000) 2024-04-27T03:24:14.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:14.260+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:14.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:14.260+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:14.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:14.260+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:14.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:14.261+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:14.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:14.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:14.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:14.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:14.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:14.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:14.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:14.208+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:14.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:14.147+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:14.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:14.147+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:14.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:14.147+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:14.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:14.148+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:14.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:14.474+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:14.475356+0000) 2024-04-27T03:24:14.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:14.505+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:14.505881+0000) 2024-04-27T03:24:14.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:14.861+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:14.862240+0000) 2024-04-27T03:24:14.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:24:14.863+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61273 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:24:14.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:14.861+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:14.861874+0000) 2024-04-27T03:24:14.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:14.673+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:14.674421+0000) 2024-04-27T03:24:14.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:14.861+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:14.861812+0000) 2024-04-27T03:24:15.242 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:15.227+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:15.242 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:15.227+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:15.242 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:15.227+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:15.242 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:15.228+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:15.243 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:15.177+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:15.243 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:15.177+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:15.243 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:15.177+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:15.243 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:15.178+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:15.511 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:15.474+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:15.475519+0000) 2024-04-27T03:24:15.511 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:15.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:15.512 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:15.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:15.512 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:15.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:15.512 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:15.236+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:15.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:15.505+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:15.506134+0000) 2024-04-27T03:24:15.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:15.673+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:15.674609+0000) 2024-04-27T03:24:16.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:16.151+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:16.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:16.151+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:16.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:16.151+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:16.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:16.152+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:16.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:16.249+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:16.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:16.249+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:16.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:16.249+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:16.279 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:16.250+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:16.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:16.476+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:16.477610+0000) 2024-04-27T03:24:16.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:16.508+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:16.509065+0000) 2024-04-27T03:24:16.682 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:16.271+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:16.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:16.271+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:16.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:16.271+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:16.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:16.272+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:16.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:16.675+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:16.676764+0000) 2024-04-27T03:24:17.281 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:17.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:17.281 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:17.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:17.281 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:17.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:17.281 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:17.212+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:17.281 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:17.172+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:17.281 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:17.172+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:17.282 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:17.172+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:17.282 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:17.173+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:17.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:17.479+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:17.479893+0000) 2024-04-27T03:24:17.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:17.511+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:17.512327+0000) 2024-04-27T03:24:17.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:17.274+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:17.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:17.274+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:17.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:17.274+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:17.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:17.275+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:17.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:17.678+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:17.679077+0000) 2024-04-27T03:24:18.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:18.185+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:18.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:18.185+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:18.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:18.185+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:18.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:18.185+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:18.305 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:18.191+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:18.305 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:18.191+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:18.305 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:18.191+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:18.305 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:18.192+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:18.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:18.481+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:18.482014+0000) 2024-04-27T03:24:18.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:18.514+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:18.515263+0000) 2024-04-27T03:24:18.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:18.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:18.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:18.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:18.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:18.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:18.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:18.298+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:18.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:18.680+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:18.681379+0000) 2024-04-27T03:24:19.314 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:19.195+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:19.314 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:19.195+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:19.314 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:19.195+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:19.314 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:19.196+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:19.314 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:19.214+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:19.314 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:19.214+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:19.314 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:19.214+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:19.315 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:19.215+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:19.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:19.482+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:19.483597+0000) 2024-04-27T03:24:19.689 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:19.516+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:19.517234+0000) 2024-04-27T03:24:19.689 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:19.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:19.689 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:19.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:19.689 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:19.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:19.689 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:19.308+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:19.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:24:19.871+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61313 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:24:19.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:19.869+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:19.869846+0000) 2024-04-27T03:24:19.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:19.868+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:19.869080+0000) 2024-04-27T03:24:19.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:19.682+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:19.683195+0000) 2024-04-27T03:24:19.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:19.868+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:19.868886+0000) 2024-04-27T03:24:20.283 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:20.208+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:20.283 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:20.208+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:20.283 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:20.208+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:20.284 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:20.209+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:20.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:20.198+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:20.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:20.198+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:20.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:20.198+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:20.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:20.199+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:20.567 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:20.482+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:20.483784+0000) 2024-04-27T03:24:20.568 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:20.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:20.568 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:20.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:20.568 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:20.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:20.568 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:20.352+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:20.568 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:20.516+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:20.517466+0000) 2024-04-27T03:24:20.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:20.682+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:20.683366+0000) 2024-04-27T03:24:21.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:21.229+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:21.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:21.229+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:21.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:21.229+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:21.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:21.229+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:21.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:21.243+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:21.346 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:21.243+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:21.346 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:21.243+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:21.346 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:21.244+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:21.346 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:21.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:21.346 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:21.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:21.346 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:21.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:21.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:21.339+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:21.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:21.487+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:21.488004+0000) 2024-04-27T03:24:21.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:21.520+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:21.520829+0000) 2024-04-27T03:24:21.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:21.684+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:21.685720+0000) 2024-04-27T03:24:22.395 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:22.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:22.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:22.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:22.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:22.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:22.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:22.242+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:22.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:22.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:22.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:22.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:22.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:22.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:22.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:22.262+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:22.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:22.489+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:22.490132+0000) 2024-04-27T03:24:22.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:22.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:22.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:22.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:22.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:22.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:22.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:22.389+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:22.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:22.524+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:22.525620+0000) 2024-04-27T03:24:22.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:22.689+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:22.689929+0000) 2024-04-27T03:24:23.292 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:23.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:23.292 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:23.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:23.292 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:23.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:23.292 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:23.248+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:23.556 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:23.529+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:23.530421+0000) 2024-04-27T03:24:23.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:23.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:23.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:23.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:23.557 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:23.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:23.557 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:23.343+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:23.557 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:23.285+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:23.557 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:23.285+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:23.557 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:23.285+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:23.557 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:23.286+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:23.557 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:23.491+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:23.492313+0000) 2024-04-27T03:24:23.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:23.691+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:23.692309+0000) 2024-04-27T03:24:24.537 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:24.281+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:24.537 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:24.281+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:24.538 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:24.281+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:24.538 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:24.283+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:24.538 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:24.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:24.538 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:24.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:24.538 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:24.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:24.538 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:24.358+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:24.538 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:24.270+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:24.538 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:24.270+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:24.539 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:24.270+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:24.539 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:24.271+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:24.539 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:24.492+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:24.493466+0000) 2024-04-27T03:24:24.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:24.531+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:24.532124+0000) 2024-04-27T03:24:24.879 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:24.692+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:24.693609+0000) 2024-04-27T03:24:25.170 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:24:24.878+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61352 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:24:25.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:24.876+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:24.876885+0000) 2024-04-27T03:24:25.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:24.875+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:24.876638+0000) 2024-04-27T03:24:25.171 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:24.875+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:24.876741+0000) 2024-04-27T03:24:25.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:25.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:25.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:25.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:25.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:25.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:25.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:25.364+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:25.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:25.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:25.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:25.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:25.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:25.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:25.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:25.277+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:25.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:25.319+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:25.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:25.319+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:25.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:25.319+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:25.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:25.320+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:25.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:25.492+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:25.493729+0000) 2024-04-27T03:24:25.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:25.531+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:25.532334+0000) 2024-04-27T03:24:25.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:25.692+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:25.693784+0000) 2024-04-27T03:24:26.372 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:26.231+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:26.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:26.231+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:26.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:26.231+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:26.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:26.231+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:26.627 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:26.495+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:26.495811+0000) 2024-04-27T03:24:26.628 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:26.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:26.628 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:26.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:26.628 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:26.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:26.628 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:26.407+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:26.628 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:26.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:26.628 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:26.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:26.628 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:26.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:26.628 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:26.367+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:26.628 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:26.535+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:26.536635+0000) 2024-04-27T03:24:26.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:26.695+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:26.696433+0000) 2024-04-27T03:24:27.504 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:27.399+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:27.504 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:27.399+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:27.504 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:27.399+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:27.504 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:27.399+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:27.504 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:27.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:27.504 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:27.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:27.505 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:27.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:27.505 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:27.415+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:27.505 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:27.232+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:27.505 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:27.232+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:27.505 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:27.232+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:27.505 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:27.233+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:27.505 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:27.497+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:27.498493+0000) 2024-04-27T03:24:27.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:27.699+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:27.700660+0000) 2024-04-27T03:24:27.979 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:27.537+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:27.538775+0000) 2024-04-27T03:24:28.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:28.229+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:28.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:28.229+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:28.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:28.229+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:28.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:28.231+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:28.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:28.500+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:28.501669+0000) 2024-04-27T03:24:28.709 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:28.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:28.709 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:28.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:28.709 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:28.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:28.709 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:28.395+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:28.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:28.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:28.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:28.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:28.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:28.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:28.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:28.367+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:28.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:28.542+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:28.543114+0000) 2024-04-27T03:24:28.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:28.702+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:28.703110+0000) 2024-04-27T03:24:29.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:29.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:29.360 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:29.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:29.360 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:29.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:29.360 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:29.217+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:29.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:29.502+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:29.502844+0000) 2024-04-27T03:24:29.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:29.353+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:29.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:29.353+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:29.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:29.353+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:29.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:29.354+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:29.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:29.543+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:29.544270+0000) 2024-04-27T03:24:29.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:29.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:29.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:29.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:29.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:29.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:29.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:29.364+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:29.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:24:29.885+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61392 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:24:29.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:29.882+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:29.883669+0000) 2024-04-27T03:24:29.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:29.882+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:29.883613+0000) 2024-04-27T03:24:29.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:29.703+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:29.704635+0000) 2024-04-27T03:24:29.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:29.883+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:29.883971+0000) 2024-04-27T03:24:30.284 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:30.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:30.284 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:30.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:30.284 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:30.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:30.284 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:30.212+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:30.549 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:30.502+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:30.503024+0000) 2024-04-27T03:24:30.550 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:30.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:30.550 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:30.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:30.550 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:30.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:30.550 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:30.367+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:30.550 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:30.339+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:30.550 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:30.339+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:30.551 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:30.339+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:30.551 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:30.340+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:30.551 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:30.543+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:30.544464+0000) 2024-04-27T03:24:30.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:30.704+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:30.704827+0000) 2024-04-27T03:24:31.354 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:31.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:31.354 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:31.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:31.354 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:31.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:31.354 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:31.212+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:31.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:31.505+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:31.506440+0000) 2024-04-27T03:24:31.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:31.357+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:31.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:31.357+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:31.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:31.357+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:31.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:31.358+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:31.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:31.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:31.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:31.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:31.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:31.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:31.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:31.348+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:31.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:31.547+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:31.548626+0000) 2024-04-27T03:24:31.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:31.708+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:31.709025+0000) 2024-04-27T03:24:32.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:32.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:32.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:32.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:32.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:32.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:32.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:32.195+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:32.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:32.507+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:32.508524+0000) 2024-04-27T03:24:32.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:32.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:32.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:32.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:32.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:32.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:32.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:32.380+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:32.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:32.371+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:32.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:32.371+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:32.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:32.371+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:32.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:32.372+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:32.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:32.550+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:32.550861+0000) 2024-04-27T03:24:32.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:32.710+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:32.711278+0000) 2024-04-27T03:24:33.398 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:33.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:33.398 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:33.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:33.398 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:33.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:33.398 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:33.235+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:33.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:33.509+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:33.510674+0000) 2024-04-27T03:24:33.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:33.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:33.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:33.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:33.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:33.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:33.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:33.404+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:33.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:33.415+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:33.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:33.415+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:33.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:33.415+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:33.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:33.416+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:33.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:33.552+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:33.553072+0000) 2024-04-27T03:24:33.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:33.714+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:33.715427+0000) 2024-04-27T03:24:34.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:34.188+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:34.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:34.188+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:34.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:34.188+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:34.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:34.189+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:34.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:34.511+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:34.511842+0000) 2024-04-27T03:24:34.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:34.355+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:34.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:34.355+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:34.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:34.355+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:34.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:34.356+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:34.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:34.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:34.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:34.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:34.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:34.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:34.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:34.466+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:34.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:34.553+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:34.554228+0000) 2024-04-27T03:24:34.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:34.890+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:34.891078+0000) 2024-04-27T03:24:34.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:34.890+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:34.891475+0000) 2024-04-27T03:24:34.977 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:24:34.892+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61431 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:24:34.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:34.716+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:34.717600+0000) 2024-04-27T03:24:34.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:34.889+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:34.890408+0000) 2024-04-27T03:24:35.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:35.179+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:35.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:35.179+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:35.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:35.179+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:35.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:35.180+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:35.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:35.511+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:35.512065+0000) 2024-04-27T03:24:35.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:35.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:35.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:35.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:35.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:35.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:35.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:35.369+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:35.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:35.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:35.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:35.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:35.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:35.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:35.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:35.453+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:35.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:35.553+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:35.554476+0000) 2024-04-27T03:24:35.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:35.717+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:35.717839+0000) 2024-04-27T03:24:36.330 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:36.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:36.330 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:36.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:36.330 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:36.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:36.331 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:36.177+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:36.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:36.513+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:36.514158+0000) 2024-04-27T03:24:36.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:36.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:36.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:36.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:36.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:36.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:36.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:36.324+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:36.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:36.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:36.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:36.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:36.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:36.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:36.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:36.501+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:36.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:36.556+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:36.557177+0000) 2024-04-27T03:24:36.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:36.719+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:36.720044+0000) 2024-04-27T03:24:37.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:37.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:37.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:37.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:37.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:37.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:37.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:37.167+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:37.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:37.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:37.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:37.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:37.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:37.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:37.565 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:37.516+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:37.516841+0000) 2024-04-27T03:24:37.565 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:37.292+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:37.565 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:37.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:37.565 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:37.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:37.565 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:37.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:37.565 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:37.474+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:37.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:37.721+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:37.722687+0000) 2024-04-27T03:24:37.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:37.558+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:37.559460+0000) 2024-04-27T03:24:38.336 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:38.169+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:38.336 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:38.169+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:38.336 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:38.169+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:38.336 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:38.170+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:38.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:38.518+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:38.518926+0000) 2024-04-27T03:24:38.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:38.329+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:38.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:38.329+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:38.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:38.329+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:38.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:38.330+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:38.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:38.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:38.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:38.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:38.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:38.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:38.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:38.438+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:38.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:38.561+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:38.561942+0000) 2024-04-27T03:24:39.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:38.725+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:38.725991+0000) 2024-04-27T03:24:39.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:39.162+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:39.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:39.162+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:39.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:39.162+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:39.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:39.163+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:39.569 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:39.519+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:39.520597+0000) 2024-04-27T03:24:39.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:39.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:39.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:39.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:39.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:39.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:39.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:39.464+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:39.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:39.296+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:39.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:39.296+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:39.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:39.296+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:39.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:39.297+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:39.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:39.726+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:39.727359+0000) 2024-04-27T03:24:39.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:39.562+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:39.563647+0000) 2024-04-27T03:24:40.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:39.897+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:39.898533+0000) 2024-04-27T03:24:40.172 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:24:39.900+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61471 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:24:40.172 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:39.897+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:39.898645+0000) 2024-04-27T03:24:40.172 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:39.897+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:39.898190+0000) 2024-04-27T03:24:40.172 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:40.139+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:40.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:40.139+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:40.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:40.139+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:40.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:40.140+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:40.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:40.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:40.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:40.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:40.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:40.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:40.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:40.329+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:40.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:40.519+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:40.520794+0000) 2024-04-27T03:24:40.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:40.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:40.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:40.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:40.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:40.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:40.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:40.463+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:40.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:40.563+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:40.563896+0000) 2024-04-27T03:24:41.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:40.726+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:40.727632+0000) 2024-04-27T03:24:41.332 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:41.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:41.332 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:41.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:41.332 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:41.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:41.332 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:41.121+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:41.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:41.522+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:41.522846+0000) 2024-04-27T03:24:41.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:41.438+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:41.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:41.438+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:41.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:41.438+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:41.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:41.439+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:41.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:41.565+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:41.566535+0000) 2024-04-27T03:24:41.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:41.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:41.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:41.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:41.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:41.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:41.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:41.326+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:41.987 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:41.729+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:41.729903+0000) 2024-04-27T03:24:42.358 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:42.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:42.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:42.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:42.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:42.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:42.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:42.151+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:42.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:42.526+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:42.527066+0000) 2024-04-27T03:24:42.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:42.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:42.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:42.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:42.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:42.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:42.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:42.353+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:42.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:42.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:42.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:42.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:42.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:42.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:42.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:42.459+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:42.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:42.568+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:42.569601+0000) 2024-04-27T03:24:43.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:42.731+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:42.732220+0000) 2024-04-27T03:24:43.382 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:43.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:43.382 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:43.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:43.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:43.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:43.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:43.159+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:43.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:43.529+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:43.530175+0000) 2024-04-27T03:24:43.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:43.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:43.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:43.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:43.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:43.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:43.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:43.376+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:43.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:43.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:43.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:43.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:43.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:43.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:43.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:43.485+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:43.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:43.573+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:43.573943+0000) 2024-04-27T03:24:44.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:43.733+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:43.734474+0000) 2024-04-27T03:24:44.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:44.189+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:44.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:44.189+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:44.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:44.189+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:44.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:44.189+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:44.531+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:44.531876+0000) 2024-04-27T03:24:44.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:44.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:44.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:44.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:44.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:44.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:44.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:44.367+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:44.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:44.505+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:44.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:44.505+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:44.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:44.505+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:44.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:44.505+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:44.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:44.574+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:44.575335+0000) 2024-04-27T03:24:45.022 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:24:44.908+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61510 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:24:45.023 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:44.905+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:44.906229+0000) 2024-04-27T03:24:45.023 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:44.905+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:44.905954+0000) 2024-04-27T03:24:45.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:44.735+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:44.736795+0000) 2024-04-27T03:24:45.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:44.904+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:44.905671+0000) 2024-04-27T03:24:45.372 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:45.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:45.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:45.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:45.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:45.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:45.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:45.151+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:45.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:45.531+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:45.532062+0000) 2024-04-27T03:24:45.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:45.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:45.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:45.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:45.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:45.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:45.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:45.367+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:45.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:45.539+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:45.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:45.539+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:45.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:45.539+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:45.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:45.540+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:45.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:45.574+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:45.575598+0000) 2024-04-27T03:24:46.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:45.736+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:45.736980+0000) 2024-04-27T03:24:46.329 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:46.107+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:46.329 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:46.107+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:46.329 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:46.107+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:46.330 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:46.108+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:46.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:46.533+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:46.534206+0000) 2024-04-27T03:24:46.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:46.323+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:46.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:46.323+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:46.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:46.323+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:46.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:46.323+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:46.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:46.738+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:46.739035+0000) 2024-04-27T03:24:46.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:46.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:46.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:46.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:46.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:46.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:46.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:46.574+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:46.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:46.576+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:46.577654+0000) 2024-04-27T03:24:47.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:47.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:47.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:47.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:47.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:47.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:47.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:47.086+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:47.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:47.535+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:47.536297+0000) 2024-04-27T03:24:47.585 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:47.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:47.585 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:47.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:47.585 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:47.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:47.585 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:47.295+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:47.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:47.740+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:47.741218+0000) 2024-04-27T03:24:47.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:47.578+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:47.579725+0000) 2024-04-27T03:24:47.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:47.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:47.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:47.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:47.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:47.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:47.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:47.602+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:48.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:48.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:48.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:48.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:48.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:48.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:48.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:48.094+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:48.634 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:48.537+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:48.538463+0000) 2024-04-27T03:24:48.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:48.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:48.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:48.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:48.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:48.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:48.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:48.336+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:48.635 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:48.581+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:48.581892+0000) 2024-04-27T03:24:48.635 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:48.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:48.635 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:48.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:48.635 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:48.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:48.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:48.629+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:48.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:48.742+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:48.743400+0000) 2024-04-27T03:24:49.296 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:49.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:49.296 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:49.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:49.296 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:49.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:49.296 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:49.055+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:49.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:49.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:49.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:49.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:49.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:49.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:49.588 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:49.290+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:49.588 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:49.539+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:49.539814+0000) 2024-04-27T03:24:49.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:49.743+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:49.744667+0000) 2024-04-27T03:24:49.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:49.582+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:49.583072+0000) 2024-04-27T03:24:49.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:49.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:49.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:49.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:49.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:49.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:49.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:49.633+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:50.173 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:24:49.915+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61550 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:24:50.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:49.913+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:49.914357+0000) 2024-04-27T03:24:50.173 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:49.914+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:49.914979+0000) 2024-04-27T03:24:50.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:49.915+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:49.915992+0000) 2024-04-27T03:24:50.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:50.094+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:50.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:50.094+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:50.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:50.094+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:50.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:50.095+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:50.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:50.299+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:50.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:50.299+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:50.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:50.299+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:50.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:50.301+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:50.851 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:50.539+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:50.539982+0000) 2024-04-27T03:24:50.851 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:50.744+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:50.744821+0000) 2024-04-27T03:24:50.852 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:50.582+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:50.583268+0000) 2024-04-27T03:24:50.852 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:50.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:50.852 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:50.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:50.852 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:50.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:50.852 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:50.638+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:51.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:51.063+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:51.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:51.063+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:51.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:51.063+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:51.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:51.063+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:51.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:51.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:51.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:51.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:51.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:51.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:51.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:51.332+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:51.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:51.543+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:51.544176+0000) 2024-04-27T03:24:51.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:51.584+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:51.585506+0000) 2024-04-27T03:24:51.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:51.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:51.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:51.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:51.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:51.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:51.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:51.638+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:52.020 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:51.748+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:51.749780+0000) 2024-04-27T03:24:52.318 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:52.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:52.318 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:52.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:52.318 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:52.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:52.318 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:52.015+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:52.595 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:52.546+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:52.546978+0000) 2024-04-27T03:24:52.595 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:52.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:52.595 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:52.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:52.595 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:52.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:52.595 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:52.312+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:52.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:52.753+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:52.754757+0000) 2024-04-27T03:24:52.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:52.588+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:52.589653+0000) 2024-04-27T03:24:52.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:52.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:52.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:52.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:52.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:52.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:52.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:52.632+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:53.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:53.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:53.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:53.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:53.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:53.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:53.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:53.056+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:53.597 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:53.550+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:53.551209+0000) 2024-04-27T03:24:53.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:53.303+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:53.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:53.303+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:53.598 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:53.303+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:53.598 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:53.304+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:53.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:53.758+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:53.759763+0000) 2024-04-27T03:24:53.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:53.591+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:53.592017+0000) 2024-04-27T03:24:53.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:53.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:53.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:53.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:53.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:53.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:53.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:53.664+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:54.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:54.092+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:54.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:54.092+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:54.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:54.092+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:54.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:54.093+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:54.558 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:54.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:54.558 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:54.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:54.558 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:54.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:54.558 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:54.277+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:54.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:54.551+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:54.552718+0000) 2024-04-27T03:24:54.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:54.760+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:54.761028+0000) 2024-04-27T03:24:54.923 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:54.593+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:54.593969+0000) 2024-04-27T03:24:54.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:54.697+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:54.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:54.697+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:54.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:54.697+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:54.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:54.698+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:55.173 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:24:54.923+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61589 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:24:55.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:54.920+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:54.921655+0000) 2024-04-27T03:24:55.174 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:54.920+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:54.921249+0000) 2024-04-27T03:24:55.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:54.920+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:54.921396+0000) 2024-04-27T03:24:55.175 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:55.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:55.175 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:55.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:55.175 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:55.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:55.175 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:55.133+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:55.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:55.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:55.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:55.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:55.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:55.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:55.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:55.325+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:55.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:55.552+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:55.552930+0000) 2024-04-27T03:24:55.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:55.760+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:55.761224+0000) 2024-04-27T03:24:55.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:55.593+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:55.594153+0000) 2024-04-27T03:24:55.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:55.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:55.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:55.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:55.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:55.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:55.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:55.711+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:56.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:56.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:56.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:56.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:56.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:56.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:56.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:56.101+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:56.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:56.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:56.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:56.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:56.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:56.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:56.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:56.285+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:56.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:56.761+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:56.763557+0000) 2024-04-27T03:24:56.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:56.556+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:56.557128+0000) 2024-04-27T03:24:56.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:56.597+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:56.598426+0000) 2024-04-27T03:24:56.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:56.718+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:56.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:56.718+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:56.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:56.718+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:56.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:56.718+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:57.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:57.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:57.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:57.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:57.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:57.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:57.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:57.090+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:57.565 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:57.291+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:57.565 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:57.291+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:57.565 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:57.291+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:57.565 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:57.291+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:57.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:57.766+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:57.768541+0000) 2024-04-27T03:24:57.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:57.557+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:57.559448+0000) 2024-04-27T03:24:57.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:57.600+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:57.602670+0000) 2024-04-27T03:24:57.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:57.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:57.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:57.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:57.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:57.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:57.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:57.766+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:58.314 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:58.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:58.315 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:58.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:58.315 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:58.061+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:58.315 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:58.062+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:58.567 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:58.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:58.567 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:58.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:58.568 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:58.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:58.568 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:58.307+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:58.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:58.560+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:58.562290+0000) 2024-04-27T03:24:58.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:58.771+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:58.773681+0000) 2024-04-27T03:24:58.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:58.604+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:58.605912+0000) 2024-04-27T03:24:58.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:58.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:58.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:58.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:58.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:58.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:58.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:58.720+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:59.361 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:59.074+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:59.361 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:59.074+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:59.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:59.074+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:24:59.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:59.075+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:59.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:59.562+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:59.564193+0000) 2024-04-27T03:24:59.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:59.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:59.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:59.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:59.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:59.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:24:59.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:59.352+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:24:59.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:59.774+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:59.776593+0000) 2024-04-27T03:24:59.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:59.605+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:59.607108+0000) 2024-04-27T03:24:59.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:59.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:59.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:59.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:59.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:59.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:24:59.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:59.717+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:00.221 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:24:59.930+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61629 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:25:00.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:24:59.927+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:59.929347+0000) 2024-04-27T03:25:00.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:24:59.929+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:59.931599+0000) 2024-04-27T03:25:00.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:24:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:24:59.927+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:24:59.929381+0000) 2024-04-27T03:25:00.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:00.032+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:00.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:00.032+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:00.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:00.032+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:00.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:00.033+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:00.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:00.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:00.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:00.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:00.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:00.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:00.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:00.355+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:00.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:00.775+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:00.776828+0000) 2024-04-27T03:25:00.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:00.562+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:00.564392+0000) 2024-04-27T03:25:00.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:00.605+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:00.607321+0000) 2024-04-27T03:25:00.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:00.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:00.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:00.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:00.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:00.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:00.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:00.684+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:01.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:01.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:01.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:01.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:01.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:01.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:01.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:01.023+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:01.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:01.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:01.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:01.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:01.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:01.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:01.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:01.382+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:01.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:01.777+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:01.779452+0000) 2024-04-27T03:25:01.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:01.564+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:01.566600+0000) 2024-04-27T03:25:01.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:01.607+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:01.609580+0000) 2024-04-27T03:25:01.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:01.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:01.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:01.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:01.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:01.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:01.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:01.711+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:02.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:02.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:02.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:02.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:02.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:02.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:02.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:02.041+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:02.617 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:02.567+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:02.568823+0000) 2024-04-27T03:25:02.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:02.355+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:02.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:02.355+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:02.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:02.355+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:02.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:02.355+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:02.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:02.779+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:02.781622+0000) 2024-04-27T03:25:02.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:02.610+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:02.611845+0000) 2024-04-27T03:25:02.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:02.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:02.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:02.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:02.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:02.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:02.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:02.693+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:03.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:03.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:03.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:03.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:03.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:03.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:03.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:03.053+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:03.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:03.569+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:03.571499+0000) 2024-04-27T03:25:03.620 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:03.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:03.620 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:03.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:03.621 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:03.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:03.621 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:03.344+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:03.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:03.784+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:03.785844+0000) 2024-04-27T03:25:03.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:03.613+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:03.614977+0000) 2024-04-27T03:25:03.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:03.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:03.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:03.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:03.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:03.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:03.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:03.688+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:04.312 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:04.098+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:04.312 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:04.098+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:04.312 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:04.098+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:04.312 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:04.099+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:04.578 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:04.304+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:04.578 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:04.304+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:04.578 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:04.304+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:04.578 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:04.305+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:04.938 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:04.571+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:04.573302+0000) 2024-04-27T03:25:04.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:04.786+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:04.788222+0000) 2024-04-27T03:25:04.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:04.614+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:04.616780+0000) 2024-04-27T03:25:04.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:04.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:04.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:04.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:04.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:04.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:04.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:04.672+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:05.221 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:25:04.937+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61668 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:25:05.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:04.934+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:04.936593+0000) 2024-04-27T03:25:05.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:04.936+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:04.938071+0000) 2024-04-27T03:25:05.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:04.937+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:04.938873+0000) 2024-04-27T03:25:05.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:05.095+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:05.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:05.095+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:05.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:05.095+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:05.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:05.096+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:05.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:05.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:05.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:05.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:05.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:05.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:05.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:05.308+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:05.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:05.571+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:05.573517+0000) 2024-04-27T03:25:05.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:05.786+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:05.788425+0000) 2024-04-27T03:25:05.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:05.615+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:05.616966+0000) 2024-04-27T03:25:05.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:05.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:05.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:05.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:05.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:05.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:05.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:05.690+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:06.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:06.068+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:06.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:06.068+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:06.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:06.068+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:06.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:06.068+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:06.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:06.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:06.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:06.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:06.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:06.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:06.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:06.326+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:06.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:06.575+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:06.577712+0000) 2024-04-27T03:25:06.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:06.790+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:06.792554+0000) 2024-04-27T03:25:06.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:06.617+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:06.619053+0000) 2024-04-27T03:25:06.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:06.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:06.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:06.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:06.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:06.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:06.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:06.735+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:07.371 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:07.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:07.371 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:07.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:07.371 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:07.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:07.372 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:07.112+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:07.626 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:07.578+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:07.580124+0000) 2024-04-27T03:25:07.627 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:07.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:07.627 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:07.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:07.627 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:07.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:07.627 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:07.365+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:07.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:07.792+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:07.794740+0000) 2024-04-27T03:25:07.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:07.619+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:07.621208+0000) 2024-04-27T03:25:07.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:07.732+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:07.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:07.732+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:07.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:07.732+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:07.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:07.732+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:08.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:08.087+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:08.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:08.087+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:08.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:08.087+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:08.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:08.087+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:08.587 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:08.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:08.588 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:08.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:08.588 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:08.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:08.588 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:08.326+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:08.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:08.795+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:08.796842+0000) 2024-04-27T03:25:08.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:08.580+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:08.582377+0000) 2024-04-27T03:25:08.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:08.622+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:08.624093+0000) 2024-04-27T03:25:08.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:08.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:08.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:08.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:08.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:08.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:08.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:08.752+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:09.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:09.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:09.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:09.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:09.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:09.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:09.310 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:09.066+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:09.589 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:09.303+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:09.589 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:09.303+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:09.590 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:09.303+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:09.590 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:09.304+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:09.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:09.583+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:09.583840+0000) 2024-04-27T03:25:09.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:09.797+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:09.798359+0000) 2024-04-27T03:25:09.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:09.624+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:09.625334+0000) 2024-04-27T03:25:09.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:09.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:09.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:09.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:09.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:09.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:09.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:09.756+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:10.222 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:25:09.946+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61708 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:25:10.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:09.943+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:09.944703+0000) 2024-04-27T03:25:10.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:09.943+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:09.944428+0000) 2024-04-27T03:25:10.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:09.944+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:09.944895+0000) 2024-04-27T03:25:10.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:10.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:10.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:10.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:10.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:10.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:10.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:10.034+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:10.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:10.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:10.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:10.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:10.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:10.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:10.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:10.276+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:10.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:10.583+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:10.584170+0000) 2024-04-27T03:25:10.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:10.797+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:10.798632+0000) 2024-04-27T03:25:10.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:10.624+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:10.625508+0000) 2024-04-27T03:25:10.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:10.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:10.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:10.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:10.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:10.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:10.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:10.792+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:11.289 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:11.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:11.289 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:11.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:11.289 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:11.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:11.289 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:11.053+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:11.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:11.283+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:11.592 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:11.283+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:11.592 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:11.283+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:11.592 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:11.283+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:11.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:11.585+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:11.586280+0000) 2024-04-27T03:25:11.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:11.800+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:11.800840+0000) 2024-04-27T03:25:11.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:11.629+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:11.630517+0000) 2024-04-27T03:25:11.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:11.768+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:11.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:11.768+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:11.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:11.768+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:11.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:11.769+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:12.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:12.023+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:12.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:12.023+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:12.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:12.023+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:12.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:12.024+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:12.594 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:12.316+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:12.595 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:12.316+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:12.595 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:12.316+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:12.595 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:12.317+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:12.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:12.588+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:12.589304+0000) 2024-04-27T03:25:12.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:12.802+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:12.803134+0000) 2024-04-27T03:25:12.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:12.634+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:12.635489+0000) 2024-04-27T03:25:12.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:12.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:12.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:12.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:12.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:12.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:12.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:12.805+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:13.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:13.005+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:13.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:13.005+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:13.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:13.005+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:13.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:13.006+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:13.643 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:13.591+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:13.592164+0000) 2024-04-27T03:25:13.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:13.360+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:13.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:13.360+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:13.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:13.360+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:13.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:13.361+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:13.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:13.804+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:13.805235+0000) 2024-04-27T03:25:13.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:13.636+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:13.637761+0000) 2024-04-27T03:25:13.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:13.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:13.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:13.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:13.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:13.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:13.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:13.845+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:14.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:13.980+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:14.358 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:13.980+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:14.358 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:13.980+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:14.358 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:13.980+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:14.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:14.592+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:14.593599+0000) 2024-04-27T03:25:14.644 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:14.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:14.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:14.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:14.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:14.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:14.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:14.352+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:14.954 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:14.805+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:14.806667+0000) 2024-04-27T03:25:14.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:14.638+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:14.639173+0000) 2024-04-27T03:25:14.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:14.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:14.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:14.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:14.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:14.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:14.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:14.852+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:15.222 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:25:14.953+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61747 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:25:15.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:14.950+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:14.951611+0000) 2024-04-27T03:25:15.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:14.952+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:14.952933+0000) 2024-04-27T03:25:15.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:14.987+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:15.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:14.987+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:15.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:14.987+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:15.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:14.989+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:15.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:14.952+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:14.953148+0000) 2024-04-27T03:25:15.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:15.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:15.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:15.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:15.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:15.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:15.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:15.385+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:15.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:15.806+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:15.806842+0000) 2024-04-27T03:25:15.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:15.592+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:15.593781+0000) 2024-04-27T03:25:15.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:15.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:15.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:15.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:15.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:15.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:15.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:15.638+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:15.639415+0000) 2024-04-27T03:25:15.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:15.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:15.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:15.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:15.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:15.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:15.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:15.824+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:16.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:15.942+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:16.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:16.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:16.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:16.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:16.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:16.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:16.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:16.398+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:16.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:16.595+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:16.595838+0000) 2024-04-27T03:25:16.879 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:16.810+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:16.811034+0000) 2024-04-27T03:25:16.880 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:16.640+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:16.641546+0000) 2024-04-27T03:25:17.130 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:16.900+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:17.130 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:16.900+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:17.130 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:16.900+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:17.131 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:16.901+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:17.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:16.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:17.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:16.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:17.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:16.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:17.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:16.874+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:17.597+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:17.598091+0000) 2024-04-27T03:25:17.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:17.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:17.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:17.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:17.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:17.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:17.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:17.409+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:17.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:17.645+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:17.646545+0000) 2024-04-27T03:25:18.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:17.847+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:18.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:17.847+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:18.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:17.847+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:18.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:17.849+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:18.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:17.926+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:18.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:17.926+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:18.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:17.926+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:18.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:17.927+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:18.111 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:17.812+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:17.813184+0000) 2024-04-27T03:25:18.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:18.599+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:18.600237+0000) 2024-04-27T03:25:18.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:18.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:18.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:18.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:18.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:18.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:18.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:18.391+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:18.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:18.816+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:18.817219+0000) 2024-04-27T03:25:18.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:18.882+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:18.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:18.882+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:18.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:18.882+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:18.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:18.883+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:18.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:18.649+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:18.650685+0000) 2024-04-27T03:25:18.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:18.845+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:18.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:18.845+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:18.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:18.845+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:18.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:18.846+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:19.657 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:19.601+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:19.601797+0000) 2024-04-27T03:25:19.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:19.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:19.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:19.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:19.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:19.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:19.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:19.411+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:19.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:19.877+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:19.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:19.877+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:19.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:19.877+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:19.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:19.877+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:19.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:19.817+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:19.818617+0000) 2024-04-27T03:25:19.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:19.651+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:19.652010+0000) 2024-04-27T03:25:19.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:19.832+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:19.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:19.832+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:19.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:19.832+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:19.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:19.833+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:20.227 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:25:19.962+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61787 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:25:20.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:19.960+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:19.960999+0000) 2024-04-27T03:25:20.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:19.958+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:19.959615+0000) 2024-04-27T03:25:20.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:19.958+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:19.959297+0000) 2024-04-27T03:25:20.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:20.450+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:20.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:20.450+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:20.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:20.450+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:20.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:20.451+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:20.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:20.601+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:20.601968+0000) 2024-04-27T03:25:20.883 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:20.818+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:20.818808+0000) 2024-04-27T03:25:20.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:20.651+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:20.652484+0000) 2024-04-27T03:25:20.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:20.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:20.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:20.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:20.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:20.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:20.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:20.813+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:21.133 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:20.876+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:21.133 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:20.876+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:21.133 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:20.876+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:21.133 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:20.878+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:21.662 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:21.603+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:21.604096+0000) 2024-04-27T03:25:21.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:21.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:21.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:21.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:21.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:21.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:21.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:21.455+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:21.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:21.823+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:21.823884+0000) 2024-04-27T03:25:21.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:21.879+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:21.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:21.879+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:21.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:21.879+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:21.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:21.880+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:21.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:21.656+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:21.657390+0000) 2024-04-27T03:25:21.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:21.787+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:21.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:21.787+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:21.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:21.787+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:21.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:21.788+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:22.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:22.605+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:22.606412+0000) 2024-04-27T03:25:22.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:22.661+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:22.662278+0000) 2024-04-27T03:25:22.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:22.462+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:22.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:22.462+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:22.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:22.462+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:22.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:22.463+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:23.110 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:22.828+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:22.828912+0000) 2024-04-27T03:25:23.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:22.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:23.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:22.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:23.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:22.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:23.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:22.891+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:23.111 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:22.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:23.111 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:22.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:23.111 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:22.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:23.111 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:22.773+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:23.672 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:23.608+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:23.609339+0000) 2024-04-27T03:25:23.672 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:23.511+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:23.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:23.511+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:23.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:23.511+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:23.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:23.511+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:23.936 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:23.833+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:23.834012+0000) 2024-04-27T03:25:23.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:23.666+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:23.667308+0000) 2024-04-27T03:25:23.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:23.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:23.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:23.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:23.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:23.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:23.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:23.801+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:24.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:23.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:24.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:23.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:24.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:23.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:24.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:23.931+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:24.785 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:24.611+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:24.611816+0000) 2024-04-27T03:25:24.786 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:24.669+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:24.670336+0000) 2024-04-27T03:25:24.786 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:24.496+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:24.786 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:24.496+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:24.786 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:24.496+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:24.786 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:24.497+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:25.094 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:25:24.969+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61826 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:25:25.094 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:24.836+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:24.836978+0000) 2024-04-27T03:25:25.094 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:24.969+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:24.970373+0000) 2024-04-27T03:25:25.094 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:24.779+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:25.095 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:24.779+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:25.095 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:24.779+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:25.095 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:24.780+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:25.095 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:24.967+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:24.967853+0000) 2024-04-27T03:25:25.095 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:24.951+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:25.095 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:24.951+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:25.095 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:24.951+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:25.095 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:24.952+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:25.095 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:24.967+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:24.968648+0000) 2024-04-27T03:25:25.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:25.611+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:25.612035+0000) 2024-04-27T03:25:25.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:25.486+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:25.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:25.486+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:25.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:25.486+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:25.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:25.486+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:25.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:25.836+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:25.837175+0000) 2024-04-27T03:25:25.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:25.669+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:25.670524+0000) 2024-04-27T03:25:25.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:25.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:25.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:25.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:25.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:25.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:25.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:25.755+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:26.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:25.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:26.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:25.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:26.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:25.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:26.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:25.932+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:26.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:26.614+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:26.615070+0000) 2024-04-27T03:25:26.845 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:26.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:26.845 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:26.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:26.845 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:26.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:26.845 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:26.474+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:26.845 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:26.672+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:26.673170+0000) 2024-04-27T03:25:26.845 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:26.709+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:26.845 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:26.709+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:26.845 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:26.709+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:26.845 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:26.710+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:27.111 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:26.838+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:26.839331+0000) 2024-04-27T03:25:27.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:26.886+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:27.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:26.886+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:27.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:26.886+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:27.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:26.888+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:27.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:27.617+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:27.617879+0000) 2024-04-27T03:25:27.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:27.501+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:27.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:27.501+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:27.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:27.501+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:27.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:27.502+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:27.936 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:27.841+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:27.842635+0000) 2024-04-27T03:25:27.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:27.674+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:27.675342+0000) 2024-04-27T03:25:27.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:27.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:27.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:27.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:27.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:27.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:27.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:27.678+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:28.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:27.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:28.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:27.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:28.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:27.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:28.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:27.931+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:28.850 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:28.620+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:28.621054+0000) 2024-04-27T03:25:28.850 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:28.677+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:28.678279+0000) 2024-04-27T03:25:28.850 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:28.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:28.851 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:28.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:28.851 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:28.681+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:28.851 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:28.682+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:28.851 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:28.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:28.851 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:28.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:28.851 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:28.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:28.851 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:28.471+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:28.851 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:28.844+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:28.844928+0000) 2024-04-27T03:25:29.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:28.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:29.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:28.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:29.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:28.939+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:29.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:28.940+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:29.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:29.621+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:29.622346+0000) 2024-04-27T03:25:29.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:29.491+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:29.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:29.491+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:29.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:29.491+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:29.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:29.492+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:29.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:29.846+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:29.846849+0000) 2024-04-27T03:25:29.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:29.679+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:29.679870+0000) 2024-04-27T03:25:29.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:29.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:29.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:29.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:29.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:29.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:29.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:29.699+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:30.286 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:29.973+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:29.974599+0000) 2024-04-27T03:25:30.287 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:25:29.976+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61866 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:25:30.287 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:29.974+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:29.974959+0000) 2024-04-27T03:25:30.287 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:29.988+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:30.287 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:29.988+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:30.287 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:29.988+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:30.287 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:29.989+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:30.287 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:29.974+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:29.974801+0000) 2024-04-27T03:25:30.627 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:30.452+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:30.628 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:30.452+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:30.628 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:30.452+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:30.628 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:30.452+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:30.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:30.621+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:30.622612+0000) 2024-04-27T03:25:30.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:30.846+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:30.847043+0000) 2024-04-27T03:25:30.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:30.679+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:30.680097+0000) 2024-04-27T03:25:30.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:30.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:30.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:30.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:30.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:30.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:30.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:30.734+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:31.249 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:30.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:31.249 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:30.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:31.249 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:30.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:31.249 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:30.991+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:31.630 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:31.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:31.630 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:31.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:31.630 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:31.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:31.630 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:31.449+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:31.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:31.624+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:31.624995+0000) 2024-04-27T03:25:31.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:31.849+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:31.850222+0000) 2024-04-27T03:25:31.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:31.683+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:31.684359+0000) 2024-04-27T03:25:31.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:31.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:31.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:31.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:31.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:31.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:31.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:31.712+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:32.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:32.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:32.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:32.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:32.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:32.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:32.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:32.011+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:32.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:32.626+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:32.627465+0000) 2024-04-27T03:25:32.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:32.448+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:32.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:32.448+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:32.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:32.448+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:32.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:32.448+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:32.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:32.851+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:32.852311+0000) 2024-04-27T03:25:32.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:32.685+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:32.686499+0000) 2024-04-27T03:25:32.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:32.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:32.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:32.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:32.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:32.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:32.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:32.704+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:33.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:32.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:33.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:32.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:33.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:32.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:33.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:32.969+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:33.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:33.628+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:33.629559+0000) 2024-04-27T03:25:33.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:33.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:33.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:33.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:33.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:33.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:33.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:33.409+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:33.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:33.853+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:33.854462+0000) 2024-04-27T03:25:33.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:33.670+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:33.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:33.670+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:33.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:33.670+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:33.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:33.671+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:33.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:33.688+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:33.689502+0000) 2024-04-27T03:25:34.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:33.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:34.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:33.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:34.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:33.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:34.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:33.934+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:34.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:34.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:34.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:34.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:34.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:34.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:34.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:34.437+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:34.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:34.855+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:34.856337+0000) 2024-04-27T03:25:34.938 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:34.630+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:34.630824+0000) 2024-04-27T03:25:34.938 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:34.932+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:34.938 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:34.932+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:34.939 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:34.932+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:34.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:34.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:34.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:34.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:34.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:34.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:34.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:34.629+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:34.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:34.689+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:34.690763+0000) 2024-04-27T03:25:35.224 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:25:34.984+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61905 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:25:35.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:34.981+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:34.982693+0000) 2024-04-27T03:25:35.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:34.981+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:34.982761+0000) 2024-04-27T03:25:35.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:34.933+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:35.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:34.981+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:34.982767+0000) 2024-04-27T03:25:35.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:35.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:35.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:35.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:35.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:35.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:35.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:35.439+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:35.861 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:35.630+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:35.630992+0000) 2024-04-27T03:25:35.862 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:35.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:35.862 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:35.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:35.862 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:35.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:35.862 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:35.594+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:35.862 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:35.690+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:35.690931+0000) 2024-04-27T03:25:36.114 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:35.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:36.114 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:35.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:36.114 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:35.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:36.114 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:35.968+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:36.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:35.855+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:35.856521+0000) 2024-04-27T03:25:36.638 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:36.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:36.639 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:36.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:36.639 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:36.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:36.639 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:36.414+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:36.639 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:36.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:36.639 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:36.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:36.639 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:36.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:36.639 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:36.558+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:36.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:36.632+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:36.633284+0000) 2024-04-27T03:25:36.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:36.860+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:36.861183+0000) 2024-04-27T03:25:36.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:36.692+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:36.692963+0000) 2024-04-27T03:25:37.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:36.923+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:37.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:36.923+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:37.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:36.923+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:37.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:36.924+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:37.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:37.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:37.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:37.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:37.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:37.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:37.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:37.413+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:37.868 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:37.635+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:37.635852+0000) 2024-04-27T03:25:37.868 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:37.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:37.869 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:37.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:37.869 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:37.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:37.869 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:37.544+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:37.869 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:37.694+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:37.695069+0000) 2024-04-27T03:25:38.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:37.862+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:37.863289+0000) 2024-04-27T03:25:38.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:37.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:38.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:37.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:38.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:37.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:38.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:37.926+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:38.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:38.639+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:38.640695+0000) 2024-04-27T03:25:38.702 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:38.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:38.703 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:38.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:38.703 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:38.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:38.703 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:38.549+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:38.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:38.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:38.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:38.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:38.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:38.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:38.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:38.414+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:38.696+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:38.697146+0000) 2024-04-27T03:25:38.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:38.866+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:38.867593+0000) 2024-04-27T03:25:38.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:38.889+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:38.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:38.889+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:38.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:38.889+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:38.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:38.891+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:39.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:39.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:39.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:39.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:39.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:39.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:39.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:39.383+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:39.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:39.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:39.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:39.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:39.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:39.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:39.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:39.528+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:39.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:39.697+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:39.698714+0000) 2024-04-27T03:25:39.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:39.868+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:39.868964+0000) 2024-04-27T03:25:39.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:39.641+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:39.642053+0000) 2024-04-27T03:25:39.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:39.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:39.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:39.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:39.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:39.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:39.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:39.873+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:40.287 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:25:39.991+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61945 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:25:40.287 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:39.989+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:39.990364+0000) 2024-04-27T03:25:40.288 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:39.989+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:39.989902+0000) 2024-04-27T03:25:40.288 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:39.988+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:39.989566+0000) 2024-04-27T03:25:40.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:40.514+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:40.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:40.514+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:40.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:40.514+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:40.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:40.514+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:40.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:40.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:40.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:40.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:40.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:40.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:40.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:40.415+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:40.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:40.698+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:40.698903+0000) 2024-04-27T03:25:40.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:40.868+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:40.869155+0000) 2024-04-27T03:25:40.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:40.641+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:40.642258+0000) 2024-04-27T03:25:40.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:40.904+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:40.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:40.904+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:40.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:40.904+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:41.178 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:40.905+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:41.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:41.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:41.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:41.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:41.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:41.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:41.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:41.392+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:41.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:41.644+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:41.645160+0000) 2024-04-27T03:25:41.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:41.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:41.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:41.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:41.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:41.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:41.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:41.546+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:41.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:41.700+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:41.701054+0000) 2024-04-27T03:25:42.155 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:41.916+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:42.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:41.916+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:42.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:41.916+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:42.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:41.918+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:42.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:41.871+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:41.872643+0000) 2024-04-27T03:25:42.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:42.646+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:42.647624+0000) 2024-04-27T03:25:42.708 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:42.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:42.708 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:42.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:42.709 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:42.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:42.709 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:42.406+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:42.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:42.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:42.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:42.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:42.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:42.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:42.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:42.577+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:42.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:42.874+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:42.875610+0000) 2024-04-27T03:25:42.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:42.702+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:42.703207+0000) 2024-04-27T03:25:43.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:42.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:43.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:42.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:43.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:42.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:43.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:42.956+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:43.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:43.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:43.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:43.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:43.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:43.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:43.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:43.370+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:43.831 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:43.649+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:43.649923+0000) 2024-04-27T03:25:43.832 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:43.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:43.832 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:43.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:43.832 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:43.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:43.832 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:43.565+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:43.832 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:43.704+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:43.705595+0000) 2024-04-27T03:25:44.114 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:43.877+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:43.878685+0000) 2024-04-27T03:25:44.114 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:43.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:44.114 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:43.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:44.114 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:43.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:44.114 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:43.956+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:44.657 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:44.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:44.657 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:44.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:44.657 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:44.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:44.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:44.531+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:44.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:44.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:44.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:44.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:44.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:44.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:44.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:44.398+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:44.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:44.879+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:44.880268+0000) 2024-04-27T03:25:44.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:44.706+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:44.707713+0000) 2024-04-27T03:25:44.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:44.650+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:44.651386+0000) 2024-04-27T03:25:44.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:44.914+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:44.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:44.914+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:45.178 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:25:44.999+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 61984 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:25:45.179 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:44.997+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:44.998192+0000) 2024-04-27T03:25:45.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:44.996+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:44.997420+0000) 2024-04-27T03:25:45.179 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:44.914+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:45.179 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:44.915+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:45.179 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:44.996+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:44.997466+0000) 2024-04-27T03:25:45.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:45.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:45.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:45.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:45.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:45.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:45.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:45.396+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:45.885 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:45.650+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:45.651568+0000) 2024-04-27T03:25:45.886 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:45.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:45.886 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:45.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:45.886 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:45.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:45.886 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:45.538+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:45.886 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:45.707+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:45.707888+0000) 2024-04-27T03:25:46.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:45.879+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:45.880507+0000) 2024-04-27T03:25:46.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:45.923+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:46.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:45.923+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:46.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:45.923+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:46.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:45.924+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:46.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:46.655+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:46.655819+0000) 2024-04-27T03:25:46.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:46.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:46.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:46.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:46.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:46.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:46.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:46.406+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:46.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:46.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:46.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:46.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:46.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:46.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:46.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:46.530+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:46.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:46.710+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:46.711028+0000) 2024-04-27T03:25:46.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:46.883+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:46.884730+0000) 2024-04-27T03:25:46.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:46.880+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:46.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:46.880+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:46.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:46.880+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:46.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:46.880+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:47.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:47.447+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:47.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:47.447+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:47.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:47.447+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:47.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:47.447+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:47.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:47.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:47.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:47.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:47.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:47.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:47.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:47.579+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:47.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:47.714+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:47.715123+0000) 2024-04-27T03:25:47.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:47.886+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:47.886968+0000) 2024-04-27T03:25:47.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:47.657+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:47.658078+0000) 2024-04-27T03:25:47.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:47.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:47.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:47.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:47.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:47.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:48.178 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:47.923+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:48.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:48.660+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:48.661274+0000) 2024-04-27T03:25:48.892 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:48.481+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:48.892 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:48.481+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:48.892 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:48.481+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:48.892 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:48.482+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:48.892 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:48.588+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:48.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:48.588+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:48.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:48.588+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:48.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:48.589+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:48.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:48.716+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:48.717284+0000) 2024-04-27T03:25:49.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:48.889+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:48.889822+0000) 2024-04-27T03:25:49.160 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:48.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:49.160 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:48.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:49.160 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:48.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:49.160 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:48.886+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:49.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:49.661+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:49.662569+0000) 2024-04-27T03:25:49.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:49.476+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:49.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:49.476+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:49.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:49.476+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:49.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:49.476+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:49.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:49.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:49.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:49.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:49.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:49.569+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:49.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:49.570+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:49.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:49.719+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:49.719849+0000) 2024-04-27T03:25:50.160 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:25:50.006+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62024 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:25:50.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:49.891+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:49.892373+0000) 2024-04-27T03:25:50.161 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:50.004+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:50.004936+0000) 2024-04-27T03:25:50.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:49.927+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:50.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:49.927+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:50.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:49.927+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:50.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:49.927+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:50.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:50.005+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:50.005941+0000) 2024-04-27T03:25:50.161 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:50.004+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:50.005100+0000) 2024-04-27T03:25:50.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:50.661+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:50.662773+0000) 2024-04-27T03:25:50.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:50.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:50.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:50.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:50.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:50.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:50.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:50.517+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:50.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:50.891+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:50.892558+0000) 2024-04-27T03:25:50.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:50.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:50.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:50.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:50.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:50.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:50.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:50.548+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:50.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:50.719+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:50.720093+0000) 2024-04-27T03:25:51.163 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:50.914+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:51.163 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:50.914+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:51.163 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:50.914+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:51.164 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:50.915+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:51.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:51.664+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:51.664847+0000) 2024-04-27T03:25:51.901 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:51.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:51.901 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:51.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:51.901 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:51.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:51.901 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:51.501+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:51.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:51.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:51.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:51.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:51.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:51.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:51.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:51.551+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:51.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:51.721+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:51.722694+0000) 2024-04-27T03:25:52.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:51.894+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:51.895539+0000) 2024-04-27T03:25:52.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:51.935+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:52.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:51.935+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:52.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:51.935+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:52.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:51.936+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:52.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:52.666+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:52.667175+0000) 2024-04-27T03:25:52.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:52.484+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:52.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:52.484+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:52.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:52.484+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:52.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:52.484+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:52.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:52.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:52.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:52.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:52.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:52.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:52.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:52.573+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:52.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:52.725+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:52.725932+0000) 2024-04-27T03:25:53.164 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:52.946+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:53.165 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:52.946+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:53.165 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:52.946+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:53.165 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:52.947+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:53.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:52.899+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:52.900242+0000) 2024-04-27T03:25:53.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:53.669+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:53.670250+0000) 2024-04-27T03:25:53.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:53.611+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:53.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:53.611+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:53.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:53.611+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:53.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:53.612+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:53.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:53.450+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:53.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:53.450+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:53.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:53.450+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:53.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:53.451+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:54.030 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:53.727+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:53.728645+0000) 2024-04-27T03:25:54.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:53.905+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:54.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:53.905+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:54.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:53.905+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:54.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:53.907+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:54.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:53.902+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:53.902951+0000) 2024-04-27T03:25:54.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:54.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:54.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:54.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:54.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:54.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:54.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:54.621+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:54.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:54.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:54.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:54.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:54.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:54.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:54.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:54.474+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:54.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:54.670+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:54.671598+0000) 2024-04-27T03:25:54.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:54.729+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:54.729995+0000) 2024-04-27T03:25:54.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:54.903+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:54.904745+0000) 2024-04-27T03:25:55.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:55.011+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:55.012103+0000) 2024-04-27T03:25:55.226 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:25:55.014+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62063 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:25:55.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:54.926+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:55.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:54.926+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:55.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:54.926+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:55.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:54.927+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:55.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:55.012+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:55.013131+0000) 2024-04-27T03:25:55.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:55.011+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:55.012232+0000) 2024-04-27T03:25:55.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:55.466+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:55.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:55.466+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:55.598 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:55.466+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:55.598 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:55.467+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:55.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:55.670+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:55.671790+0000) 2024-04-27T03:25:55.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:55.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:55.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:55.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:55.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:55.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:55.909 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:55.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:55.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:55.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:55.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:55.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:55.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:55.591+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:55.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:55.729+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:55.730183+0000) 2024-04-27T03:25:56.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:55.903+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:56.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:55.904+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:55.904903+0000) 2024-04-27T03:25:56.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:56.675+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:56.675952+0000) 2024-04-27T03:25:56.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:56.469+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:56.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:56.469+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:56.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:56.469+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:56.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:56.469+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:56.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:56.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:56.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:56.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:56.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:56.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:56.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:56.635+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:57.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:56.731+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:56.732309+0000) 2024-04-27T03:25:57.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:56.907+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:56.908050+0000) 2024-04-27T03:25:57.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:56.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:57.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:56.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:57.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:56.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:57.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:56.931+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:57.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:57.479+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:57.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:57.479+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:57.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:57.479+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:57.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:57.479+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:57.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:57.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:57.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:57.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:57.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:57.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:57.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:57.591+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:57.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:57.909+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:57.910347+0000) 2024-04-27T03:25:57.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:57.733+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:57.734591+0000) 2024-04-27T03:25:57.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:57.677+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:57.678585+0000) 2024-04-27T03:25:57.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:57.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:57.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:57.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:57.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:57.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:58.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:57.948+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:58.788 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:58.680+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:58.681154+0000) 2024-04-27T03:25:58.788 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:58.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:58.788 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:58.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:58.788 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:58.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:58.788 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:58.519+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:58.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:58.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:58.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:58.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:58.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:58.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:58.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:58.593+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:58.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:58.735+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:58.736760+0000) 2024-04-27T03:25:59.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:58.913+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:58.915471+0000) 2024-04-27T03:25:59.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:58.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:59.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:58.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:59.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:58.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:25:59.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:58.960+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:59.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:59.680+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:59.682641+0000) 2024-04-27T03:25:59.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:59.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:59.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:59.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:59.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:59.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:25:59.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:59.580+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:59.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:25:59.736+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:59.738172+0000) 2024-04-27T03:25:59.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:59.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:59.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:59.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:59.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:59.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:25:59.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:59.528+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:25:59.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:25:59.916+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:25:59.918559+0000) 2024-04-27T03:26:00.180 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:26:00.020+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62103 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:26:00.180 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:00.017+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:00.019755+0000) 2024-04-27T03:26:00.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:00.018+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:00.020281+0000) 2024-04-27T03:26:00.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:59.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:00.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:59.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:00.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:59.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:00.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:25:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:25:59.932+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:00.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:00.018+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:00.020355+0000) 2024-04-27T03:26:00.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:00.681+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:00.682844+0000) 2024-04-27T03:26:00.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:00.538+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:00.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:00.538+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:00.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:00.538+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:00.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:00.539+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:00.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:00.736+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:00.738381+0000) 2024-04-27T03:26:00.901 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:00.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:00.901 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:00.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:00.901 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:00.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:00.901 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:00.489+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:01.172 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:00.963+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:01.172 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:00.963+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:01.172 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:00.963+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:01.172 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:00.964+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:01.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:00.917+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:00.918822+0000) 2024-04-27T03:26:01.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:01.683+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:01.684958+0000) 2024-04-27T03:26:01.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:01.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:01.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:01.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:01.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:01.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:01.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:01.443+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:01.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:01.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:01.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:01.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:01.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:01.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:01.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:01.555+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:02.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:01.919+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:01.921454+0000) 2024-04-27T03:26:02.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:01.948+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:02.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:01.948+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:02.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:01.948+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:02.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:01.949+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:02.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:01.738+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:01.740504+0000) 2024-04-27T03:26:02.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:02.557+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:02.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:02.557+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:02.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:02.557+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:02.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:02.558+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:02.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:02.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:02.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:02.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:02.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:02.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:02.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:02.456+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:02.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:02.740+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:02.742662+0000) 2024-04-27T03:26:02.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:02.922+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:02.924356+0000) 2024-04-27T03:26:02.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:02.685+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:02.687276+0000) 2024-04-27T03:26:03.407 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:02.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:03.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:02.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:03.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:02.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:03.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:02.974+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:03.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:03.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:03.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:03.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:03.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:03.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:03.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:03.445+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:03.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:03.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:03.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:03.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:03.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:03.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:03.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:03.575+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:03.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:03.688+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:03.690226+0000) 2024-04-27T03:26:03.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:03.926+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:03.928583+0000) 2024-04-27T03:26:03.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:03.743+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:03.744807+0000) 2024-04-27T03:26:04.407 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:03.991+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:04.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:03.991+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:04.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:03.991+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:04.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:03.993+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:04.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:04.472+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:04.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:04.472+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:04.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:04.472+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:04.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:04.472+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:04.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:04.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:04.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:04.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:04.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:04.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:04.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:04.582+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:04.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:04.744+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:04.746114+0000) 2024-04-27T03:26:04.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:04.690+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:04.692102+0000) 2024-04-27T03:26:04.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:04.928+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:04.929958+0000) 2024-04-27T03:26:05.407 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:26:05.028+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62142 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:26:05.408 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:05.025+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:05.027410+0000) 2024-04-27T03:26:05.408 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:05.025+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:05.027158+0000) 2024-04-27T03:26:05.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:05.026+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:05.027940+0000) 2024-04-27T03:26:05.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:05.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:05.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:05.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:05.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:05.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:05.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:05.027+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:05.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:05.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:05.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:05.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:05.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:05.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:05.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:05.573+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:05.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:05.514+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:05.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:05.514+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:05.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:05.514+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:05.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:05.514+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:05.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:05.690+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:05.692365+0000) 2024-04-27T03:26:05.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:05.744+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:05.746299+0000) 2024-04-27T03:26:05.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:05.928+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:05.930142+0000) 2024-04-27T03:26:06.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:06.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:06.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:06.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:06.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:06.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:06.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:06.076+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:06.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:06.588+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:06.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:06.588+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:06.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:06.588+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:06.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:06.588+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:06.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:06.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:06.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:06.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:06.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:06.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:06.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:06.530+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:06.937 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:06.693+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:06.694842+0000) 2024-04-27T03:26:06.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:06.746+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:06.748388+0000) 2024-04-27T03:26:07.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:06.930+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:06.932223+0000) 2024-04-27T03:26:07.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:07.059+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:07.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:07.059+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:07.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:07.059+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:07.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:07.060+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:07.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:07.697+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:07.699114+0000) 2024-04-27T03:26:07.940 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:07.569+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:07.940 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:07.569+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:07.941 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:07.569+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:07.941 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:07.570+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:07.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:07.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:07.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:07.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:07.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:07.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:07.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:07.557+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:07.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:07.749+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:07.751323+0000) 2024-04-27T03:26:08.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:07.933+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:07.934890+0000) 2024-04-27T03:26:08.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:08.070+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:08.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:08.070+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:08.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:08.070+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:08.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:08.071+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:08.942 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:08.699+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:08.701361+0000) 2024-04-27T03:26:08.943 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:08.570+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:08.943 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:08.570+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:08.943 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:08.570+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:08.943 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:08.570+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:08.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:08.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:08.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:08.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:08.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:08.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:08.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:08.527+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:08.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:08.752+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:08.754311+0000) 2024-04-27T03:26:09.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:08.935+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:08.937328+0000) 2024-04-27T03:26:09.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:09.066+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:09.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:09.066+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:09.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:09.066+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:09.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:09.066+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:09.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:09.701+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:09.702959+0000) 2024-04-27T03:26:09.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:09.615+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:09.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:09.615+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:09.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:09.615+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:09.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:09.615+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:09.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:09.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:09.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:09.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:09.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:09.564+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:09.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:09.565+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:09.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:09.753+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:09.755685+0000) 2024-04-27T03:26:10.226 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:26:10.035+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62182 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:26:10.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:09.937+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:09.939004+0000) 2024-04-27T03:26:10.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:10.033+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:10.034980+0000) 2024-04-27T03:26:10.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:10.032+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:10.034323+0000) 2024-04-27T03:26:10.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:10.034+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:10.036417+0000) 2024-04-27T03:26:10.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:10.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:10.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:10.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:10.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:10.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:10.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:10.055+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:10.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:10.702+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:10.703234+0000) 2024-04-27T03:26:10.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:10.599+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:10.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:10.599+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:10.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:10.599+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:10.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:10.601+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:10.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:10.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:10.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:10.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:10.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:10.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:10.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:10.546+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:10.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:10.755+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:10.755933+0000) 2024-04-27T03:26:11.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:10.937+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:10.939246+0000) 2024-04-27T03:26:11.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:11.079+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:11.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:11.079+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:11.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:11.079+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:11.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:11.080+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:11.763 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:11.705+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:11.706653+0000) 2024-04-27T03:26:11.764 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:11.562+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:11.764 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:11.562+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:11.764 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:11.562+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:11.764 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:11.562+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:11.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:11.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:11.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:11.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:11.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:11.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:11.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:11.506+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:12.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:11.757+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:11.758158+0000) 2024-04-27T03:26:12.033 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:11.940+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:11.941601+0000) 2024-04-27T03:26:12.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:12.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:12.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:12.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:12.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:12.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:12.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:12.110+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:12.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:12.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:12.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:12.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:12.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:12.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:12.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:12.500+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:12.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:12.552+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:12.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:12.552+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:12.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:12.552+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:12.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:12.553+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:12.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:12.708+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:12.709380+0000) 2024-04-27T03:26:12.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:12.760+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:12.761313+0000) 2024-04-27T03:26:12.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:12.943+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:12.943916+0000) 2024-04-27T03:26:13.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:13.127+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:13.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:13.127+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:13.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:13.127+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:13.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:13.128+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:13.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:13.543+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:13.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:13.543+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:13.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:13.543+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:13.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:13.544+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:13.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:13.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:13.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:13.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:13.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:13.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:13.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:13.501+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:13.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:13.711+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:13.711945+0000) 2024-04-27T03:26:13.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:13.763+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:13.764398+0000) 2024-04-27T03:26:13.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:13.946+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:13.947098+0000) 2024-04-27T03:26:14.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:14.117+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:14.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:14.117+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:14.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:14.117+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:14.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:14.118+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:14.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:14.515+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:14.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:14.515+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:14.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:14.515+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:14.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:14.516+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:14.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:14.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:14.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:14.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:14.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:14.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:14.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:14.457+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:14.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:14.713+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:14.713837+0000) 2024-04-27T03:26:14.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:14.765+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:14.765889+0000) 2024-04-27T03:26:14.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:14.947+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:14.948668+0000) 2024-04-27T03:26:15.409 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:15.041+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:15.042037+0000) 2024-04-27T03:26:15.409 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:26:15.044+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62221 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:26:15.410 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:15.041+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:15.041957+0000) 2024-04-27T03:26:15.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:15.042+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:15.043083+0000) 2024-04-27T03:26:15.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:15.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:15.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:15.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:15.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:15.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:15.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:15.136+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:15.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:15.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:15.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:15.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:15.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:15.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:15.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:15.537+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:15.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:15.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:15.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:15.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:15.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:15.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:15.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:15.486+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:15.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:15.713+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:15.714054+0000) 2024-04-27T03:26:15.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:15.948+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:15.948915+0000) 2024-04-27T03:26:15.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:15.765+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:15.766139+0000) 2024-04-27T03:26:16.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:16.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:16.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:16.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:16.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:16.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:16.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:16.113+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:16.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:16.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:16.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:16.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:16.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:16.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:16.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:16.520+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:16.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:16.543+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:16.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:16.543+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:16.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:16.543+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:16.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:16.543+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:16.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:16.767+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:16.768229+0000) 2024-04-27T03:26:16.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:16.950+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:16.951014+0000) 2024-04-27T03:26:16.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:16.715+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:16.716510+0000) 2024-04-27T03:26:17.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:17.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:17.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:17.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:17.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:17.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:17.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:17.110+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:17.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:17.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:17.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:17.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:17.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:17.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:17.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:17.568+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:17.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:17.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:17.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:17.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:17.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:17.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:17.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:17.484+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:17.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:17.718+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:17.719438+0000) 2024-04-27T03:26:17.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:17.770+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:17.771619+0000) 2024-04-27T03:26:17.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:17.952+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:17.953257+0000) 2024-04-27T03:26:18.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:18.095+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:18.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:18.095+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:18.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:18.095+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:18.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:18.096+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:18.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:18.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:18.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:18.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:18.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:18.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:18.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:18.461+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:18.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:18.618+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:18.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:18.618+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:18.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:18.618+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:18.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:18.618+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:19.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:18.773+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:18.774736+0000) 2024-04-27T03:26:19.034 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:18.954+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:18.955379+0000) 2024-04-27T03:26:19.034 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:18.721+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:18.722647+0000) 2024-04-27T03:26:19.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:19.071+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:19.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:19.071+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:19.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:19.071+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:19.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:19.072+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:19.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:19.647+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:19.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:19.647+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:19.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:19.647+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:19.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:19.648+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:19.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:19.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:19.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:19.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:19.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:19.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:19.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:19.475+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:20.034 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:19.723+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:19.724033+0000) 2024-04-27T03:26:20.034 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:19.775+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:19.776114+0000) 2024-04-27T03:26:20.034 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:19.956+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:19.957303+0000) 2024-04-27T03:26:20.289 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:26:20.052+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62261 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:26:20.290 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:20.049+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:20.050263+0000) 2024-04-27T03:26:20.290 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:20.048+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:20.049276+0000) 2024-04-27T03:26:20.290 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:20.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:20.290 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:20.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:20.290 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:20.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:20.290 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:20.093+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:20.291 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:20.048+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:20.049625+0000) 2024-04-27T03:26:20.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:20.429+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:20.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:20.429+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:20.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:20.429+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:20.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:20.430+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:20.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:20.723+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:20.724311+0000) 2024-04-27T03:26:20.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:20.775+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:20.776387+0000) 2024-04-27T03:26:20.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:20.667+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:20.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:20.667+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:20.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:20.667+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:20.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:20.667+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:21.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:20.956+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:20.957501+0000) 2024-04-27T03:26:21.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:21.114+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:21.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:21.114+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:21.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:21.114+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:21.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:21.115+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:21.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:21.469+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:21.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:21.469+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:21.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:21.469+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:21.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:21.469+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:21.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:21.777+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:21.778550+0000) 2024-04-27T03:26:21.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:21.727+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:21.728500+0000) 2024-04-27T03:26:21.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:21.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:21.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:21.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:21.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:21.711+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:21.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:21.711+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:21.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:21.958+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:21.959739+0000) 2024-04-27T03:26:22.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:22.073+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:22.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:22.073+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:22.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:22.073+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:22.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:22.074+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:22.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:22.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:22.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:22.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:22.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:22.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:22.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:22.484+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:22.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:22.730+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:22.730798+0000) 2024-04-27T03:26:22.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:22.782+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:22.782840+0000) 2024-04-27T03:26:22.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:22.671+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:22.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:22.671+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:22.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:22.671+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:22.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:22.671+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:23.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:22.963+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:22.964487+0000) 2024-04-27T03:26:23.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:23.057+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:23.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:23.057+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:23.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:23.057+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:23.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:23.058+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:23.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:23.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:23.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:23.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:23.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:23.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:23.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:23.464+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:23.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:23.732+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:23.732822+0000) 2024-04-27T03:26:23.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:23.661+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:23.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:23.661+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:23.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:23.661+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:23.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:23.662+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:23.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:23.786+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:23.787591+0000) 2024-04-27T03:26:24.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:23.967+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:23.968616+0000) 2024-04-27T03:26:24.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:24.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:24.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:24.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:24.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:24.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:24.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:24.048+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:24.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:24.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:24.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:24.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:24.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:24.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:24.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:24.443+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:24.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:24.733+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:24.734331+0000) 2024-04-27T03:26:24.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:24.788+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:24.789262+0000) 2024-04-27T03:26:24.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:24.624+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:24.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:24.624+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:24.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:24.624+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:24.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:24.624+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:25.229 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:26:25.060+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62300 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:26:25.229 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:25.057+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:25.058077+0000) 2024-04-27T03:26:25.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:24.970+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:24.971731+0000) 2024-04-27T03:26:25.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:25.057+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:25.058027+0000) 2024-04-27T03:26:25.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:25.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:25.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:25.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:25.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:25.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:25.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:25.008+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:25.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:25.057+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:25.058310+0000) 2024-04-27T03:26:25.794 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:25.733+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:25.734548+0000) 2024-04-27T03:26:25.795 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:25.673+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:25.795 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:25.673+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:25.795 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:25.673+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:25.795 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:25.673+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:25.795 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:25.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:25.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:25.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:25.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:25.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:25.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:25.491+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:26.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:25.788+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:25.789455+0000) 2024-04-27T03:26:26.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:25.971+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:25.971944+0000) 2024-04-27T03:26:26.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:26.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:26.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:26.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:26.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:26.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:26.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:26.053+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:26.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:26.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:26.669 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:26.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:26.669 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:26.488+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:26.669 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:26.489+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:26.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:26.735+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:26.736654+0000) 2024-04-27T03:26:26.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:26.791+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:26.792639+0000) 2024-04-27T03:26:26.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:26.662+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:26.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:26.662+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:26.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:26.662+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:26.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:26.663+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:27.229 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:26.973+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:26.974141+0000) 2024-04-27T03:26:27.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:27.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:27.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:27.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:27.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:27.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:27.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:27.037+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:27.800 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:27.740+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:27.740888+0000) 2024-04-27T03:26:27.801 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:27.631+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:27.801 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:27.631+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:27.801 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:27.631+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:27.801 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:27.632+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:27.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:27.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:27.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:27.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:27.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:27.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:27.802 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:27.499+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:28.119 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:27.976+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:27.977203+0000) 2024-04-27T03:26:28.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:27.794+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:27.795386+0000) 2024-04-27T03:26:28.119 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:27.990+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:28.119 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:27.990+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:28.119 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:27.990+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:28.120 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:27.991+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:28.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:28.600+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:28.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:28.600+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:28.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:28.600+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:28.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:28.600+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:28.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:28.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:28.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:28.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:28.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:28.530+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:28.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:28.531+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:29.034 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:28.978+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:28.979359+0000) 2024-04-27T03:26:29.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:28.742+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:28.743246+0000) 2024-04-27T03:26:29.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:28.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:29.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:28.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:29.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:28.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:29.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:28.973+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:29.035 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:28.797+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:28.797840+0000) 2024-04-27T03:26:29.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:29.551+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:29.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:29.551+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:29.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:29.551+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:29.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:29.552+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:29.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:29.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:29.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:29.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:29.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:29.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:29.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:29.502+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:30.035 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:29.798+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:29.799395+0000) 2024-04-27T03:26:30.035 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:29.980+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:29.980997+0000) 2024-04-27T03:26:30.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:29.743+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:29.744774+0000) 2024-04-27T03:26:30.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:29.934+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:30.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:29.934+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:30.035 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:29.934+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:30.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:29.935+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:30.289 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:26:30.067+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62340 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:26:30.290 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:30.063+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:30.064400+0000) 2024-04-27T03:26:30.290 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:30.065+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:30.066364+0000) 2024-04-27T03:26:30.290 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:30.064+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:30.065612+0000) 2024-04-27T03:26:30.548 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:30.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:30.548 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:30.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:30.548 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:30.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:30.548 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:30.462+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:30.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:30.744+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:30.744969+0000) 2024-04-27T03:26:30.805 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:30.542+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:30.805 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:30.542+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:30.805 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:30.542+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:30.805 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:30.542+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:31.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:30.980+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:30.981186+0000) 2024-04-27T03:26:31.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:30.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:31.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:30.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:31.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:30.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:31.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:30.955+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:31.109 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:30.798+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:30.799566+0000) 2024-04-27T03:26:31.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:31.496+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:31.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:31.496+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:31.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:31.496+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:31.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:31.496+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:31.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:31.498+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:31.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:31.498+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:31.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:31.498+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:31.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:31.498+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:32.010 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:31.801+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:31.801851+0000) 2024-04-27T03:26:32.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:31.982+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:31.983280+0000) 2024-04-27T03:26:32.011 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:31.746+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:31.747077+0000) 2024-04-27T03:26:32.011 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:32.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:32.011 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:32.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:32.011 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:32.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:32.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:32.005+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:32.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:32.541+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:32.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:32.541+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:32.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:32.541+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:32.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:32.541+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:32.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:32.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:32.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:32.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:32.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:32.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:32.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:32.516+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:33.022 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:32.748+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:32.749131+0000) 2024-04-27T03:26:33.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:32.984+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:32.985739+0000) 2024-04-27T03:26:33.022 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:32.803+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:32.804561+0000) 2024-04-27T03:26:33.306 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:33.015+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:33.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:33.015+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:33.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:33.015+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:33.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:33.016+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:33.593 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:33.492+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:33.593 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:33.492+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:33.593 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:33.492+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:33.593 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:33.492+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:33.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:33.750+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:33.751246+0000) 2024-04-27T03:26:33.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:33.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:33.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:33.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:33.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:33.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:33.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:33.587+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:33.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:33.805+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:33.806647+0000) 2024-04-27T03:26:34.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:33.988+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:33.988967+0000) 2024-04-27T03:26:34.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:34.006+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:34.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:34.006+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:34.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:34.006+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:34.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:34.007+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:34.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:34.448+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:34.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:34.448+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:34.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:34.448+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:34.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:34.448+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:34.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:34.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:34.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:34.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:34.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:34.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:34.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:34.598+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:35.035 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:34.989+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:34.990448+0000) 2024-04-27T03:26:35.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:34.808+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:34.809750+0000) 2024-04-27T03:26:35.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:34.751+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:34.752702+0000) 2024-04-27T03:26:35.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:34.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:35.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:34.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:35.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:34.966+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:35.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:34.966+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:35.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:35.072+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:35.073664+0000) 2024-04-27T03:26:35.411 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:26:35.075+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62379 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:26:35.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:35.071+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:35.072543+0000) 2024-04-27T03:26:35.412 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:35.071+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:35.072543+0000) 2024-04-27T03:26:35.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:35.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:35.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:35.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:35.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:35.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:35.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:35.430+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:35.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:35.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:35.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:35.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:35.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:35.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:35.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:35.605+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:36.035 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:35.809+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:35.809943+0000) 2024-04-27T03:26:36.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:35.989+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:35.990733+0000) 2024-04-27T03:26:36.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:35.752+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:35.752914+0000) 2024-04-27T03:26:36.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:35.956+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:36.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:35.957+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:36.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:35.957+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:36.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:35.957+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:36.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:36.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:36.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:36.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:36.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:36.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:36.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:36.447+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:36.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:36.755+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:36.756120+0000) 2024-04-27T03:26:36.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:36.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:36.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:36.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:36.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:36.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:36.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:36.652+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:36.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:36.811+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:36.812046+0000) 2024-04-27T03:26:37.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:36.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:37.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:36.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:37.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:36.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:37.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:36.975+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:37.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:36.993+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:36.993909+0000) 2024-04-27T03:26:37.764 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:37.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:37.764 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:37.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:37.764 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:37.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:37.764 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:37.483+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:37.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:37.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:37.765 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:37.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:37.765 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:37.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:37.765 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:37.630+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:38.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:37.997+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:37.998200+0000) 2024-04-27T03:26:38.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:37.813+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:37.814212+0000) 2024-04-27T03:26:38.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:37.758+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:37.758855+0000) 2024-04-27T03:26:38.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:38.005+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:38.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:38.005+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:38.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:38.005+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:38.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:38.006+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:38.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:38.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:38.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:38.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:38.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:38.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:38.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:38.446+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:38.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:38.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:38.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:38.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:38.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:38.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:38.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:38.627+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:39.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:39.000+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:39.001072+0000) 2024-04-27T03:26:39.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:38.815+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:38.816780+0000) 2024-04-27T03:26:39.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:38.761+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:38.762283+0000) 2024-04-27T03:26:39.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:38.985+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:39.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:38.985+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:39.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:38.985+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:39.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:38.986+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:39.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:39.650+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:39.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:39.650+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:39.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:39.650+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:39.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:39.651+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:39.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:39.447+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:39.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:39.447+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:39.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:39.447+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:39.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:39.447+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:40.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:39.818+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:39.818991+0000) 2024-04-27T03:26:40.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:40.003+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:40.003899+0000) 2024-04-27T03:26:40.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:39.762+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:39.763764+0000) 2024-04-27T03:26:40.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:39.960+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:40.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:39.960+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:40.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:39.960+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:40.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:39.961+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:40.290 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:26:40.082+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62419 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:26:40.290 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:40.080+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:40.081331+0000) 2024-04-27T03:26:40.291 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:40.080+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:40.080939+0000) 2024-04-27T03:26:40.291 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:40.080+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:40.080909+0000) 2024-04-27T03:26:40.630 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:40.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:40.630 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:40.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:40.630 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:40.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:40.630 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:40.403+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:40.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:40.763+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:40.764057+0000) 2024-04-27T03:26:40.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:40.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:40.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:40.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:40.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:40.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:40.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:40.626+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:40.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:40.818+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:40.819289+0000) 2024-04-27T03:26:41.184 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:41.003+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:41.004187+0000) 2024-04-27T03:26:41.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:40.915+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:41.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:40.915+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:41.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:40.915+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:41.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:40.916+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:41.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:41.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:41.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:41.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:41.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:41.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:41.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:41.373+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:41.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:41.768+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:41.768840+0000) 2024-04-27T03:26:41.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:41.884+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:41.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:41.884+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:41.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:41.884+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:41.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:41.886+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:41.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:41.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:41.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:41.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:41.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:41.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:41.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:41.649+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:41.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:41.820+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:41.821403+0000) 2024-04-27T03:26:42.379 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:42.005+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:42.006281+0000) 2024-04-27T03:26:42.379 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:42.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:42.379 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:42.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:42.379 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:42.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:42.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:42.373+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:42.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:42.690+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:42.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:42.690+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:42.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:42.690+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:42.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:42.691+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:42.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:42.822+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:42.823580+0000) 2024-04-27T03:26:42.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:42.772+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:42.773089+0000) 2024-04-27T03:26:42.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:42.849+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:42.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:42.849+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:42.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:42.849+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:42.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:42.849+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:43.349 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:43.008+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:43.009308+0000) 2024-04-27T03:26:43.349 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:43.342+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:43.349 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:43.342+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:43.349 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:43.342+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:43.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:43.343+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:43.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:43.774+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:43.775197+0000) 2024-04-27T03:26:43.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:43.869+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:43.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:43.869+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:43.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:43.869+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:43.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:43.870+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:43.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:43.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:43.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:43.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:43.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:43.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:43.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:43.713+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:43.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:43.825+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:43.825885+0000) 2024-04-27T03:26:44.332 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:44.011+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:44.012045+0000) 2024-04-27T03:26:44.332 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:44.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:44.332 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:44.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:44.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:44.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:44.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:44.326+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:45.018 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:44.777+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:44.777934+0000) 2024-04-27T03:26:45.019 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:44.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:45.019 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:44.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:45.019 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:44.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:45.019 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:44.852+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:45.019 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:44.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:45.019 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:44.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:45.019 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:44.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:45.019 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:44.748+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:45.019 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:44.826+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:44.827338+0000) 2024-04-27T03:26:45.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:45.087+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:45.088461+0000) 2024-04-27T03:26:45.381 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:26:45.089+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62458 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:26:45.381 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:45.087+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:45.088284+0000) 2024-04-27T03:26:45.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:45.012+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:45.013470+0000) 2024-04-27T03:26:45.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:45.087+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:45.087852+0000) 2024-04-27T03:26:45.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:45.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:45.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:45.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:45.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:45.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:45.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:45.375+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:46.037 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:46.012+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:46.013740+0000) 2024-04-27T03:26:46.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:45.777+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:45.778121+0000) 2024-04-27T03:26:46.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:45.806+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:46.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:45.806+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:46.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:45.806+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:46.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:45.807+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:46.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:45.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:46.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:45.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:46.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:45.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:46.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:45.791+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:46.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:45.826+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:45.827532+0000) 2024-04-27T03:26:46.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:46.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:46.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:46.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:46.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:46.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:46.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:46.367+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:47.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:46.779+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:46.780166+0000) 2024-04-27T03:26:47.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:46.804+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:47.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:46.804+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:47.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:46.804+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:47.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:46.805+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:47.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:47.015+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:47.015899+0000) 2024-04-27T03:26:47.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:46.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:47.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:46.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:47.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:46.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:47.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:46.797+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:47.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:46.829+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:46.830476+0000) 2024-04-27T03:26:47.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:47.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:47.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:47.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:47.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:47.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:47.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:47.354+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:48.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:48.017+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:48.018068+0000) 2024-04-27T03:26:48.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:47.831+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:47.832601+0000) 2024-04-27T03:26:48.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:47.833+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:48.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:47.833+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:48.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:47.833+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:48.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:47.834+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:48.122 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:47.784+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:47.784917+0000) 2024-04-27T03:26:48.122 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:47.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:48.122 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:47.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:48.122 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:47.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:48.123 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:47.787+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:48.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:48.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:48.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:48.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:48.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:48.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:48.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:48.331+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:49.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:49.022+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:49.022911+0000) 2024-04-27T03:26:49.121 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:48.787+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:48.787823+0000) 2024-04-27T03:26:49.122 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:48.809+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:49.122 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:48.809+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:49.122 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:48.809+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:49.122 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:48.810+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:49.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:48.834+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:48.835430+0000) 2024-04-27T03:26:49.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:48.861+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:49.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:48.861+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:49.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:48.861+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:49.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:48.862+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:49.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:49.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:49.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:49.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:49.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:49.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:49.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:49.298+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:49.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T03:26:49.610+0000 7ff0e4ed9700 0 monclient(hunting): authenticate timed out after 300 2024-04-27T03:26:50.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:49.788+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:49.789298+0000) 2024-04-27T03:26:50.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:49.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:50.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:49.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:50.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:49.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:50.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:49.828+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:50.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:50.025+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:50.026079+0000) 2024-04-27T03:26:50.098 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:49.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:50.098 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:49.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:50.098 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:49.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:50.098 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:49.816+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:50.098 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:49.836+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:49.837384+0000) 2024-04-27T03:26:50.413 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:50.094+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:50.095032+0000) 2024-04-27T03:26:50.413 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:26:50.096+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62498 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:26:50.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:50.093+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:50.094441+0000) 2024-04-27T03:26:50.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:50.094+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:50.095063+0000) 2024-04-27T03:26:50.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:50.306+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:50.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:50.306+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:50.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:50.306+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:50.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:50.306+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:51.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:51.025+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:51.026281+0000) 2024-04-27T03:26:51.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:50.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:51.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:50.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:51.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:50.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:51.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:50.826+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:51.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:50.836+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:50.837564+0000) 2024-04-27T03:26:51.123 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:50.788+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:50.789501+0000) 2024-04-27T03:26:51.123 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:50.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:51.123 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:50.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:51.123 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:50.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:51.123 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:50.839+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:51.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:51.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:51.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:51.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:51.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:51.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:51.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:51.313+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:52.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:52.027+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:52.028510+0000) 2024-04-27T03:26:52.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:51.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:52.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:51.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:52.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:51.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:52.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:51.802+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:52.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:51.839+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:51.840137+0000) 2024-04-27T03:26:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:51.790+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:51.791573+0000) 2024-04-27T03:26:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:51.879+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:51.879+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:51.879+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:52.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:51.880+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:52.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:52.300+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:52.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:52.300+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:52.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:52.300+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:52.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:52.300+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:53.037 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:52.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:53.037 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:52.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:53.037 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:52.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:53.037 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:52.759+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:53.037 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:52.841+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:52.842420+0000) 2024-04-27T03:26:53.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:52.793+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:52.793853+0000) 2024-04-27T03:26:53.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:52.868+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:53.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:52.868+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:53.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:52.868+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:53.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:52.868+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:53.308 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:53.030+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:53.031768+0000) 2024-04-27T03:26:53.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:53.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:53.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:53.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:53.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:53.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:53.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:53.303+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:54.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:53.795+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:53.795985+0000) 2024-04-27T03:26:54.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:53.898+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:54.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:53.898+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:54.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:53.898+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:54.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:53.899+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:54.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:53.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:54.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:53.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:54.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:53.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:54.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:53.771+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:54.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:53.845+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:53.846447+0000) 2024-04-27T03:26:54.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:54.035+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:54.036091+0000) 2024-04-27T03:26:54.356 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:54.349+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:54.356 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:54.349+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:54.356 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:54.349+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:54.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:54.350+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:55.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:54.797+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:54.797954+0000) 2024-04-27T03:26:55.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:54.870+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:55.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:54.870+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:55.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:54.871+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:55.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:54.871+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:55.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:54.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:55.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:54.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:55.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:54.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:55.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:54.782+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:55.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:54.848+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:54.849472+0000) 2024-04-27T03:26:55.378 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:26:55.104+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62538 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:26:55.379 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:55.101+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:55.102615+0000) 2024-04-27T03:26:55.379 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:55.036+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:55.037739+0000) 2024-04-27T03:26:55.379 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:55.101+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:55.102343+0000) 2024-04-27T03:26:55.379 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:55.102+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:55.103329+0000) 2024-04-27T03:26:55.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:55.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:55.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:55.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:55.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:55.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:55.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:55.373+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:56.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:55.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:56.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:55.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:56.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:55.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:56.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:55.741+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:56.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:55.848+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:55.849742+0000) 2024-04-27T03:26:56.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:55.797+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:55.798183+0000) 2024-04-27T03:26:56.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:55.913+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:56.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:55.913+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:56.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:55.913+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:56.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:55.915+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:56.372 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:56.037+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:56.037936+0000) 2024-04-27T03:26:56.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:56.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:56.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:56.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:56.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:56.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:56.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:56.367+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:56.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:56.799+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:56.800505+0000) 2024-04-27T03:26:56.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:56.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:56.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:56.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:56.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:56.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:56.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:56.942+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:56.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:56.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:56.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:56.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:56.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:56.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:56.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:56.713+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:56.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:56.852+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:56.852998+0000) 2024-04-27T03:26:57.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:57.040+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:57.041099+0000) 2024-04-27T03:26:57.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:57.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:57.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:57.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:57.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:57.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:57.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:57.382+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:57.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:57.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:57.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:57.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:57.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:57.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:57.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:57.688+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:57.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:57.854+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:57.855253+0000) 2024-04-27T03:26:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:57.804+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:57.805284+0000) 2024-04-27T03:26:58.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:58.042+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:58.043779+0000) 2024-04-27T03:26:58.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:58.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:58.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:58.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:58.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:58.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:58.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:57.978+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:58.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:57.978+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:58.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:57.978+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:58.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:57.979+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:58.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:58.336+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:58.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:58.806+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:58.807405+0000) 2024-04-27T03:26:58.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:58.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:58.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:58.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:58.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:58.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:58.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:58.695+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:58.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:58.856+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:58.857486+0000) 2024-04-27T03:26:59.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:58.977+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:59.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:58.977+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:59.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:58.977+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:26:59.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:58.978+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:59.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:59.045+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:59.045877+0000) 2024-04-27T03:26:59.378 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:59.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:59.378 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:59.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:59.378 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:59.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:26:59.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:26:59.371+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:59.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:59.808+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:59.808836+0000) 2024-04-27T03:26:59.961 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:59.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:59.961 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:59.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:59.961 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:59.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:26:59.961 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:59.693+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:26:59.961 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:26:59.858+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:26:59.859281+0000) 2024-04-27T03:27:00.224 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:27:00.112+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62577 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:27:00.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:00.109+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:00.110279+0000) 2024-04-27T03:27:00.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:00.047+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:00.048717+0000) 2024-04-27T03:27:00.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:00.108+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:00.109601+0000) 2024-04-27T03:27:00.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:59.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:00.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:59.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:00.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:59.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:00.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:26:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:26:59.956+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:00.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:00.109+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:00.110549+0000) 2024-04-27T03:27:00.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:00.377+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:00.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:00.377+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:00.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:00.377+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:00.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:00.377+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:00.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:00.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:00.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:00.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:00.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:00.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:00.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:00.717+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:00.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:00.857+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:00.859451+0000) 2024-04-27T03:27:00.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:00.807+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:00.809050+0000) 2024-04-27T03:27:00.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:00.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:00.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:00.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:00.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:00.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:00.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:00.926+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:01.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:01.047+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:01.048905+0000) 2024-04-27T03:27:01.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:01.385+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:01.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:01.385+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:01.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:01.385+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:01.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:01.386+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:02.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:01.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:02.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:01.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:02.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:01.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:02.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:01.723+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:02.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:01.862+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:01.864444+0000) 2024-04-27T03:27:02.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:01.809+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:01.811157+0000) 2024-04-27T03:27:02.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:01.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:02.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:01.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:02.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:01.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:02.040 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:01.968+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:02.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:02.052+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:02.053958+0000) 2024-04-27T03:27:02.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:02.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:02.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:02.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:02.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:02.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:02.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:02.406+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:02.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:02.811+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:02.813251+0000) 2024-04-27T03:27:02.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:02.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:02.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:02.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:02.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:02.937+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:02.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:02.938+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:02.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:02.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:02.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:02.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:02.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:02.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:02.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:02.689+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:02.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:02.867+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:02.869494+0000) 2024-04-27T03:27:03.231 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:03.057+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:03.059027+0000) 2024-04-27T03:27:03.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:03.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:03.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:03.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:03.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:03.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:03.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:03.381+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:03.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:03.813+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:03.815385+0000) 2024-04-27T03:27:03.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:03.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:03.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:03.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:03.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:03.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:03.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:03.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:03.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:03.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:03.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:03.669+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:03.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:03.872+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:03.874546+0000) 2024-04-27T03:27:04.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:04.062+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:04.064092+0000) 2024-04-27T03:27:04.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:03.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:04.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:03.969+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:04.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:04.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:04.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:04.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:04.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:04.375+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:04.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:04.375+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:04.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:04.815+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:04.817428+0000) 2024-04-27T03:27:04.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:04.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:04.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:04.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:04.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:04.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:04.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:04.631+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:04.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:04.876+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:04.877953+0000) 2024-04-27T03:27:05.226 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:27:05.118+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62618 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:27:05.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:05.116+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:05.117976+0000) 2024-04-27T03:27:05.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:05.065+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:05.067556+0000) 2024-04-27T03:27:05.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:05.116+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:05.118507+0000) 2024-04-27T03:27:05.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:04.948+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:05.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:04.948+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:05.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:04.948+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:05.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:04.949+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:05.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:05.116+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:05.118083+0000) 2024-04-27T03:27:05.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:05.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:05.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:05.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:05.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:05.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:05.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:05.369+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:05.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:05.815+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:05.817661+0000) 2024-04-27T03:27:05.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:05.612+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:05.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:05.612+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:05.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:05.612+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:05.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:05.613+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:06.187 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:06.066+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:06.067869+0000) 2024-04-27T03:27:06.187 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:05.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:06.187 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:05.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:06.187 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:05.985+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:06.187 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:05.985+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:06.187 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:05.876+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:05.878215+0000) 2024-04-27T03:27:06.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:06.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:06.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:06.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:06.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:06.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:06.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:06.351+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:06.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:06.817+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:06.819785+0000) 2024-04-27T03:27:06.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:06.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:06.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:06.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:06.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:06.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:06.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:06.659+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:06.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:06.878+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:06.880351+0000) 2024-04-27T03:27:07.337 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:07.034+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:07.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:07.034+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:07.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:07.034+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:07.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:07.036+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:07.338 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:07.068+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:07.070071+0000) 2024-04-27T03:27:07.338 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:07.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:07.338 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:07.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:07.338 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:07.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:07.621 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:07.331+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:07.888 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:07.822+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:07.824002+0000) 2024-04-27T03:27:07.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:07.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:07.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:07.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:07.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:07.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:07.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:07.615+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:08.187 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:08.071+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:08.073157+0000) 2024-04-27T03:27:08.187 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:08.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:08.187 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:08.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:08.187 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:08.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:08.187 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:08.055+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:08.188 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:07.881+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:07.883304+0000) 2024-04-27T03:27:08.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:08.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:08.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:08.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:08.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:08.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:08.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:08.309+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:08.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:08.824+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:08.826297+0000) 2024-04-27T03:27:08.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:08.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:08.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:08.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:08.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:08.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:08.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:08.659+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:08.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:08.884+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:08.886262+0000) 2024-04-27T03:27:09.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:09.073+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:09.075577+0000) 2024-04-27T03:27:09.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:09.327+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:09.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:09.327+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:09.335 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:09.327+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:09.335 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:09.068+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:09.335 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:09.068+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:09.335 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:09.068+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:09.335 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:09.069+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:09.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:09.327+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:09.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:09.826+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:09.828076+0000) 2024-04-27T03:27:09.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:09.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:09.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:09.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:09.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:09.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:09.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:09.697+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:09.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:09.885+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:09.887764+0000) 2024-04-27T03:27:10.291 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:27:10.127+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62658 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:27:10.291 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:10.123+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:10.125111+0000) 2024-04-27T03:27:10.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:10.075+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:10.077535+0000) 2024-04-27T03:27:10.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:10.123+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:10.124920+0000) 2024-04-27T03:27:10.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:10.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:10.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:10.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:10.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:10.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:10.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:10.103+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:10.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:10.124+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:10.126682+0000) 2024-04-27T03:27:10.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:10.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:10.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:10.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:10.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:10.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:10.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:10.373+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:10.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:10.826+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:10.828284+0000) 2024-04-27T03:27:10.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:10.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:10.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:10.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:10.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:10.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:10.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:10.668+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:10.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:10.886+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:10.887984+0000) 2024-04-27T03:27:11.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:11.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:11.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:11.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:11.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:11.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:11.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:11.112+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:11.381 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:11.075+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:11.077772+0000) 2024-04-27T03:27:11.381 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:11.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:11.381 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:11.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:11.381 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:11.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:11.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:11.374+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:11.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:11.829+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:11.830878+0000) 2024-04-27T03:27:11.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:11.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:11.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:11.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:11.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:11.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:11.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:11.643+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:11.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:11.890+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:11.892759+0000) 2024-04-27T03:27:12.360 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:12.080+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:12.082421+0000) 2024-04-27T03:27:12.360 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:12.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:12.360 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:12.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:12.360 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:12.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:12.360 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:12.139+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:12.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:12.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:12.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:12.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:12.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:12.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:12.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:12.353+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:12.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:12.834+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:12.835676+0000) 2024-04-27T03:27:12.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:12.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:12.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:12.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:12.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:12.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:12.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:12.685+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:12.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:12.895+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:12.897525+0000) 2024-04-27T03:27:13.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:13.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:13.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:13.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:13.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:13.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:13.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:13.128+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:13.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:13.086+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:13.087115+0000) 2024-04-27T03:27:13.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:13.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:13.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:13.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:13.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:13.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:13.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:13.335+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:13.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:13.838+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:13.838957+0000) 2024-04-27T03:27:13.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:13.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:13.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:13.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:13.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:13.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:13.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:13.673+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:13.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:13.900+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:13.900956+0000) 2024-04-27T03:27:14.350 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:14.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:14.350 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:14.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:14.350 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:14.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:14.350 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:14.148+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:14.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:14.088+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:14.089446+0000) 2024-04-27T03:27:14.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:14.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:14.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:14.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:14.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:14.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:14.665 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:14.344+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:14.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:14.840+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:14.840964+0000) 2024-04-27T03:27:14.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:14.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:14.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:14.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:14.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:14.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:14.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:14.660+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:14.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:14.902+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:14.902984+0000) 2024-04-27T03:27:15.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:15.132+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:15.133369+0000) 2024-04-27T03:27:15.321 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:27:15.136+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62697 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:27:15.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:15.090+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:15.091466+0000) 2024-04-27T03:27:15.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:15.133+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:15.134196+0000) 2024-04-27T03:27:15.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:15.135+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:15.136403+0000) 2024-04-27T03:27:15.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:15.195+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:15.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:15.195+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:15.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:15.195+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:15.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:15.196+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:15.587 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:15.314+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:15.587 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:15.314+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:15.587 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:15.314+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:15.587 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:15.315+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:15.846 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:15.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:15.846 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:15.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:15.846 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:15.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:15.846 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:15.707+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:16.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:15.840+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:15.841153+0000) 2024-04-27T03:27:16.097 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:15.902+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:15.903144+0000) 2024-04-27T03:27:16.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:16.168+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:16.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:16.168+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:16.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:16.168+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:16.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:16.169+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:16.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:16.090+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:16.091653+0000) 2024-04-27T03:27:16.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:16.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:16.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:16.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:16.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:16.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:16.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:16.331+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:17.040 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:16.842+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:16.843720+0000) 2024-04-27T03:27:17.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:16.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:17.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:16.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:17.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:16.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:17.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:16.732+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:17.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:16.904+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:16.905425+0000) 2024-04-27T03:27:17.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:17.093+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:17.093866+0000) 2024-04-27T03:27:17.337 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:17.192+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:17.337 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:17.192+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:17.337 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:17.192+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:17.337 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:17.194+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:17.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:17.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:17.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:17.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:17.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:17.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:17.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:17.331+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:18.040 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:17.845+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:17.846001+0000) 2024-04-27T03:27:18.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:17.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:18.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:17.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:18.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:17.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:18.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:17.778+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:18.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:17.909+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:17.909840+0000) 2024-04-27T03:27:18.336 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:18.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:18.336 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:18.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:18.336 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:18.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:18.336 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:18.188+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:18.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:18.095+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:18.096140+0000) 2024-04-27T03:27:18.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:18.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:18.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:18.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:18.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:18.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:18.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:18.330+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:19.041 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:18.847+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:18.848203+0000) 2024-04-27T03:27:19.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:18.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:19.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:18.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:19.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:18.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:19.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:18.739+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:19.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:18.913+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:18.914480+0000) 2024-04-27T03:27:19.335 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:19.098+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:19.099400+0000) 2024-04-27T03:27:19.335 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:19.173+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:19.335 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:19.173+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:19.335 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:19.173+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:19.335 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:19.173+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:19.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:19.329+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:19.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:19.329+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:19.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:19.329+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:19.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:19.329+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:20.041 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:19.850+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:19.851095+0000) 2024-04-27T03:27:20.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:19.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:20.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:19.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:20.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:19.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:20.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:19.771+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:20.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:19.915+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:19.916073+0000) 2024-04-27T03:27:20.292 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:27:20.144+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62737 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:27:20.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:20.102+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:20.102820+0000) 2024-04-27T03:27:20.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:20.141+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:20.142261+0000) 2024-04-27T03:27:20.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:20.142+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:20.143016+0000) 2024-04-27T03:27:20.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:20.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:20.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:20.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:20.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:20.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:20.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:20.181+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:20.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:20.140+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:20.141396+0000) 2024-04-27T03:27:20.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:20.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:20.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:20.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:20.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:20.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:20.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:20.342+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:21.041 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:20.850+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:20.851269+0000) 2024-04-27T03:27:21.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:20.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:21.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:20.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:21.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:20.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:21.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:20.762+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:21.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:20.915+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:20.916280+0000) 2024-04-27T03:27:21.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:21.163+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:21.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:21.163+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:21.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:21.163+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:21.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:21.163+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:21.386 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:21.102+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:21.103023+0000) 2024-04-27T03:27:21.386 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:21.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:21.386 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:21.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:21.386 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:21.379+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:21.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:21.380+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:22.041 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:21.853+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:21.854232+0000) 2024-04-27T03:27:22.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:21.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:22.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:21.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:22.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:21.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:22.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:21.782+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:22.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:21.919+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:21.920554+0000) 2024-04-27T03:27:22.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:22.106+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:22.107255+0000) 2024-04-27T03:27:22.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:22.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:22.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:22.124+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:22.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:22.124+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:22.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:22.125+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:22.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:22.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:22.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:22.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:22.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:22.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:22.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:22.352+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:23.041 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:22.855+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:22.856650+0000) 2024-04-27T03:27:23.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:22.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:23.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:22.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:23.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:22.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:23.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:22.743+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:23.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:22.924+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:22.924840+0000) 2024-04-27T03:27:23.394 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:23.108+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:23.109468+0000) 2024-04-27T03:27:23.394 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:23.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:23.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:23.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:23.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:23.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:23.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:23.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:23.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:23.094+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:23.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:23.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:23.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:23.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:23.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:23.388+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:24.041 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:23.858+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:23.858933+0000) 2024-04-27T03:27:24.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:23.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:24.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:23.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:24.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:23.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:24.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:23.742+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:24.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:23.928+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:23.929178+0000) 2024-04-27T03:27:24.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:24.111+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:24.112307+0000) 2024-04-27T03:27:24.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:24.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:24.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:24.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:24.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:24.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:24.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:24.119+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:24.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:24.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:24.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:24.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:24.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:24.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:24.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:24.439+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:25.041 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:24.861+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:24.861847+0000) 2024-04-27T03:27:25.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:24.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:25.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:24.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:25.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:24.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:25.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:24.767+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:25.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:24.931+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:24.931839+0000) 2024-04-27T03:27:25.416 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:27:25.151+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62776 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:27:25.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:25.113+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:25.114388+0000) 2024-04-27T03:27:25.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:25.150+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:25.150912+0000) 2024-04-27T03:27:25.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:25.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:25.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:25.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:25.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:25.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:25.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:25.102+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:25.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:25.149+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:25.150012+0000) 2024-04-27T03:27:25.418 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:25.149+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:25.149855+0000) 2024-04-27T03:27:25.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:25.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:25.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:25.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:25.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:25.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:25.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:25.415+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:26.107 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:25.861+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:25.862050+0000) 2024-04-27T03:27:26.107 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:26.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:26.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:26.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:26.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:25.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:26.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:25.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:26.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:25.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:26.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:25.807+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:26.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:25.931+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:25.932061+0000) 2024-04-27T03:27:26.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:26.113+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:26.114567+0000) 2024-04-27T03:27:26.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:26.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:26.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:26.102+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:26.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:26.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:26.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:26.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:26.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:26.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:26.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:26.421+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:27.122 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:26.863+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:26.864628+0000) 2024-04-27T03:27:27.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:26.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:27.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:26.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:27.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:26.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:27.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:26.817+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:27.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:26.933+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:26.934699+0000) 2024-04-27T03:27:27.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:27.116+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:27.117646+0000) 2024-04-27T03:27:27.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:27.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:27.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:27.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:27.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:27.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:27.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:27.131+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:27.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:27.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:27.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:27.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:27.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:27.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:27.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:27.430+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:28.126 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:27.866+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:27.866897+0000) 2024-04-27T03:27:28.126 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:28.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:28.126 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:28.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:28.126 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:28.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:28.126 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:28.092+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:28.126 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:27.849+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:28.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:27.849+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:28.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:27.849+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:28.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:27.850+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:28.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:27.937+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:27.937841+0000) 2024-04-27T03:27:28.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:28.120+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:28.120835+0000) 2024-04-27T03:27:28.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:28.466+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:28.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:28.466+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:28.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:28.466+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:28.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:28.467+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:29.127 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:28.868+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:28.869207+0000) 2024-04-27T03:27:29.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:28.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:29.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:28.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:29.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:28.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:29.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:28.826+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:29.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:28.939+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:28.940663+0000) 2024-04-27T03:27:29.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:29.123+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:29.124087+0000) 2024-04-27T03:27:29.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:29.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:29.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:29.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:29.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:29.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:29.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:29.130+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:29.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:29.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:29.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:29.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:29.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:29.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:29.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:29.471+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:30.127 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:29.871+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:29.872155+0000) 2024-04-27T03:27:30.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:29.797+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:30.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:29.797+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:30.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:29.797+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:30.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:29.797+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:30.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:29.941+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:29.942243+0000) 2024-04-27T03:27:30.417 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:27:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:27:30.159+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62816 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:27:30.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:30.156+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:30.157231+0000) 2024-04-27T03:27:30.418 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:30.124+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:30.125756+0000) 2024-04-27T03:27:30.418 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:30.156+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:30.157280+0000) 2024-04-27T03:27:30.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:30.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:30.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:30.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:30.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:30.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:30.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:30.149+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:30.419 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:30.156+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:30.157716+0000) 2024-04-27T03:27:30.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:30.447+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:30.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:30.447+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:30.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:30.447+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:30.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:30.447+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:31.127 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:30.871+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:30.872386+0000) 2024-04-27T03:27:31.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:30.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:31.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:30.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:31.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:30.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:31.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:30.791+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:31.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:30.941+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:30.942426+0000) 2024-04-27T03:27:31.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:31.125+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:31.125954+0000) 2024-04-27T03:27:31.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:31.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:31.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:31.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:31.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:31.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:31.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:31.139+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:31.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:31.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:31.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:31.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:31.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:31.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:31.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:31.423+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:32.042 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:31.874+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:31.875126+0000) 2024-04-27T03:27:32.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:31.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:32.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:31.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:32.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:31.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:32.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:31.760+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:32.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:31.944+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:31.945517+0000) 2024-04-27T03:27:32.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:32.127+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:32.128204+0000) 2024-04-27T03:27:32.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:32.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:32.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:32.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:32.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:32.103+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:32.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:32.104+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:32.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:32.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:32.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:32.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:32.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:32.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:32.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:32.433+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:33.042 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:32.877+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:32.878095+0000) 2024-04-27T03:27:33.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:32.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:33.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:32.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:33.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:32.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:33.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:32.749+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:33.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:32.947+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:32.948297+0000) 2024-04-27T03:27:33.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:33.129+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:33.130325+0000) 2024-04-27T03:27:33.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:33.080+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:33.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:33.080+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:33.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:33.080+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:33.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:33.081+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:33.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:33.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:33.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:33.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:33.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:33.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:33.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:33.386+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:33.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:33.879+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:33.880738+0000) 2024-04-27T03:27:33.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:33.704+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:33.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:33.704+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:33.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:33.704+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:33.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:33.705+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:33.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:33.950+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:33.951056+0000) 2024-04-27T03:27:34.270 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:34.131+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:34.132638+0000) 2024-04-27T03:27:34.270 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:34.073+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:34.270 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:34.073+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:34.270 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:34.073+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:34.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:34.074+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:34.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:34.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:34.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:34.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:34.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:34.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:34.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:34.405+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:35.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:34.882+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:34.882794+0000) 2024-04-27T03:27:35.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:34.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:35.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:34.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:35.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:34.728+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:35.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:34.729+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:35.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:34.952+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:34.952943+0000) 2024-04-27T03:27:35.412 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:27:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:27:35.166+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62855 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:27:35.413 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:35.163+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:35.164243+0000) 2024-04-27T03:27:35.413 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:35.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:35.413 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:35.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:35.413 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:35.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:35.413 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:35.042+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:35.413 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:35.163+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:35.164300+0000) 2024-04-27T03:27:35.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:35.134+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:35.134824+0000) 2024-04-27T03:27:35.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:35.163+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:35.164143+0000) 2024-04-27T03:27:35.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:35.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:35.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:35.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:35.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:35.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:35.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:35.407+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:35.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:35.882+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:35.883056+0000) 2024-04-27T03:27:35.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:35.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:35.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:35.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:35.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:35.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:35.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:35.697+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:36.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:35.952+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:35.953216+0000) 2024-04-27T03:27:36.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:36.134+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:36.135014+0000) 2024-04-27T03:27:36.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:36.066+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:36.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:36.066+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:36.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:36.066+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:36.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:36.068+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:36.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:36.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:36.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:36.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:36.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:36.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:36.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:36.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:36.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:36.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:36.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:36.383+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:36.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:36.886+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:36.887248+0000) 2024-04-27T03:27:36.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:36.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:36.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:36.663+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:37.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:36.956+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:36.957393+0000) 2024-04-27T03:27:37.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:37.136+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:37.137079+0000) 2024-04-27T03:27:37.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:37.086+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:37.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:37.086+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:37.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:37.086+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:37.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:37.087+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:37.499 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:37.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:37.499 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:37.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:37.499 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:37.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:37.499 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:37.396+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:37.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:37.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:37.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:37.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:37.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:37.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:37.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:37.690+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:38.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:37.958+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:37.959559+0000) 2024-04-27T03:27:38.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:37.888+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:37.889348+0000) 2024-04-27T03:27:38.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:38.048+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:38.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:38.048+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:38.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:38.048+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:38.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:38.050+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:38.418 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:38.139+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:38.139901+0000) 2024-04-27T03:27:38.418 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:38.360+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:38.418 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:38.360+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:38.418 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:38.360+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:38.418 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:38.361+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:38.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:38.890+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:38.891517+0000) 2024-04-27T03:27:38.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:38.687+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:38.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:38.687+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:38.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:38.687+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:38.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:38.688+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:39.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:38.961+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:38.962609+0000) 2024-04-27T03:27:39.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:39.142+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:39.143062+0000) 2024-04-27T03:27:39.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:39.080+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:39.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:39.080+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:39.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:39.080+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:39.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:39.081+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:39.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:39.345+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:39.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:39.345+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:39.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:39.345+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:39.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:39.345+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:39.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:39.892+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:39.893234+0000) 2024-04-27T03:27:39.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:39.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:39.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:39.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:39.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:39.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:39.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:39.689+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:40.226 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:27:40.175+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62895 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:27:40.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:40.144+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:40.145281+0000) 2024-04-27T03:27:40.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:40.171+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:40.172258+0000) 2024-04-27T03:27:40.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:39.963+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:39.964322+0000) 2024-04-27T03:27:40.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:40.173+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:40.174227+0000) 2024-04-27T03:27:40.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:40.059+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:40.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:40.059+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:40.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:40.059+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:40.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:40.060+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:40.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:40.171+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:40.172576+0000) 2024-04-27T03:27:40.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:40.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:40.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:40.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:40.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:40.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:40.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:40.357+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:40.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:40.892+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:40.893425+0000) 2024-04-27T03:27:40.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:40.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:40.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:40.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:40.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:40.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:40.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:40.689+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:40.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:40.963+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:40.964534+0000) 2024-04-27T03:27:41.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:41.144+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:41.145446+0000) 2024-04-27T03:27:41.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:41.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:41.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:41.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:41.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:41.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:41.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:41.110+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:41.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:41.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:41.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:41.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:41.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:41.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:41.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:41.392+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:41.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:41.897+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:41.898000+0000) 2024-04-27T03:27:41.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:41.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:41.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:41.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:41.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:41.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:41.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:41.688+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:42.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:41.966+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:41.967630+0000) 2024-04-27T03:27:42.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:42.149+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:42.150313+0000) 2024-04-27T03:27:42.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:42.127+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:42.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:42.127+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:42.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:42.127+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:42.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:42.128+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:42.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:42.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:42.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:42.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:42.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:42.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:42.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:42.368+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:42.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:42.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:42.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:42.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:42.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:42.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:42.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:42.689+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:43.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:42.969+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:42.969915+0000) 2024-04-27T03:27:43.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:42.900+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:42.901495+0000) 2024-04-27T03:27:43.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:43.134+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:43.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:43.134+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:43.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:43.134+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:43.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:43.136+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:43.418 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:43.151+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:43.152606+0000) 2024-04-27T03:27:43.418 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:43.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:43.418 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:43.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:43.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:43.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:43.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:43.331+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:43.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:43.903+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:43.903815+0000) 2024-04-27T03:27:43.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:43.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:43.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:43.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:43.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:43.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:43.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:43.694+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:44.237 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:43.973+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:43.974236+0000) 2024-04-27T03:27:44.237 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:44.154+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:44.155614+0000) 2024-04-27T03:27:44.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:44.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:44.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:44.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:44.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:44.150+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:44.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:44.151+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:44.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:44.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:44.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:44.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:44.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:44.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:44.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:44.373+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:44.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:44.904+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:44.905359+0000) 2024-04-27T03:27:44.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:44.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:44.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:44.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:44.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:44.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:44.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:44.695+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:45.237 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:27:45.182+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62934 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:27:45.238 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:44.975+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:44.976246+0000) 2024-04-27T03:27:45.238 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:45.179+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:45.180520+0000) 2024-04-27T03:27:45.238 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:45.158+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:45.159289+0000) 2024-04-27T03:27:45.238 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:45.181+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:45.182158+0000) 2024-04-27T03:27:45.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:45.152+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:45.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:45.152+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:45.239 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:45.152+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:45.239 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:45.153+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:45.239 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:45.180+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:45.181283+0000) 2024-04-27T03:27:45.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:45.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:45.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:45.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:45.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:45.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:45.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:45.390+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:45.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:45.904+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:45.905628+0000) 2024-04-27T03:27:45.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:45.684+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:45.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:45.684+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:45.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:45.684+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:45.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:45.685+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:46.237 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:45.975+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:45.976444+0000) 2024-04-27T03:27:46.238 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:46.158+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:46.159496+0000) 2024-04-27T03:27:46.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:46.185+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:46.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:46.185+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:46.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:46.185+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:46.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:46.187+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:46.650 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:46.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:46.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:46.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:46.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:46.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:46.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:46.407+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:46.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:46.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:46.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:46.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:46.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:46.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:46.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:46.645+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:47.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:46.909+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:46.910653+0000) 2024-04-27T03:27:47.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:47.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:47.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:47.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:47.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:47.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:47.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:47.155+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:47.167 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:46.978+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:46.979318+0000) 2024-04-27T03:27:47.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:47.160+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:47.161601+0000) 2024-04-27T03:27:47.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:47.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:47.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:47.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:47.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:47.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:47.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:47.366+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:47.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:47.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:47.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:47.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:47.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:47.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:47.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:47.637+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:48.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:47.914+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:47.915624+0000) 2024-04-27T03:27:48.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:48.182+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:48.188 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:47.981+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:47.982264+0000) 2024-04-27T03:27:48.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:48.163+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:48.164359+0000) 2024-04-27T03:27:48.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:48.182+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:48.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:48.182+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:48.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:48.183+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:48.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:48.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:48.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:48.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:48.342+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:48.923 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:48.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:48.923 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:48.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:48.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:48.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:48.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:48.635+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:49.173 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:48.983+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:48.984427+0000) 2024-04-27T03:27:49.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:48.917+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:48.918493+0000) 2024-04-27T03:27:49.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:49.155+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:49.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:49.155+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:49.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:49.155+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:49.175 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:49.156+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:49.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:49.167+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:49.168649+0000) 2024-04-27T03:27:49.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:49.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:49.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:49.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:49.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:49.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:49.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:49.384+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:49.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:49.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:49.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:49.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:49.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:49.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:49.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:49.598+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:50.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:49.985+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:49.985959+0000) 2024-04-27T03:27:50.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:49.919+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:49.920362+0000) 2024-04-27T03:27:50.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:50.112+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:50.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:50.112+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:50.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:50.112+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:50.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:50.112+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:50.432 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:27:50.190+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 62974 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:27:50.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:50.188+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:50.189556+0000) 2024-04-27T03:27:50.432 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:50.188+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:50.189356+0000) 2024-04-27T03:27:50.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:50.170+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:50.171593+0000) 2024-04-27T03:27:50.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:50.187+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:50.187882+0000) 2024-04-27T03:27:50.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:50.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:50.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:50.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:50.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:50.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:50.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:50.426+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:50.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:50.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:50.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:50.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:50.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:50.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:50.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:50.625+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:51.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:50.985+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:50.986161+0000) 2024-04-27T03:27:51.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:50.919+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:50.920561+0000) 2024-04-27T03:27:51.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:51.125+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:51.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:51.125+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:51.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:51.125+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:51.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:51.126+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:51.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:51.171+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:51.171969+0000) 2024-04-27T03:27:51.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:51.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:51.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:51.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:51.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:51.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:51.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:51.381+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:51.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:51.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:51.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:51.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:51.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:51.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:51.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:51.628+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:52.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:51.987+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:51.988308+0000) 2024-04-27T03:27:52.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:52.174+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:52.174876+0000) 2024-04-27T03:27:52.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:51.924+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:51.924792+0000) 2024-04-27T03:27:52.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:52.080+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:52.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:52.080+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:52.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:52.080+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:52.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:52.081+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:52.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:52.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:52.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:52.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:52.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:52.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:52.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:52.411+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:52.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:52.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:52.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:52.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:52.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:52.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:52.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:52.581+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:53.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:52.989+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:52.990425+0000) 2024-04-27T03:27:53.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:53.177+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:53.178072+0000) 2024-04-27T03:27:53.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:52.927+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:52.928077+0000) 2024-04-27T03:27:53.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:53.104+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:53.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:53.104+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:53.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:53.104+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:53.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:53.105+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:53.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:53.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:53.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:53.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:53.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:53.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:53.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:53.437+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:53.840 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:53.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:53.840 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:53.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:53.840 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:53.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:53.840 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:53.533+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:54.130 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:53.929+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:53.930564+0000) 2024-04-27T03:27:54.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:53.991+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:53.992624+0000) 2024-04-27T03:27:54.401 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:54.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:54.401 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:54.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:54.401 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:54.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:54.401 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:54.155+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:54.401 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:54.179+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:54.180774+0000) 2024-04-27T03:27:54.401 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:54.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:54.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:54.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:54.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:54.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:54.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:54.395+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:54.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:54.553+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:54.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:54.553+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:54.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:54.553+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:54.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:54.553+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:55.187 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:54.994+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:54.994831+0000) 2024-04-27T03:27:55.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:54.931+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:54.932237+0000) 2024-04-27T03:27:55.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:55.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:55.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:55.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:55.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:55.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:55.188 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:55.155+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:55.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:27:55.198+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63014 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:27:55.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:55.195+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:55.196010+0000) 2024-04-27T03:27:55.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:55.197+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:55.197846+0000) 2024-04-27T03:27:55.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:55.181+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:55.182489+0000) 2024-04-27T03:27:55.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:55.196+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:55.197722+0000) 2024-04-27T03:27:55.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:55.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:55.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:55.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:55.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:55.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:55.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:55.375+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:55.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:55.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:55.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:55.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:55.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:55.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:55.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:55.581+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:56.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:56.181+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:56.182679+0000) 2024-04-27T03:27:56.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:55.994+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:55.995116+0000) 2024-04-27T03:27:56.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:55.931+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:55.932467+0000) 2024-04-27T03:27:56.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:56.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:56.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:56.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:56.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:56.145+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:56.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:56.147+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:56.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:56.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:56.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:56.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:56.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:56.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:56.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:56.421+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:56.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:56.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:56.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:56.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:56.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:56.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:56.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:56.591+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:57.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:56.998+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:56.998949+0000) 2024-04-27T03:27:57.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:56.934+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:56.935447+0000) 2024-04-27T03:27:57.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:57.130+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:57.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:57.130+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:57.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:57.130+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:57.193 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:57.131+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:57.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:57.186+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:57.187349+0000) 2024-04-27T03:27:57.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:57.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:57.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:57.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:57.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:57.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:57.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:57.414+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:57.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:57.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:57.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:57.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:57.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:57.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:57.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:57.575+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:58.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:58.002+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:58.003252+0000) 2024-04-27T03:27:58.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:57.938+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:57.939491+0000) 2024-04-27T03:27:58.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:58.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:58.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:58.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:58.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:58.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:58.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:58.139+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:58.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:58.188+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:58.189762+0000) 2024-04-27T03:27:58.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:58.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:58.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:58.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:58.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:58.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:58.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:58.397+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:58.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:58.573+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:58.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:58.573+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:58.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:58.573+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:58.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:58.573+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:59.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:59.192+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:59.192996+0000) 2024-04-27T03:27:59.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:59.007+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:59.008105+0000) 2024-04-27T03:27:59.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:58.943+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:58.943877+0000) 2024-04-27T03:27:59.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:59.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:59.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:59.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:59.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:59.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:27:59.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:59.119+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:59.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:59.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:59.605 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:59.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:59.605 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:59.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:27:59.605 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:27:59.362+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:27:59.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:59.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:59.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:59.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:59.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:59.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:27:59.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:27:59.599+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:00.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:27:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:27:59.945+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:27:59.946100+0000) 2024-04-27T03:28:00.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:00.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:00.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:00.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:00.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:00.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:00.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:00.155+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:00.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:00.009+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:00.010220+0000) 2024-04-27T03:28:00.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:00.194+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:00.194834+0000) 2024-04-27T03:28:00.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:28:00.205+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63053 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:28:00.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:00.202+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:00.203765+0000) 2024-04-27T03:28:00.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:00.204+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:00.205614+0000) 2024-04-27T03:28:00.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:00.203+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:00.204388+0000) 2024-04-27T03:28:00.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:00.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:00.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:00.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:00.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:00.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:00.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:00.391+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:00.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:00.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:00.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:00.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:00.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:00.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:00.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:00.596+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:01.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:01.009+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:01.010412+0000) 2024-04-27T03:28:01.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:01.194+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:01.195141+0000) 2024-04-27T03:28:01.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:00.945+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:00.946288+0000) 2024-04-27T03:28:01.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:01.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:01.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:01.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:01.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:01.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:01.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:01.165+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:01.616 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:01.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:01.616 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:01.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:01.616 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:01.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:01.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:01.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:01.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:01.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:01.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:01.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:01.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:01.384+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:01.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:01.611+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:02.212 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:01.948+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:01.949188+0000) 2024-04-27T03:28:02.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:02.196+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:02.197364+0000) 2024-04-27T03:28:02.212 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:02.011+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:02.012507+0000) 2024-04-27T03:28:02.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:02.206+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:02.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:02.206+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:02.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:02.206+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:02.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:02.207+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:02.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:02.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:02.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:02.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:02.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:02.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:02.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:02.337+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:02.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:02.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:02.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:02.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:02.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:02.629+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:02.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:02.630+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:03.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:02.950+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:02.952183+0000) 2024-04-27T03:28:03.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:03.015+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:03.016837+0000) 2024-04-27T03:28:03.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:03.197+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:03.199724+0000) 2024-04-27T03:28:03.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:03.296+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:03.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:03.296+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:03.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:03.296+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:03.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:03.297+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:03.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:03.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:03.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:03.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:03.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:03.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:03.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:03.213+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:03.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:03.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:03.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:03.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:03.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:03.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:03.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:03.625+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:04.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:04.017+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:04.019021+0000) 2024-04-27T03:28:04.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:03.955+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:03.956950+0000) 2024-04-27T03:28:04.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:04.200+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:04.202414+0000) 2024-04-27T03:28:04.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:04.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:04.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:04.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:04.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:04.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:04.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:04.256+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:04.657 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:04.230+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:04.657 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:04.230+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:04.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:04.230+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:04.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:04.231+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:04.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:04.650+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:04.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:04.650+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:04.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:04.650+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:04.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:04.651+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:05.216 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:05.020+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:05.022479+0000) 2024-04-27T03:28:05.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:05.202+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:05.204548+0000) 2024-04-27T03:28:05.216 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:04.956+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:04.958790+0000) 2024-04-27T03:28:05.216 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:05.201+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:05.216 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:05.201+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:05.216 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:05.201+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:05.216 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:05.202+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:05.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:05.209+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:05.211441+0000) 2024-04-27T03:28:05.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:28:05.212+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63093 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:28:05.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:05.208+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:05.210740+0000) 2024-04-27T03:28:05.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:05.208+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:05.210721+0000) 2024-04-27T03:28:05.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:05.252+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:05.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:05.252+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:05.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:05.252+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:05.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:05.252+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:05.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:05.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:05.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:05.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:05.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:05.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:05.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:05.679+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:06.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:05.957+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:05.958986+0000) 2024-04-27T03:28:06.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:06.020+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:06.022755+0000) 2024-04-27T03:28:06.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:06.202+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:06.204738+0000) 2024-04-27T03:28:06.651 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:06.241+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:06.651 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:06.241+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:06.651 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:06.241+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:06.651 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:06.242+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:06.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:06.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:06.652 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:06.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:06.652 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:06.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:06.652 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:06.248+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:06.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:06.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:06.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:06.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:06.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:06.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:06.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:06.645+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:07.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:07.206+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:07.207946+0000) 2024-04-27T03:28:07.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:06.961+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:06.963053+0000) 2024-04-27T03:28:07.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:07.023+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:07.025534+0000) 2024-04-27T03:28:07.624 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:07.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:07.625 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:07.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:07.625 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:07.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:07.625 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:07.236+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:07.625 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:07.240+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:07.625 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:07.240+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:07.625 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:07.240+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:07.625 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:07.241+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:07.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:07.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:07.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:07.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:07.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:07.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:07.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:07.618+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:08.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:07.963+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:07.965336+0000) 2024-04-27T03:28:08.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:08.025+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:08.027781+0000) 2024-04-27T03:28:08.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:08.210+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:08.212154+0000) 2024-04-27T03:28:08.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:08.224+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:08.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:08.224+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:08.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:08.224+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:08.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:08.225+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:08.645 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:08.265+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:08.645 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:08.265+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:08.645 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:08.265+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:08.646 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:08.265+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:08.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:08.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:08.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:08.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:08.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:08.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:08.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:08.638+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:09.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:08.968+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:08.969895+0000) 2024-04-27T03:28:09.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:09.028+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:09.029920+0000) 2024-04-27T03:28:09.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:09.199+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:09.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:09.199+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:09.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:09.199+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:09.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:09.199+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:09.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:09.213+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:09.214860+0000) 2024-04-27T03:28:09.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:09.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:09.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:09.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:09.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:09.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:09.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:09.282+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:09.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:09.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:09.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:09.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:09.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:09.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:09.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:09.678+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:10.232 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:28:10.220+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63132 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:28:10.232 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:09.971+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:09.972982+0000) 2024-04-27T03:28:10.232 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:10.217+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:10.219694+0000) 2024-04-27T03:28:10.232 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:10.152+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:10.233 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:10.152+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:10.233 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:10.152+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:10.233 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:10.152+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:10.233 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:10.217+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:10.219182+0000) 2024-04-27T03:28:10.233 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:10.217+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:10.219348+0000) 2024-04-27T03:28:10.233 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:10.030+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:10.032076+0000) 2024-04-27T03:28:10.233 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:10.218+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:10.220505+0000) 2024-04-27T03:28:10.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:10.239+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:10.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:10.239+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:10.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:10.239+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:10.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:10.240+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:10.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:10.718+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:10.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:10.718+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:10.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:10.718+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:10.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:10.719+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:11.240 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:11.030+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:11.032270+0000) 2024-04-27T03:28:11.240 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:10.971+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:10.973255+0000) 2024-04-27T03:28:11.240 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:11.205+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:11.241 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:11.205+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:11.241 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:11.205+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:11.241 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:11.207+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:11.241 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:11.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:11.241 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:11.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:11.241 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:11.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:11.241 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:11.144+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:11.241 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:11.220+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:11.222244+0000) 2024-04-27T03:28:11.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:11.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:11.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:11.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:11.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:11.686+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:11.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:11.687+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:12.232 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:12.035+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:12.036957+0000) 2024-04-27T03:28:12.232 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:12.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:12.232 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:12.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:12.232 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:12.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:12.232 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:12.149+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:12.232 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:11.973+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:11.975472+0000) 2024-04-27T03:28:12.233 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:12.222+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:12.233 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:12.222+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:12.233 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:12.222+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:12.233 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:12.223+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:12.665 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:12.225+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:12.226985+0000) 2024-04-27T03:28:12.666 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:12.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:12.666 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:12.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:12.666 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:12.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:12.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:12.660+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:13.236 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:13.037+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:13.039212+0000) 2024-04-27T03:28:13.237 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:13.165+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:13.237 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:13.165+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:13.237 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:13.165+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:13.237 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:13.165+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:13.237 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:12.977+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:12.979725+0000) 2024-04-27T03:28:13.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:13.224+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:13.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:13.224+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:13.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:13.224+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:13.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:13.224+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:13.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:13.229+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:13.231575+0000) 2024-04-27T03:28:13.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:13.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:13.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:13.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:13.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:13.691+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:13.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:13.693+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:14.212 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:13.980+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:13.981817+0000) 2024-04-27T03:28:14.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:14.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:14.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:14.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:14.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:14.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:14.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:14.122+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:14.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:14.039+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:14.041336+0000) 2024-04-27T03:28:14.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:14.231+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:14.233392+0000) 2024-04-27T03:28:14.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:14.205+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:14.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:14.205+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:14.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:14.205+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:14.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:14.206+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:14.988 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:14.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:14.988 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:14.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:14.989 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:14.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:14.989 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:14.721+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:15.238 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:28:15.229+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63172 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:28:15.239 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:14.982+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:14.983548+0000) 2024-04-27T03:28:15.239 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:15.226+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:15.226976+0000) 2024-04-27T03:28:15.239 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:15.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:15.239 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:15.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:15.239 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:15.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:15.239 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:15.146+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:15.240 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:15.226+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:15.227622+0000) 2024-04-27T03:28:15.240 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:15.232+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:15.233611+0000) 2024-04-27T03:28:15.240 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:15.042+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:15.043625+0000) 2024-04-27T03:28:15.240 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:15.226+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:15.227185+0000) 2024-04-27T03:28:15.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:15.240+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:15.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:15.240+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:15.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:15.240+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:15.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:15.241+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:16.047 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:15.982+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:15.983778+0000) 2024-04-27T03:28:16.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:15.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:16.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:15.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:16.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:15.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:16.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:15.760+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:16.315 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:16.043+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:16.043828+0000) 2024-04-27T03:28:16.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:16.259+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:16.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:16.259+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:16.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:16.259+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:16.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:16.261+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:16.316 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:16.130+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:16.316 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:16.130+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:16.316 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:16.130+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:16.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:16.130+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:16.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:16.237+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:16.237819+0000) 2024-04-27T03:28:17.027 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:16.985+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:16.985828+0000) 2024-04-27T03:28:17.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:16.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:17.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:16.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:17.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:16.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:17.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:16.748+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:17.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:17.047+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:17.048094+0000) 2024-04-27T03:28:17.318 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:17.290+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:17.318 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:17.290+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:17.318 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:17.290+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:17.318 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:17.291+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:17.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:17.125+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:17.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:17.125+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:17.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:17.125+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:17.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:17.126+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:17.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:17.239+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:17.240506+0000) 2024-04-27T03:28:18.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:17.987+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:17.988047+0000) 2024-04-27T03:28:18.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:17.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:18.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:17.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:18.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:17.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:18.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:17.743+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:18.318 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:18.264+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:18.318 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:18.264+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:18.318 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:18.264+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:18.318 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:18.265+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:18.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:18.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:18.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:18.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:18.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:18.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:18.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:18.077+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:18.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:18.241+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:18.242690+0000) 2024-04-27T03:28:18.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:18.049+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:18.050733+0000) 2024-04-27T03:28:19.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:18.989+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:18.990389+0000) 2024-04-27T03:28:19.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:18.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:19.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:18.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:19.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:18.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:19.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:18.782+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:19.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:19.052+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:19.052840+0000) 2024-04-27T03:28:19.318 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:19.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:19.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:19.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:19.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:19.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:19.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:19.053+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:19.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:19.243+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:19.244403+0000) 2024-04-27T03:28:19.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:19.276+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:19.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:19.276+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:19.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:19.276+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:19.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:19.277+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:20.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:19.991+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:19.992136+0000) 2024-04-27T03:28:20.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:19.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:20.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:19.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:20.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:19.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:20.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:19.750+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:20.318 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:28:20.236+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63211 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:28:20.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:20.055+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:20.056073+0000) 2024-04-27T03:28:20.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:20.234+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:20.235400+0000) 2024-04-27T03:28:20.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:20.234+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:20.235419+0000) 2024-04-27T03:28:20.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:20.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:20.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:20.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:20.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:20.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:20.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:20.281+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:20.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:20.088+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:20.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:20.088+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:20.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:20.088+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:20.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:20.089+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:20.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:20.234+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:20.234896+0000) 2024-04-27T03:28:20.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:20.243+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:20.244619+0000) 2024-04-27T03:28:20.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:20.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:20.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:20.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:20.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:20.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:20.998 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:20.729+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:21.252 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:20.991+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:20.992328+0000) 2024-04-27T03:28:21.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:21.070+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:21.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:21.070+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:21.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:21.070+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:21.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:21.070+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:21.253 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:21.055+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:21.056416+0000) 2024-04-27T03:28:21.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:21.246+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:21.247596+0000) 2024-04-27T03:28:21.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:21.288+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:21.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:21.288+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:21.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:21.288+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:21.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:21.289+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:21.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:21.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:21.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:21.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:21.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:21.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:21.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:21.714+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:22.255 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:22.057+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:22.058641+0000) 2024-04-27T03:28:22.255 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:21.993+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:21.994694+0000) 2024-04-27T03:28:22.255 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:22.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:22.255 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:22.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:22.256 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:22.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:22.256 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:22.091+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:22.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:22.249+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:22.250070+0000) 2024-04-27T03:28:22.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:22.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:22.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:22.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:22.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:22.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:22.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:22.335+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:22.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:22.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:22.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:22.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:22.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:22.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:22.961 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:22.689+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:23.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:22.996+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:22.996989+0000) 2024-04-27T03:28:23.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:23.060+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:23.060840+0000) 2024-04-27T03:28:23.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:23.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:23.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:23.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:23.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:23.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:23.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:23.127+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:23.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:23.251+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:23.252380+0000) 2024-04-27T03:28:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:23.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:23.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:23.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:23.386+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:24.004 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:23.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:24.004 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:23.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:24.004 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:23.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:24.004 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:23.736+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:24.260 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:23.998+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:23.999402+0000) 2024-04-27T03:28:24.260 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:24.062+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:24.063220+0000) 2024-04-27T03:28:24.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:24.105+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:24.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:24.105+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:24.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:24.105+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:24.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:24.106+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:24.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:24.254+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:24.255283+0000) 2024-04-27T03:28:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:24.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:24.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:24.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:24.341+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:25.049 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:25.002+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:25.003216+0000) 2024-04-27T03:28:25.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:24.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:25.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:24.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:25.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:24.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:25.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:24.782+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:25.323 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:28:25.244+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63251 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:28:25.324 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:25.241+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:25.242342+0000) 2024-04-27T03:28:25.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:25.064+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:25.065392+0000) 2024-04-27T03:28:25.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:25.241+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:25.241871+0000) 2024-04-27T03:28:25.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:25.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:25.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:25.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:25.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:25.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:25.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:25.118+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:25.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:25.241+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:25.241925+0000) 2024-04-27T03:28:25.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:25.254+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:25.255475+0000) 2024-04-27T03:28:25.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:25.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:25.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:25.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:25.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:25.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:25.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:25.389+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:26.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:26.002+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:26.003441+0000) 2024-04-27T03:28:26.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:25.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:26.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:25.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:26.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:25.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:26.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:25.772+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:26.326 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:26.064+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:26.065618+0000) 2024-04-27T03:28:26.326 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:26.074+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:26.326 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:26.074+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:26.326 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:26.074+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:26.326 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:26.075+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:26.327 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:26.257+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:26.258343+0000) 2024-04-27T03:28:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:26.384+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:26.384+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:26.384+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:26.386+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:27.049 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:27.005+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:27.006309+0000) 2024-04-27T03:28:27.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:26.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:27.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:26.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:27.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:26.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:27.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:26.762+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:27.328 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:27.067+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:27.068684+0000) 2024-04-27T03:28:27.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:27.093+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:27.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:27.093+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:27.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:27.093+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:27.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:27.094+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:27.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:27.261+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:27.262597+0000) 2024-04-27T03:28:27.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:27.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:27.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:27.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:27.350+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:28.049 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:28.007+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:28.008550+0000) 2024-04-27T03:28:28.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:27.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:28.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:27.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:28.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:27.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:28.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:27.794+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:28.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:28.070+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:28.070825+0000) 2024-04-27T03:28:28.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:28.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:28.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:28.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:28.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:28.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:28.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:28.100+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:28.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:28.264+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:28.264829+0000) 2024-04-27T03:28:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:28.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:28.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:28.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:28.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:28.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:28.396+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:29.049 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:29.010+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:29.011433+0000) 2024-04-27T03:28:29.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:28.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:29.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:28.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:29.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:28.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:29.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:28.790+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:29.328 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:29.072+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:29.073008+0000) 2024-04-27T03:28:29.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:29.117+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:29.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:29.117+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:29.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:29.117+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:29.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:29.118+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:29.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:29.266+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:29.267641+0000) 2024-04-27T03:28:29.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:29.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:29.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:29.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:29.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:29.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:29.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:29.411+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:30.125 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:30.012+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:30.013133+0000) 2024-04-27T03:28:30.125 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:29.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:30.126 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:29.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:30.126 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:29.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:30.126 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:29.840+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:30.126 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:30.074+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:30.075663+0000) 2024-04-27T03:28:30.376 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:28:30.251+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63290 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:28:30.376 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:30.249+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:30.250201+0000) 2024-04-27T03:28:30.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:30.249+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:30.250165+0000) 2024-04-27T03:28:30.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:30.119+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:30.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:30.119+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:30.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:30.119+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:30.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:30.120+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:30.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:30.249+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:30.250754+0000) 2024-04-27T03:28:30.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:30.267+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:30.267964+0000) 2024-04-27T03:28:30.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:30.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:30.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:30.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:30.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:30.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:30.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:30.407+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:31.081 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:31.012+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:31.013421+0000) 2024-04-27T03:28:31.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:30.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:31.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:30.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:31.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:30.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:31.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:30.800+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:31.082 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:31.075+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:31.075946+0000) 2024-04-27T03:28:31.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:31.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:31.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:31.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:31.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:31.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:31.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:31.106+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:31.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:31.269+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:31.270325+0000) 2024-04-27T03:28:31.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:31.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:31.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:31.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:31.383+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:32.049 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:32.015+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:32.016084+0000) 2024-04-27T03:28:32.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:31.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:32.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:31.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:32.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:31.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:32.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:31.778+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:32.333 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:32.079+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:32.080696+0000) 2024-04-27T03:28:32.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:32.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:32.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:32.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:32.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:32.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:32.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:32.084+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:32.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:32.271+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:32.272442+0000) 2024-04-27T03:28:32.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:32.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:32.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:32.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:32.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:32.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:32.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:32.419+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:32.991 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:32.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:32.992 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:32.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:32.992 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:32.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:32.992 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:32.734+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:33.280 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:33.020+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:33.020995+0000) 2024-04-27T03:28:33.281 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:33.084+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:33.085613+0000) 2024-04-27T03:28:33.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:33.057+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:33.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:33.057+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:33.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:33.057+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:33.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:33.058+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:33.708 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:33.274+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:33.275431+0000) 2024-04-27T03:28:33.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:33.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:33.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:33.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:33.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:33.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:33.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:33.370+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:33.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:33.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:33.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:33.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:33.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:33.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:33.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:33.703+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:34.282 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:34.025+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:34.026080+0000) 2024-04-27T03:28:34.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:34.089+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:34.090680+0000) 2024-04-27T03:28:34.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:34.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:34.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:34.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:34.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:34.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:34.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:34.036+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:34.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:34.276+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:34.277489+0000) 2024-04-27T03:28:34.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:34.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:34.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:34.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:34.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:34.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:34.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:34.335+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:34.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:34.690+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:34.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:34.690+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:34.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:34.690+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:34.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:34.691+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:35.259 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:35.029+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:35.029912+0000) 2024-04-27T03:28:35.259 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:35.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:35.259 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:35.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:35.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:35.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:35.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:35.080+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:35.260 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:35.093+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:35.094472+0000) 2024-04-27T03:28:35.653 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:28:35.261+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63330 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:28:35.653 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:35.257+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:35.258598+0000) 2024-04-27T03:28:35.653 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:35.276+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:35.277680+0000) 2024-04-27T03:28:35.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:35.258+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:35.258973+0000) 2024-04-27T03:28:35.654 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:35.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:35.654 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:35.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:35.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:35.256+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:35.257253+0000) 2024-04-27T03:28:35.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:35.317+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:35.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:35.317+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:35.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:35.317+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:35.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:35.318+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:35.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:35.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:35.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:35.648+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:36.285 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:36.029+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:36.030079+0000) 2024-04-27T03:28:36.285 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:36.093+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:36.094704+0000) 2024-04-27T03:28:36.286 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:36.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:36.286 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:36.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:36.286 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:36.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:36.286 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:36.041+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:36.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:36.279+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:36.280505+0000) 2024-04-27T03:28:36.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:36.303+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:36.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:36.303+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:36.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:36.303+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:36.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:36.304+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:36.675 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:36.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:36.676 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:36.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:36.676 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:36.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:36.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:36.670+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:37.287 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:37.031+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:37.032732+0000) 2024-04-27T03:28:37.287 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:37.097+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:37.097993+0000) 2024-04-27T03:28:37.287 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:37.000+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:37.287 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:37.000+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:37.288 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:37.000+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:37.288 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:37.001+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:37.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:37.282+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:37.283416+0000) 2024-04-27T03:28:37.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:37.281+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:37.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:37.281+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:37.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:37.281+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:37.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:37.282+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:37.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:37.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:37.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:37.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:37.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:37.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:37.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:37.679+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:38.260 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:38.035+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:38.035986+0000) 2024-04-27T03:28:38.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:38.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:38.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:38.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:38.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:38.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:38.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:38.022+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:38.261 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:38.099+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:38.100670+0000) 2024-04-27T03:28:38.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:38.285+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:38.286342+0000) 2024-04-27T03:28:38.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:38.254+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:38.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:38.254+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:38.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:38.254+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:38.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:38.255+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:38.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:38.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:38.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:38.714+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:39.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:39.037+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:39.038224+0000) 2024-04-27T03:28:39.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:39.104+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:39.105023+0000) 2024-04-27T03:28:39.293 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:39.007+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:39.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:39.007+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:39.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:39.007+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:39.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:39.007+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:39.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:39.287+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:39.288239+0000) 2024-04-27T03:28:39.707 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:39.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:39.707 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:39.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:39.707 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:39.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:39.707 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:39.305+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:39.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:39.701+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:39.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:39.701+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:39.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:39.701+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:39.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:39.702+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:40.268 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:40.041+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:40.042152+0000) 2024-04-27T03:28:40.268 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:40.108+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:40.108841+0000) 2024-04-27T03:28:40.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:40.019+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:40.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:40.019+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:40.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:40.019+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:40.269 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:40.019+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:40.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:40.266+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:40.267155+0000) 2024-04-27T03:28:40.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:28:40.268+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63369 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:28:40.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:40.266+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:40.267578+0000) 2024-04-27T03:28:40.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:40.287+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:40.288450+0000) 2024-04-27T03:28:40.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:40.266+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:40.266896+0000) 2024-04-27T03:28:40.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:40.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:40.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:40.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:40.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:40.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:40.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:40.289+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:40.992 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:40.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:40.993 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:40.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:40.993 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:40.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:40.993 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:40.723+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:41.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:41.041+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:41.042351+0000) 2024-04-27T03:28:41.296 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:41.108+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:41.109081+0000) 2024-04-27T03:28:41.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:41.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:41.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:41.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:41.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:41.013+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:41.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:41.014+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:41.602 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:41.289+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:41.290693+0000) 2024-04-27T03:28:41.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:41.320+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:41.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:41.320+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:41.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:41.320+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:41.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:41.321+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:41.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:41.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:41.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:41.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:41.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:41.688+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:41.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:41.689+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:42.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:42.043+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:42.044465+0000) 2024-04-27T03:28:42.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:42.112+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:42.113388+0000) 2024-04-27T03:28:42.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:42.051+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:42.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:42.051+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:42.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:42.051+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:42.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:42.052+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:42.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:42.293+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:42.294016+0000) 2024-04-27T03:28:42.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:42.365+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:42.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:42.365+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:42.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:42.365+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:42.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:42.367+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:42.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:42.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:42.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:42.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:42.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:42.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:42.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:42.681+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:43.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:43.046+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:43.047756+0000) 2024-04-27T03:28:43.303 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:43.114+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:43.115475+0000) 2024-04-27T03:28:43.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:43.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:43.304 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:43.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:43.304 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:43.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:43.304 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:43.100+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:43.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:43.297+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:43.298273+0000) 2024-04-27T03:28:43.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:43.331+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:43.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:43.331+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:43.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:43.331+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:43.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:43.332+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:43.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:43.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:43.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:43.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:43.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:43.680+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:43.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:43.681+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:44.305 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:44.049+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:44.050065+0000) 2024-04-27T03:28:44.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:44.117+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:44.118479+0000) 2024-04-27T03:28:44.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:44.092+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:44.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:44.092+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:44.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:44.092+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:44.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:44.093+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:44.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:44.299+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:44.300118+0000) 2024-04-27T03:28:44.643 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:44.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:44.643 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:44.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:44.643 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:44.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:44.643 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:44.313+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:44.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:44.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:44.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:44.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:44.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:44.637+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:44.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:44.638+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:45.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:45.050+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:45.051720+0000) 2024-04-27T03:28:45.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:45.120+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:45.120821+0000) 2024-04-27T03:28:45.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:45.070+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:45.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:45.070+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:45.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:45.070+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:45.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:45.071+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:45.682 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:28:45.276+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63409 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:28:45.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:45.275+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:45.276142+0000) 2024-04-27T03:28:45.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:45.299+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:45.300320+0000) 2024-04-27T03:28:45.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:45.272+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:45.273621+0000) 2024-04-27T03:28:45.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:45.273+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:45.273962+0000) 2024-04-27T03:28:45.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:45.325+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:45.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:45.325+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:45.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:45.325+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:45.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:45.326+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:45.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:45.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:45.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:45.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:45.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:45.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:45.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:45.677+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:46.291 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:46.120+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:46.121056+0000) 2024-04-27T03:28:46.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:46.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:46.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:46.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:46.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:46.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:46.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:46.031+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:46.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:46.051+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:46.051891+0000) 2024-04-27T03:28:46.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:46.285+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:46.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:46.285+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:46.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:46.285+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:46.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:46.287+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:46.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:46.301+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:46.302559+0000) 2024-04-27T03:28:46.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:46.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:46.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:46.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:46.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:46.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:46.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:46.672+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:47.295 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:47.122+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:47.123296+0000) 2024-04-27T03:28:47.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:47.053+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:47.053924+0000) 2024-04-27T03:28:47.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:47.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:47.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:47.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:47.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:47.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:47.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:47.028+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:47.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:47.306+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:47.306817+0000) 2024-04-27T03:28:47.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:47.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:47.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:47.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:47.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:47.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:47.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:47.290+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:47.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:47.704+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:47.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:47.704+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:47.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:47.704+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:47.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:47.705+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:48.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:48.124+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:48.125541+0000) 2024-04-27T03:28:48.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:48.055+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:48.056164+0000) 2024-04-27T03:28:48.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:48.272+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:48.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:48.272+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:48.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:48.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:48.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:48.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:48.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:48.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:48.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:48.001+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:48.272+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:48.273+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:48.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:48.308+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:48.309069+0000) 2024-04-27T03:28:49.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:48.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:49.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:48.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:49.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:48.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:49.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:48.980+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:49.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:48.749+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:49.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:48.749+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:49.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:48.749+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:49.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:48.751+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:49.312 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:49.059+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:49.060528+0000) 2024-04-27T03:28:49.313 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:49.126+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:49.127723+0000) 2024-04-27T03:28:49.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:49.309+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:49.310731+0000) 2024-04-27T03:28:49.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:49.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:49.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:49.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:49.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:49.307+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:49.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:49.307+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:49.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:49.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:49.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:49.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:49.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:49.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:49.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:49.707+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:50.283 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:50.062+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:50.062840+0000) 2024-04-27T03:28:50.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:50.128+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:50.129553+0000) 2024-04-27T03:28:50.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:49.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:50.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:49.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:50.284 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:49.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:50.284 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:49.986+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:50.720 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:28:50.284+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63448 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:28:50.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:50.281+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:50.282504+0000) 2024-04-27T03:28:50.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:50.310+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:50.310915+0000) 2024-04-27T03:28:50.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:50.281+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:50.281853+0000) 2024-04-27T03:28:50.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:50.714+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:50.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:50.714+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:50.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:50.714+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:50.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:50.280+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:50.281520+0000) 2024-04-27T03:28:50.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:50.333+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:50.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:50.333+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:50.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:50.333+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:50.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:50.334+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:50.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:50.715+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:51.318 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:51.062+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:51.063007+0000) 2024-04-27T03:28:51.318 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:50.997+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:51.318 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:50.997+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:51.318 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:50.997+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:51.318 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:50.997+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:51.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:51.129+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:51.129828+0000) 2024-04-27T03:28:51.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:51.312+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:51.313027+0000) 2024-04-27T03:28:51.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:51.328+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:51.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:51.328+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:51.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:51.328+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:51.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:51.329+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:52.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:51.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:52.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:51.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:52.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:51.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:52.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:51.753+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:52.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:52.065+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:52.065835+0000) 2024-04-27T03:28:52.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:52.133+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:52.134098+0000) 2024-04-27T03:28:52.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:52.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:52.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:52.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:52.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:52.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:52.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:52.039+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:52.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:52.314+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:52.315651+0000) 2024-04-27T03:28:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:52.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:52.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:52.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:52.335+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:53.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:52.779+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:53.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:52.779+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:53.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:52.780+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:53.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:52.781+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:53.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:53.069+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:53.070015+0000) 2024-04-27T03:28:53.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:53.136+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:53.136798+0000) 2024-04-27T03:28:53.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:53.066+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:53.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:53.067+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:53.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:53.067+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:53.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:53.067+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:53.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:53.317+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:53.318210+0000) 2024-04-27T03:28:53.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:53.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:53.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:53.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:53.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:53.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:53.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:53.353+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:54.077 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:53.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:54.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:53.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:54.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:53.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:54.109 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:53.815+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:54.358 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:54.138+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:54.139139+0000) 2024-04-27T03:28:54.358 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:54.071+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:54.072133+0000) 2024-04-27T03:28:54.358 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:54.307+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:54.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:54.307+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:54.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:54.307+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:54.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:54.308+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:54.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:54.079+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:54.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:54.079+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:54.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:54.079+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:54.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:54.079+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:54.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:54.319+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:54.320506+0000) 2024-04-27T03:28:55.079 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:55.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:55.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:55.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:55.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:55.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:55.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:55.037+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:55.080 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:54.797+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:55.080 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:54.797+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:55.080 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:54.797+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:55.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:54.797+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:55.358 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:28:55.291+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63488 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:28:55.358 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:55.140+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:55.141595+0000) 2024-04-27T03:28:55.358 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:55.288+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:55.289320+0000) 2024-04-27T03:28:55.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:55.073+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:55.074574+0000) 2024-04-27T03:28:55.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:55.265+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:55.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:55.265+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:55.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:55.265+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:55.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:55.266+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:55.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:55.289+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:55.290383+0000) 2024-04-27T03:28:55.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:55.290+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:55.291244+0000) 2024-04-27T03:28:55.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:55.319+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:55.320701+0000) 2024-04-27T03:28:56.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:55.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:56.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:55.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:56.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:55.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:56.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:55.749+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:56.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:55.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:56.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:55.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:56.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:55.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:56.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:55.989+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:56.328 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:56.141+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:56.141812+0000) 2024-04-27T03:28:56.328 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:56.074+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:56.074822+0000) 2024-04-27T03:28:56.328 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:56.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:56.328 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:56.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:56.328 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:56.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:56.328 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:56.257+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:56.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:56.322+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:56.323077+0000) 2024-04-27T03:28:57.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:56.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:57.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:56.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:57.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:56.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:57.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:56.753+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:57.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:56.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:57.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:56.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:57.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:56.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:57.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:56.971+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:57.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:57.143+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:57.143948+0000) 2024-04-27T03:28:57.332 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:57.078+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:57.079630+0000) 2024-04-27T03:28:57.332 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:57.243+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:57.332 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:57.243+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:57.332 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:57.243+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:57.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:57.244+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:57.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:57.326+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:57.327232+0000) 2024-04-27T03:28:58.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:57.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:58.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:57.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:58.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:57.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:58.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:57.758+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:58.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:57.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:58.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:57.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:58.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:57.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:58.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:57.953+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:58.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:58.145+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:58.146040+0000) 2024-04-27T03:28:58.325 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:58.083+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:58.084736+0000) 2024-04-27T03:28:58.325 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:58.244+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:58.325 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:58.244+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:58.325 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:58.244+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:58.326 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:58.245+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:58.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:58.329+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:58.330256+0000) 2024-04-27T03:28:58.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:58.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:58.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:58.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:58.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:58.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:28:58.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:58.712+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:58.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:58.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:58.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:58.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:58.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:58.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:28:58.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:58.957+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:28:59.263 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:59.088+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:59.089775+0000) 2024-04-27T03:28:59.263 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:59.147+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:59.148300+0000) 2024-04-27T03:28:59.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:59.332+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:28:59.333608+0000) 2024-04-27T03:28:59.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:59.257+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:59.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:59.257+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:59.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:59.257+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:28:59.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:28:59.259+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:00.035 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:59.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:00.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:59.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:00.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:59.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:00.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:28:59.754+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:00.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:59.976+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:00.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:59.976+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:00.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:59.976+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:00.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:28:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:28:59.977+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:00.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:00.090+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:00.091457+0000) 2024-04-27T03:29:00.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:00.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:00.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:00.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:00.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:00.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:00.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:00.213+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:00.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:00.296+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:00.297338+0000) 2024-04-27T03:29:00.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:00.151+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:00.152299+0000) 2024-04-27T03:29:00.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:00.297+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:00.297885+0000) 2024-04-27T03:29:00.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:29:00.299+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63527 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:29:00.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:00.297+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:00.298570+0000) 2024-04-27T03:29:00.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:00.333+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:00.333860+0000) 2024-04-27T03:29:01.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:00.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:01.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:00.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:01.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:00.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:01.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:00.940+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:01.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:00.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:01.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:00.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:01.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:00.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:01.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:00.782+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:01.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:01.151+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:01.152582+0000) 2024-04-27T03:29:01.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:01.090+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:01.091753+0000) 2024-04-27T03:29:01.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:01.247+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:01.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:01.247+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:01.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:01.247+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:01.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:01.248+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:01.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:01.335+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:01.336516+0000) 2024-04-27T03:29:02.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:01.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:02.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:01.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:02.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:01.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:02.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:01.759+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:02.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:01.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:02.054 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:01.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:02.054 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:01.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:02.054 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:01.893+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:02.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:02.093+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:02.093828+0000) 2024-04-27T03:29:02.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:02.266+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:02.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:02.266+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:02.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:02.266+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:02.345 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:02.267+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:02.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:02.154+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:02.154823+0000) 2024-04-27T03:29:02.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:02.338+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:02.339104+0000) 2024-04-27T03:29:03.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:02.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:03.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:02.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:03.054 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:02.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:03.054 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:02.901+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:03.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:02.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:03.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:02.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:03.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:02.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:03.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:02.796+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:03.307 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:03.156+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:03.157070+0000) 2024-04-27T03:29:03.308 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:03.095+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:03.096049+0000) 2024-04-27T03:29:03.308 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:03.301+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:03.308 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:03.301+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:03.308 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:03.301+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:03.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:03.340+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:03.341614+0000) 2024-04-27T03:29:03.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:03.303+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:04.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:03.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:04.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:03.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:04.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:03.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:04.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:03.869+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:04.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:03.826+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:04.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:03.826+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:04.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:03.826+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:04.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:03.827+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:04.358 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:04.099+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:04.099798+0000) 2024-04-27T03:29:04.358 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:04.342+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:04.343525+0000) 2024-04-27T03:29:04.358 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:04.160+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:04.161403+0000) 2024-04-27T03:29:04.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:04.352+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:04.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:04.352+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:04.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:04.352+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:04.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:04.353+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:05.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:04.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:05.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:04.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:05.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:04.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:05.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:04.868+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:05.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:04.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:05.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:04.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:05.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:04.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:05.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:04.822+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:05.368 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:29:05.306+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63567 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:29:05.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:05.100+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:05.102263+0000) 2024-04-27T03:29:05.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:05.302+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:05.304357+0000) 2024-04-27T03:29:05.369 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:05.161+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:05.163094+0000) 2024-04-27T03:29:05.369 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:05.303+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:05.305202+0000) 2024-04-27T03:29:05.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:05.303+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:05.305101+0000) 2024-04-27T03:29:05.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:05.342+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:05.343810+0000) 2024-04-27T03:29:05.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:05.381+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:05.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:05.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:05.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:05.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:05.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:05.382+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:05.995 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:05.780+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:05.995 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:05.780+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:05.995 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:05.780+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:05.995 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:05.781+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:05.996 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:05.892+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:05.996 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:05.892+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:05.996 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:05.892+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:05.996 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:05.893+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:06.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:06.100+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:06.102459+0000) 2024-04-27T03:29:06.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:06.161+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:06.163315+0000) 2024-04-27T03:29:06.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:06.344+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:06.346458+0000) 2024-04-27T03:29:06.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:06.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:06.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:06.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:06.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:06.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:06.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:06.346+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:07.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:06.932+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:07.054 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:06.932+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:07.054 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:06.932+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:07.054 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:06.932+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:07.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:06.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:07.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:06.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:07.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:06.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:07.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:06.741+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:07.354 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:07.105+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:07.107187+0000) 2024-04-27T03:29:07.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:07.163+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:07.165430+0000) 2024-04-27T03:29:07.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:07.347+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:07.349687+0000) 2024-04-27T03:29:07.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:07.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:07.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:07.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:07.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:07.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:07.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:07.380+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:07.995 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:07.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:07.996 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:07.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:07.996 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:07.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:07.996 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:07.913+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:07.996 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:07.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:07.996 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:07.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:07.996 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:07.733+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:07.996 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:07.733+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:08.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:08.107+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:08.109340+0000) 2024-04-27T03:29:08.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:08.331+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:08.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:08.331+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:08.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:08.331+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:08.338 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:08.165+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:08.167621+0000) 2024-04-27T03:29:08.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:08.332+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:08.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:08.350+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:08.352248+0000) 2024-04-27T03:29:08.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:08.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:08.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:08.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:08.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:08.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:08.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:08.918+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:08.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:08.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:08.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:08.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:08.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:08.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:08.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:08.714+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:09.251 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:09.112+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:09.114372+0000) 2024-04-27T03:29:09.252 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:09.170+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:09.171961+0000) 2024-04-27T03:29:09.688 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:09.352+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:09.353967+0000) 2024-04-27T03:29:09.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:09.329+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:09.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:09.329+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:09.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:09.329+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:09.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:09.330+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:09.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:09.681+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:09.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:09.681+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:09.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:09.681+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:09.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:09.682+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:09.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:09.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:09.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:09.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:09.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:09.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:09.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:09.872+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:10.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:10.172+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:10.173838+0000) 2024-04-27T03:29:10.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:10.116+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:10.118425+0000) 2024-04-27T03:29:10.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:10.311+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:10.313185+0000) 2024-04-27T03:29:10.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:29:10.313+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63607 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:29:10.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:10.312+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:10.313896+0000) 2024-04-27T03:29:10.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:10.352+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:10.354171+0000) 2024-04-27T03:29:10.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:10.310+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:10.311999+0000) 2024-04-27T03:29:10.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:10.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:10.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:10.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:10.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:10.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:10.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:10.360+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:10.996 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:10.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:10.996 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:10.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:10.996 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:10.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:10.996 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:10.722+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:10.997 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:10.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:10.997 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:10.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:10.997 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:10.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:10.997 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:10.873+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:11.337 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:11.172+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:11.174008+0000) 2024-04-27T03:29:11.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:11.116+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:11.118686+0000) 2024-04-27T03:29:11.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:11.330+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:11.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:11.330+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:11.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:11.330+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:11.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:11.355+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:11.356818+0000) 2024-04-27T03:29:11.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:11.332+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:11.996 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:11.727+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:11.996 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:11.727+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:11.996 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:11.727+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:11.996 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:11.728+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:11.997 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:11.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:11.997 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:11.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:11.997 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:11.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:11.997 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:11.834+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:12.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:12.119+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:12.121373+0000) 2024-04-27T03:29:12.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:12.174+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:12.176213+0000) 2024-04-27T03:29:12.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:12.357+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:12.359061+0000) 2024-04-27T03:29:12.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:12.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:12.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:12.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:12.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:12.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:12.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:12.370+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:12.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:12.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:12.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:12.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:12.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:12.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:12.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:12.849+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:12.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:12.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:12.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:12.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:12.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:12.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:12.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:12.697+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:13.356 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:13.121+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:13.123554+0000) 2024-04-27T03:29:13.356 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:13.176+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:13.178381+0000) 2024-04-27T03:29:13.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:13.359+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:13.361249+0000) 2024-04-27T03:29:13.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:13.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:13.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:13.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:13.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:13.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:13.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:13.350+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:13.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:13.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:13.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:13.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:13.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:13.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:13.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:13.708+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:13.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:13.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:13.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:13.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:13.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:13.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:13.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:13.834+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:14.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:14.180+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:14.182025+0000) 2024-04-27T03:29:14.330 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:14.126+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:14.127832+0000) 2024-04-27T03:29:14.331 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:14.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:14.331 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:14.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:14.331 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:14.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:14.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:14.324+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:14.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:14.361+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:14.363130+0000) 2024-04-27T03:29:15.054 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:14.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:15.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:14.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:15.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:14.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:15.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:14.799+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:15.055 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:14.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:15.055 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:14.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:15.055 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:14.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:15.055 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:14.754+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:15.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:15.127+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:15.129690+0000) 2024-04-27T03:29:15.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:15.182+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:15.183896+0000) 2024-04-27T03:29:15.725 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:29:15.321+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63647 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:29:15.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:15.317+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:15.319699+0000) 2024-04-27T03:29:15.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:15.319+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:15.321745+0000) 2024-04-27T03:29:15.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:15.361+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:15.363346+0000) 2024-04-27T03:29:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:15.321+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:15.323497+0000) 2024-04-27T03:29:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:15.361+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:15.361+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:15.361+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:15.362+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:16.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:15.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:16.055 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:15.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:16.055 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:15.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:16.055 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:15.781+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:16.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:15.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:16.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:15.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:16.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:15.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:16.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:15.818+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:16.371 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:16.128+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:16.129910+0000) 2024-04-27T03:29:16.371 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:16.183+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:16.184090+0000) 2024-04-27T03:29:16.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:16.365+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:16.366200+0000) 2024-04-27T03:29:16.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:16.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:16.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:16.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:16.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:16.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:16.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:16.375+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:16.996 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:16.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:16.997 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:16.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:16.997 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:16.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:16.997 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:16.826+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:16.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:16.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:16.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:16.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:16.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:16.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:16.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:16.739+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:17.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:17.131+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:17.132114+0000) 2024-04-27T03:29:17.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:17.185+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:17.186313+0000) 2024-04-27T03:29:17.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:17.368+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:17.368902+0000) 2024-04-27T03:29:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:17.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:17.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:17.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:17.376+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:17.996 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:17.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:17.997 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:17.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:17.997 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:17.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:17.997 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:17.785+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:17.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:17.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:17.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:17.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:17.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:17.735+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:17.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:17.735+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:18.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:18.188+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:18.189251+0000) 2024-04-27T03:29:18.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:18.133+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:18.134758+0000) 2024-04-27T03:29:18.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:18.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:18.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:18.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:18.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:18.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:18.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:18.341+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:18.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:18.372+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:18.373269+0000) 2024-04-27T03:29:18.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:18.718+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:18.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:18.718+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:18.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:18.718+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:18.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:18.719+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:18.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:18.766+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:18.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:18.766+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:18.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:18.766+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:18.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:18.767+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:19.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:19.137+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:19.137989+0000) 2024-04-27T03:29:19.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:19.191+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:19.192236+0000) 2024-04-27T03:29:19.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:19.375+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:19.376719+0000) 2024-04-27T03:29:19.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:19.372+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:19.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:19.372+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:19.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:19.372+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:19.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:19.373+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:20.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:19.778+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:20.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:19.778+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:20.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:19.778+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:20.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:19.779+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:20.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:19.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:20.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:19.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:20.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:19.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:20.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:19.763+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:20.329 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:20.138+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:20.139705+0000) 2024-04-27T03:29:20.329 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:20.193+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:20.193903+0000) 2024-04-27T03:29:20.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:29:20.329+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63686 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:29:20.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:20.327+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:20.328167+0000) 2024-04-27T03:29:20.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:20.328+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:20.328880+0000) 2024-04-27T03:29:20.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:20.376+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:20.376904+0000) 2024-04-27T03:29:20.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:20.327+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:20.328607+0000) 2024-04-27T03:29:20.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:20.370+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:20.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:20.370+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:20.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:20.370+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:20.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:20.371+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:21.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:20.765+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:21.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:20.765+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:21.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:20.765+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:21.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:20.766+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:21.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:20.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:21.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:20.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:21.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:20.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:21.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:20.795+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:21.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:21.139+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:21.139913+0000) 2024-04-27T03:29:21.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:21.193+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:21.194076+0000) 2024-04-27T03:29:21.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:21.378+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:21.379301+0000) 2024-04-27T03:29:21.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:21.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:21.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:21.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:21.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:21.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:21.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:21.339+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:22.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:21.747+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:22.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:21.747+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:22.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:21.747+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:22.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:21.747+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:22.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:21.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:22.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:21.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:22.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:21.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:22.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:21.823+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:22.372 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:22.141+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:22.141969+0000) 2024-04-27T03:29:22.372 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:22.197+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:22.197858+0000) 2024-04-27T03:29:22.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:22.380+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:22.381629+0000) 2024-04-27T03:29:22.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:22.366+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:22.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:22.366+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:22.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:22.366+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:22.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:22.367+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:23.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:22.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:23.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:22.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:23.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:22.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:23.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:22.745+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:23.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:22.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:23.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:22.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:23.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:22.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:23.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:22.802+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:23.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:23.145+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:23.146746+0000) 2024-04-27T03:29:23.381 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:23.201+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:23.202175+0000) 2024-04-27T03:29:23.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:23.385+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:23.386440+0000) 2024-04-27T03:29:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:23.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:23.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:23.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:23.376+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:24.055 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:23.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:24.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:23.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:24.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:23.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:24.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:23.839+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:24.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:23.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:24.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:23.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:24.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:23.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:24.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:23.784+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:24.389 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:24.148+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:24.148847+0000) 2024-04-27T03:29:24.389 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:24.204+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:24.205196+0000) 2024-04-27T03:29:24.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:24.389+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:24.390303+0000) 2024-04-27T03:29:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:24.423+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:24.423+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:24.423+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:24.425+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:25.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:24.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:25.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:24.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:25.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:24.792+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:25.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:24.793+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:25.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:24.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:25.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:24.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:25.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:24.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:25.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:24.825+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:25.336 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:25.149+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:25.150592+0000) 2024-04-27T03:29:25.337 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:25.207+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:25.208727+0000) 2024-04-27T03:29:25.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:25.335+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:25.335817+0000) 2024-04-27T03:29:25.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:29:25.337+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63726 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:29:25.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:25.334+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:25.335793+0000) 2024-04-27T03:29:25.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:25.389+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:25.390589+0000) 2024-04-27T03:29:25.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:25.333+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:25.334774+0000) 2024-04-27T03:29:25.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:25.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:25.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:25.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:25.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:25.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:25.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:25.400+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:26.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:25.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:26.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:25.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:26.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:25.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:26.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:25.800+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:26.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:25.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:26.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:25.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:26.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:25.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:26.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:25.869+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:26.364 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:26.150+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:26.150796+0000) 2024-04-27T03:29:26.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:26.208+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:26.208938+0000) 2024-04-27T03:29:26.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:26.392+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:26.392851+0000) 2024-04-27T03:29:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:26.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:26.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:26.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:26.360+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:27.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:26.808+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:27.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:26.808+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:27.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:26.808+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:27.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:26.809+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:27.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:26.828+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:27.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:26.828+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:27.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:26.828+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:27.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:26.828+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:27.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:27.154+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:27.154978+0000) 2024-04-27T03:29:27.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:27.210+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:27.211581+0000) 2024-04-27T03:29:27.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:27.395+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:27.396175+0000) 2024-04-27T03:29:27.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:27.398+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:27.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:27.398+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:27.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:27.398+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:27.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:27.399+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:28.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:27.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:28.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:27.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:28.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:27.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:28.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:27.793+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:28.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:27.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:28.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:27.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:28.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:27.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:28.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:27.800+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:28.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:28.156+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:28.157103+0000) 2024-04-27T03:29:28.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:28.213+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:28.214728+0000) 2024-04-27T03:29:28.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:28.397+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:28.398531+0000) 2024-04-27T03:29:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:28.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:28.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:28.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:28.438+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:29.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:28.783+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:29.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:28.783+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:29.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:28.783+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:29.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:28.783+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:29.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:28.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:29.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:28.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:29.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:28.771+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:29.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:28.772+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:29.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:29.160+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:29.161329+0000) 2024-04-27T03:29:29.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:29.217+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:29.218049+0000) 2024-04-27T03:29:29.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:29.401+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:29.401839+0000) 2024-04-27T03:29:29.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:29.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:29.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:29.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:29.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:29.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:29.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:29.427+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:30.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:29.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:30.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:29.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:30.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:29.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:30.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:29.774+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:30.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:29.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:30.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:29.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:30.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:29.762+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:30.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:29.763+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:30.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:30.162+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:30.163764+0000) 2024-04-27T03:29:30.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:30.219+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:30.220334+0000) 2024-04-27T03:29:30.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:29:30.344+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63765 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:29:30.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:30.341+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:30.342310+0000) 2024-04-27T03:29:30.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:30.342+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:30.343681+0000) 2024-04-27T03:29:30.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:30.401+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:30.402009+0000) 2024-04-27T03:29:30.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:30.341+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:30.342636+0000) 2024-04-27T03:29:30.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:30.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:30.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:30.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:30.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:30.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:30.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:30.455+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:31.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:30.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:31.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:30.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:31.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:30.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:31.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:30.758+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:31.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:30.813+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:31.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:30.813+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:31.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:30.813+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:31.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:30.814+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:31.395 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:31.163+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:31.163966+0000) 2024-04-27T03:29:31.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:31.219+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:31.220616+0000) 2024-04-27T03:29:31.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:31.403+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:31.404087+0000) 2024-04-27T03:29:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:31.488+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:31.488+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:31.488+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:31.489+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:32.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:31.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:32.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:31.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:32.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:31.847+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:32.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:31.847+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:32.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:31.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:32.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:31.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:32.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:31.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:32.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:31.747+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:32.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:32.167+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:32.168171+0000) 2024-04-27T03:29:32.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:32.221+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:32.222268+0000) 2024-04-27T03:29:32.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:32.407+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:32.408310+0000) 2024-04-27T03:29:32.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:32.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:32.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:32.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:32.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:32.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:32.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:32.459+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:33.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:32.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:33.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:32.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:33.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:32.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:33.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:32.752+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:33.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:32.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:33.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:32.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:33.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:32.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:33.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:32.876+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:33.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:33.170+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:33.171043+0000) 2024-04-27T03:29:33.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:33.223+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:33.224407+0000) 2024-04-27T03:29:33.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:33.411+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:33.412572+0000) 2024-04-27T03:29:33.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:33.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:33.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:33.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:33.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:33.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:33.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:33.416+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:33.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:33.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:33.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:33.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:33.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:33.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:33.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:33.916+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:33.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:33.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:33.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:33.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:33.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:33.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:33.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:33.731+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:34.397 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:34.174+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:34.175297+0000) 2024-04-27T03:29:34.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:34.226+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:34.227416+0000) 2024-04-27T03:29:34.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:34.413+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:34.414502+0000) 2024-04-27T03:29:34.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:34.438+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:34.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:34.438+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:34.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:34.438+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:34.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:34.439+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:34.998 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:34.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:34.999 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:34.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:34.999 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:34.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:34.999 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:34.956+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:34.999 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:34.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:34.999 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:34.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:34.999 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:34.740+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:34.999 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:34.741+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:35.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:35.178+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:35.179165+0000) 2024-04-27T03:29:35.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:35.230+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:35.230949+0000) 2024-04-27T03:29:35.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:29:35.354+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63805 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:29:35.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:35.350+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:35.351261+0000) 2024-04-27T03:29:35.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:35.413+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:35.414704+0000) 2024-04-27T03:29:35.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:35.349+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:35.350496+0000) 2024-04-27T03:29:35.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:35.349+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:35.349859+0000) 2024-04-27T03:29:35.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:35.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:35.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:35.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:35.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:35.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:35.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:35.447+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:36.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:35.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:36.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:35.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:36.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:35.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:36.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:35.972+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:36.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:35.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:36.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:35.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:36.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:35.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:36.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:35.755+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:36.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:36.178+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:36.179428+0000) 2024-04-27T03:29:36.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:36.230+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:36.231222+0000) 2024-04-27T03:29:36.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:36.416+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:36.416810+0000) 2024-04-27T03:29:36.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:36.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:36.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:36.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:36.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:36.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:36.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:36.414+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:37.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:36.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:37.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:36.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:37.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:36.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:37.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:36.955+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:37.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:36.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:37.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:36.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:37.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:36.777+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:37.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:36.778+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:37.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:37.183+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:37.183971+0000) 2024-04-27T03:29:37.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:37.232+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:37.233412+0000) 2024-04-27T03:29:37.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:37.418+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:37.419284+0000) 2024-04-27T03:29:37.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:37.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:37.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:37.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:37.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:37.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:37.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:37.402+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:38.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:37.824+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:38.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:37.824+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:38.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:37.824+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:38.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:37.825+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:38.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:37.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:38.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:37.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:38.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:37.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:38.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:37.948+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:38.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:38.185+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:38.186172+0000) 2024-04-27T03:29:38.402 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:38.234+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:38.235735+0000) 2024-04-27T03:29:38.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:38.421+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:38.422334+0000) 2024-04-27T03:29:38.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:38.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:38.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:38.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:38.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:38.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:38.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:38.445+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:38.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:38.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:38.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:38.823+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:38.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:38.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:38.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:38.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:38.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:38.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:38.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:38.927+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:39.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:39.187+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:39.188246+0000) 2024-04-27T03:29:39.402 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:39.237+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:39.238432+0000) 2024-04-27T03:29:39.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:39.423+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:39.424125+0000) 2024-04-27T03:29:39.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:39.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:39.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:39.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:39.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:39.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:39.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:39.461+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:40.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:39.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:40.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:39.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:40.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:39.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:40.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:39.898+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:40.154 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:39.871+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:40.154 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:39.871+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:40.154 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:39.871+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:40.154 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:39.872+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:40.402 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:29:40.362+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63844 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:29:40.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:40.189+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:40.190120+0000) 2024-04-27T03:29:40.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:40.359+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:40.360306+0000) 2024-04-27T03:29:40.403 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:40.358+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:40.359258+0000) 2024-04-27T03:29:40.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:40.239+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:40.240181+0000) 2024-04-27T03:29:40.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:40.360+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:40.360944+0000) 2024-04-27T03:29:40.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:40.423+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:40.424393+0000) 2024-04-27T03:29:40.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:40.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:40.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:40.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:40.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:40.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:40.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:40.482+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:41.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:40.913+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:41.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:40.913+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:41.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:40.913+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:41.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:40.913+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:41.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:40.886+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:41.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:40.886+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:41.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:40.886+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:41.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:40.887+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:41.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:41.189+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:41.190311+0000) 2024-04-27T03:29:41.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:41.426+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:41.427444+0000) 2024-04-27T03:29:41.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:41.239+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:41.240448+0000) 2024-04-27T03:29:41.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:41.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:41.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:41.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:41.891 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:41.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:41.891 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:41.472+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:42.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:41.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:42.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:41.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:42.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:41.884+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:42.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:41.885+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:42.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:41.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:42.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:41.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:42.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:41.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:42.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:41.899+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:42.406 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:42.192+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:42.193075+0000) 2024-04-27T03:29:42.407 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:42.242+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:42.243612+0000) 2024-04-27T03:29:42.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:42.428+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:42.429697+0000) 2024-04-27T03:29:42.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:42.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:42.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:42.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:42.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:42.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:42.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:42.492+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:43.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:42.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:43.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:42.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:43.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:42.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:43.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:42.864+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:43.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:42.944+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:43.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:42.944+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:43.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:42.944+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:43.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:42.946+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:43.407 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:43.196+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:43.197316+0000) 2024-04-27T03:29:43.407 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:43.245+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:43.246359+0000) 2024-04-27T03:29:43.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:43.431+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:43.432401+0000) 2024-04-27T03:29:43.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:43.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:43.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:43.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:43.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:43.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:43.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:43.497+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:44.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:43.903+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:44.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:43.903+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:44.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:43.903+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:44.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:43.903+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:44.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:43.904+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:44.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:43.904+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:44.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:43.904+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:44.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:43.906+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:44.208 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:44.198+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:44.199526+0000) 2024-04-27T03:29:44.406 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:44.249+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:44.250722+0000) 2024-04-27T03:29:44.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:44.433+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:44.434722+0000) 2024-04-27T03:29:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:44.473+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:44.473+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:44.473+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:44.475+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:45.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:44.913+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:45.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:44.913+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:45.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:44.913+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:45.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:44.914+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:45.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:44.899+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:45.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:44.899+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:45.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:44.899+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:45.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:44.899+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:45.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:29:45.369+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63884 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:29:45.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:45.366+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:45.366988+0000) 2024-04-27T03:29:45.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:45.434+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:45.434948+0000) 2024-04-27T03:29:45.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:45.200+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:45.201295+0000) 2024-04-27T03:29:45.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:45.366+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:45.367715+0000) 2024-04-27T03:29:45.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:45.251+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:45.252512+0000) 2024-04-27T03:29:45.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:45.366+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:45.367520+0000) 2024-04-27T03:29:45.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:45.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:45.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:45.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:45.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:45.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:45.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:45.471+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:46.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:45.892+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:46.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:45.892+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:46.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:45.892+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:46.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:45.893+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:46.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:45.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:46.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:45.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:46.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:45.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:46.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:45.874+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:46.409 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:46.251+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:46.252708+0000) 2024-04-27T03:29:46.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:46.200+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:46.201498+0000) 2024-04-27T03:29:46.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:46.436+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:46.437617+0000) 2024-04-27T03:29:46.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:46.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:46.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:46.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:46.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:46.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:46.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:46.455+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:47.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:46.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:47.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:46.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:47.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:46.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:47.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:46.863+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:47.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:46.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:47.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:46.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:47.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:46.829+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:47.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:46.830+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:47.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:47.205+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:47.206153+0000) 2024-04-27T03:29:47.411 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:47.255+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:47.255909+0000) 2024-04-27T03:29:47.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:47.439+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:47.439828+0000) 2024-04-27T03:29:47.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:47.411+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:47.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:47.411+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:47.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:47.411+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:47.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:47.413+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:48.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:47.864+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:48.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:47.864+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:48.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:47.864+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:48.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:47.865+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:48.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:47.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:48.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:47.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:48.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:47.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:48.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:47.850+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:48.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:48.208+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:48.208811+0000) 2024-04-27T03:29:48.411 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:48.257+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:48.258600+0000) 2024-04-27T03:29:48.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:48.441+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:48.441954+0000) 2024-04-27T03:29:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:48.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:48.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:48.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:48.436+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:49.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:48.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:49.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:48.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:49.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:48.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:49.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:48.883+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:49.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:48.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:49.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:48.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:49.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:48.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:49.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:48.897+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:49.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:49.211+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:49.212024+0000) 2024-04-27T03:29:49.411 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:49.260+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:49.260815+0000) 2024-04-27T03:29:49.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:49.442+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:49.443753+0000) 2024-04-27T03:29:49.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:49.409+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:49.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:49.409+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:49.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:49.409+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:49.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:49.410+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:50.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:49.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:50.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:49.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:50.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:49.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:50.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:49.898+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:50.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:49.919+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:50.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:49.919+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:50.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:49.919+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:50.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:49.919+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:50.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:29:50.378+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63923 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:29:50.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:50.261+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:50.262575+0000) 2024-04-27T03:29:50.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:50.375+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:50.376325+0000) 2024-04-27T03:29:50.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:50.374+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:50.375450+0000) 2024-04-27T03:29:50.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:50.443+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:50.443960+0000) 2024-04-27T03:29:50.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:50.213+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:50.213823+0000) 2024-04-27T03:29:50.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:50.375+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:50.375832+0000) 2024-04-27T03:29:50.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:50.408+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:50.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:50.408+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:50.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:50.408+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:50.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:50.409+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:51.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:50.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:51.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:50.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:51.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:50.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:51.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:50.894+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:51.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:50.945+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:51.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:50.945+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:51.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:50.945+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:51.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:50.947+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:51.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:51.262+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:51.262858+0000) 2024-04-27T03:29:51.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:51.445+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:51.446018+0000) 2024-04-27T03:29:51.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:51.213+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:51.214015+0000) 2024-04-27T03:29:51.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:51.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:51.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:51.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:51.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:51.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:51.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:51.426+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:52.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:51.970+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:52.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:51.970+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:52.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:51.970+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:52.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:51.971+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:52.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:51.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:52.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:51.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:52.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:51.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:52.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:51.897+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:52.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:52.266+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:52.267114+0000) 2024-04-27T03:29:52.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:52.447+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:52.448218+0000) 2024-04-27T03:29:52.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:52.217+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:52.218235+0000) 2024-04-27T03:29:52.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:52.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:52.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:52.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:52.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:52.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:52.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:52.395+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:53.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:52.972+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:53.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:52.972+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:53.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:52.972+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:53.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:52.973+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:53.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:52.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:53.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:52.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:53.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:52.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:53.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:52.947+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:53.415 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:53.268+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:53.269258+0000) 2024-04-27T03:29:53.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:53.219+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:53.220391+0000) 2024-04-27T03:29:53.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:53.348+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:53.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:53.348+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:53.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:53.348+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:53.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:53.349+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:53.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:53.449+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:53.450511+0000) 2024-04-27T03:29:54.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:53.927+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:54.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:53.927+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:54.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:53.927+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:54.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:53.928+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:54.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:53.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:54.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:53.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:54.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:53.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:54.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:53.947+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:54.415 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:54.272+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:54.273150+0000) 2024-04-27T03:29:54.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:54.221+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:54.222567+0000) 2024-04-27T03:29:54.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:54.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:54.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:54.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:54.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:54.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:54.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:54.349+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:54.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:54.452+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:54.452900+0000) 2024-04-27T03:29:55.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:54.938+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:55.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:54.938+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:55.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:54.938+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:55.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:54.938+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:55.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:54.942+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:55.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:54.942+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:55.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:54.942+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:55.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:54.943+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:55.475 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:29:55.385+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 63964 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:29:55.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:55.382+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:55.383539+0000) 2024-04-27T03:29:55.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:55.452+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:55.453183+0000) 2024-04-27T03:29:55.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:55.275+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:55.275899+0000) 2024-04-27T03:29:55.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:55.383+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:55.384267+0000) 2024-04-27T03:29:55.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:55.224+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:55.225367+0000) 2024-04-27T03:29:55.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:55.302+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:55.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:55.302+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:55.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:55.302+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:55.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:55.303+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:55.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:55.382+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:55.383729+0000) 2024-04-27T03:29:56.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:55.948+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:56.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:55.948+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:56.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:55.948+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:56.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:55.949+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:56.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:55.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:56.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:55.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:56.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:55.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:56.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:55.922+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:56.462 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:56.275+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:56.276081+0000) 2024-04-27T03:29:56.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:56.224+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:56.225565+0000) 2024-04-27T03:29:56.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:56.284+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:56.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:56.284+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:56.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:56.284+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:56.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:56.285+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:56.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:56.456+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:56.457446+0000) 2024-04-27T03:29:57.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:56.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:57.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:56.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:57.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:56.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:57.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:56.896+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:57.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:56.957+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:57.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:56.957+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:57.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:56.957+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:57.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:56.958+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:57.466 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:57.277+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:57.278220+0000) 2024-04-27T03:29:57.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:57.228+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:57.229567+0000) 2024-04-27T03:29:57.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:57.260+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:57.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:57.260+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:57.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:57.260+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:57.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:57.261+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:57.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:57.459+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:57.460363+0000) 2024-04-27T03:29:58.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:57.888+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:58.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:57.888+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:58.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:57.888+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:58.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:57.889+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:58.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:57.942+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:58.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:57.942+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:58.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:57.942+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:58.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:57.943+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:58.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:58.279+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:58.280367+0000) 2024-04-27T03:29:58.468 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:58.230+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:58.231660+0000) 2024-04-27T03:29:58.468 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:58.278+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:58.468 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:58.278+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:58.468 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:58.278+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:58.468 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:58.279+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:58.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:58.462+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:58.463127+0000) 2024-04-27T03:29:59.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:58.981+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:59.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:58.981+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:59.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:58.981+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:29:59.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:58.981+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:59.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:58.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:59.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:58.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:59.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:58.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:29:59.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:58.924+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:59.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:29:59.282+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:59.283286+0000) 2024-04-27T03:29:59.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:59.233+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:59.233956+0000) 2024-04-27T03:29:59.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:59.241+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:59.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:59.241+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:59.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:59.241+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:29:59.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:29:59.242+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:29:59.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:59.464+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:29:59.465078+0000) 2024-04-27T03:30:00.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:00.017+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:00.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:00.017+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:00.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:00.017+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:00.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:00.019+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:00.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:59.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:00.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:59.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:00.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:59.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:00.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:29:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:29:59.883+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:00.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:00.284+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:00.285043+0000) 2024-04-27T03:30:00.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:00.234+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:00.235745+0000) 2024-04-27T03:30:00.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:00.269+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:00.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:00.269+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:00.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:00.269+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:00.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:00.270+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:00.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:00.393+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:00.394127+0000) 2024-04-27T03:30:00.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:30:00.394+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64005 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:30:00.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:00.390+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:00.391076+0000) 2024-04-27T03:30:00.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:00.391+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:00.391878+0000) 2024-04-27T03:30:00.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:00.464+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:00.465282+0000) 2024-04-27T03:30:01.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:01.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:01.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:01.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:01.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:01.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:01.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:01.001+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:01.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:00.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:01.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:00.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:01.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:00.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:01.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:00.914+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:01.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:01.284+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:01.285241+0000) 2024-04-27T03:30:01.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:01.235+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:01.235947+0000) 2024-04-27T03:30:01.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:01.238+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:01.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:01.238+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:01.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:01.238+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:01.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:01.239+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:01.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:01.466+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:01.467380+0000) 2024-04-27T03:30:02.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:02.004+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:02.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:02.004+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:02.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:02.004+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:02.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:02.005+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:02.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:01.872+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:02.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:01.872+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:02.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:01.872+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:02.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:01.872+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:02.426 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:02.287+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:02.288214+0000) 2024-04-27T03:30:02.426 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:02.237+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:02.238019+0000) 2024-04-27T03:30:02.426 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:02.241+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:02.426 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:02.241+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:02.426 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:02.241+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:02.426 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:02.241+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:02.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:02.469+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:02.470517+0000) 2024-04-27T03:30:03.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:02.876+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:03.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:02.876+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:03.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:02.876+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:03.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:02.876+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:03.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:03.038+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:03.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:03.038+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:03.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:03.038+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:03.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:03.039+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:03.426 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:03.289+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:03.290079+0000) 2024-04-27T03:30:03.426 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:03.192+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:03.426 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:03.192+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:03.426 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:03.192+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:03.426 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:03.193+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:03.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:03.241+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:03.242243+0000) 2024-04-27T03:30:03.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:03.472+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:03.473587+0000) 2024-04-27T03:30:04.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:03.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:04.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:03.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:04.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:03.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:04.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:03.896+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:04.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:04.058+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:04.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:04.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:04.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:04.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:04.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:04.060+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:04.426 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:04.291+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:04.292231+0000) 2024-04-27T03:30:04.426 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:04.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:04.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:04.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:04.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:04.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:04.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:04.173+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:04.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:04.243+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:04.244338+0000) 2024-04-27T03:30:04.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:04.477+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:04.477945+0000) 2024-04-27T03:30:05.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:05.020+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:05.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:05.020+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:05.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:05.020+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:05.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:05.021+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:05.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:04.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:05.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:04.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:05.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:04.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:05.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:04.881+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:05.401 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:05.293+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:05.294784+0000) 2024-04-27T03:30:05.401 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:05.152+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:05.401 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:05.152+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:05.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:05.152+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:05.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:05.153+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:05.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:05.245+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:05.246087+0000) 2024-04-27T03:30:05.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:05.398+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:05.399343+0000) 2024-04-27T03:30:05.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:05.398+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:05.399154+0000) 2024-04-27T03:30:05.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:30:05.401+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64045 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:30:05.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:05.399+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:05.400593+0000) 2024-04-27T03:30:05.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:05.477+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:05.478236+0000) 2024-04-27T03:30:06.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:06.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:06.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:06.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:06.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:06.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:06.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:06.025+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:06.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:05.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:06.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:05.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:06.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:05.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:06.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:05.876+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:06.429 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:06.293+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:06.294967+0000) 2024-04-27T03:30:06.430 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:06.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:06.430 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:06.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:06.430 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:06.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:06.430 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:06.155+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:06.430 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:06.244+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:06.246295+0000) 2024-04-27T03:30:06.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:06.480+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:06.482503+0000) 2024-04-27T03:30:07.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:06.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:07.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:06.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:07.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:06.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:07.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:06.894+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:07.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:07.050+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:07.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:07.050+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:07.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:07.050+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:07.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:07.051+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:07.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:07.110+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:07.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:07.110+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:07.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:07.110+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:07.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:07.111+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:07.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:07.295+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:07.297261+0000) 2024-04-27T03:30:07.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:07.246+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:07.248563+0000) 2024-04-27T03:30:07.908 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:07.484+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:07.486732+0000) 2024-04-27T03:30:07.908 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:07.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:07.908 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:07.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:07.908 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:07.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:08.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:07.901+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:08.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:08.090+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:08.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:08.090+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:08.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:08.090+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:08.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:08.091+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:08.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:08.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:08.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:08.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:08.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:08.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:08.211 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:08.111+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:08.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:08.297+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:08.299396+0000) 2024-04-27T03:30:08.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:08.249+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:08.250815+0000) 2024-04-27T03:30:08.859 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:08.487+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:08.488998+0000) 2024-04-27T03:30:08.859 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:08.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:09.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:08.853+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:09.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:08.853+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:09.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:08.853+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:09.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:09.089+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:09.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:09.089+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:09.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:09.089+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:09.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:09.091+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:09.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:09.299+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:09.301480+0000) 2024-04-27T03:30:09.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:09.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:09.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:09.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:09.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:09.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:09.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:09.136+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:09.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:09.251+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:09.253478+0000) 2024-04-27T03:30:09.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:09.489+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:09.491397+0000) 2024-04-27T03:30:10.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:10.056+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:10.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:10.056+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:10.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:10.056+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:10.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:10.057+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:10.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:09.807+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:10.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:09.807+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:10.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:09.807+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:10.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:09.807+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:10.408 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:10.301+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:10.303509+0000) 2024-04-27T03:30:10.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:10.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:10.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:10.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:10.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:10.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:10.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:10.157+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:10.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:10.254+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:10.255847+0000) 2024-04-27T03:30:10.691 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:30:10.409+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64084 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:30:10.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:10.404+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:10.406346+0000) 2024-04-27T03:30:10.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:10.405+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:10.407685+0000) 2024-04-27T03:30:10.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:10.489+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:10.491629+0000) 2024-04-27T03:30:10.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:10.406+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:10.407951+0000) 2024-04-27T03:30:10.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:10.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:10.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:10.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:10.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:10.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:10.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:10.852+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:11.261 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:11.192+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:11.261 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:11.192+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:11.261 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:11.192+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:11.261 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:11.193+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:11.261 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:11.048+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:11.262 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:11.048+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:11.262 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:11.048+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:11.262 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:11.049+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:11.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:11.302+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:11.303821+0000) 2024-04-27T03:30:11.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:11.492+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:11.493980+0000) 2024-04-27T03:30:11.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:11.254+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:11.256132+0000) 2024-04-27T03:30:12.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:11.811+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:12.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:11.811+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:12.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:11.811+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:12.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:11.811+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:12.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:12.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:12.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:12.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:12.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:12.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:12.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:12.023+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:12.434 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:12.305+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:12.307001+0000) 2024-04-27T03:30:12.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:12.200+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:12.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:12.200+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:12.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:12.200+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:12.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:12.201+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:12.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:12.257+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:12.259025+0000) 2024-04-27T03:30:12.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:12.495+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:12.497109+0000) 2024-04-27T03:30:13.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:12.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:13.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:12.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:13.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:12.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:13.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:12.857+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:13.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:13.072+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:13.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:13.072+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:13.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:13.072+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:13.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:13.073+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:13.434 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:13.307+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:13.309658+0000) 2024-04-27T03:30:13.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:13.160+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:13.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:13.160+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:13.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:13.160+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:13.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:13.161+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:13.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:13.261+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:13.263281+0000) 2024-04-27T03:30:13.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:13.498+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:13.499952+0000) 2024-04-27T03:30:14.086 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:13.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:14.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:13.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:14.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:13.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:14.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:13.820+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:14.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:14.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:14.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:14.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:14.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:14.184+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:14.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:14.185+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:14.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:14.265+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:14.267542+0000) 2024-04-27T03:30:14.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:14.079+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:14.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:14.079+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:14.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:14.079+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:14.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:14.081+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:14.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:14.310+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:14.311912+0000) 2024-04-27T03:30:14.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:14.501+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:14.503596+0000) 2024-04-27T03:30:15.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:14.833+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:15.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:14.833+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:15.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:14.833+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:15.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:14.834+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:15.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:15.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:15.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:15.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:15.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:15.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:15.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:15.175+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:15.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:15.268+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:15.269916+0000) 2024-04-27T03:30:15.418 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:15.116+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:15.418 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:15.116+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:15.419 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:15.116+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:15.419 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:15.117+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:15.419 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:15.311+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:15.313737+0000) 2024-04-27T03:30:15.725 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:30:15.418+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64124 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:30:15.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:15.414+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:15.416176+0000) 2024-04-27T03:30:15.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:15.415+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:15.417032+0000) 2024-04-27T03:30:15.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:15.415+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:15.416976+0000) 2024-04-27T03:30:15.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:15.502+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:15.503828+0000) 2024-04-27T03:30:16.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:15.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:16.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:15.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:16.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:15.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:16.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:15.843+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:16.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:16.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:16.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:16.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:16.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:16.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:16.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:16.163+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:16.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:16.312+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:16.313935+0000) 2024-04-27T03:30:16.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:16.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:16.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:16.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:16.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:16.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:16.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:16.216+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:16.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:16.268+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:16.270105+0000) 2024-04-27T03:30:16.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:16.504+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:16.506102+0000) 2024-04-27T03:30:17.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:16.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:17.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:16.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:17.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:16.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:17.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:16.821+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:17.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:17.136+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:17.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:17.136+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:17.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:17.136+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:17.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:17.137+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:17.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:17.316+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:17.318768+0000) 2024-04-27T03:30:17.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:17.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:17.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:17.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:17.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:17.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:17.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:17.213+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:17.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:17.270+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:17.272730+0000) 2024-04-27T03:30:17.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:17.508+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:17.510347+0000) 2024-04-27T03:30:18.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:17.831+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:18.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:17.831+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:18.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:17.831+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:18.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:17.832+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:18.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:18.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:18.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:18.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:18.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:18.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:18.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:18.177+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:18.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:18.274+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:18.274820+0000) 2024-04-27T03:30:18.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:18.184+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:18.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:18.184+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:18.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:18.184+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:18.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:18.184+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:18.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:18.323+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:18.323932+0000) 2024-04-27T03:30:18.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:18.513+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:18.514400+0000) 2024-04-27T03:30:19.090 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:18.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:19.090 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:18.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:19.090 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:18.814+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:19.091 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:18.814+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:19.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:19.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:19.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:19.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:19.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:19.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:19.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:19.140+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:19.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:19.325+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:19.326027+0000) 2024-04-27T03:30:19.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:19.187+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:19.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:19.187+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:19.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:19.187+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:19.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:19.188+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:19.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:19.276+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:19.277525+0000) 2024-04-27T03:30:19.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:19.515+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:19.516282+0000) 2024-04-27T03:30:20.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:19.828+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:20.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:19.828+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:20.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:19.828+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:20.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:19.829+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:20.426 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:20.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:20.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:20.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:20.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:20.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:20.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:20.149+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:20.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:20.278+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:20.279653+0000) 2024-04-27T03:30:20.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:20.147+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:20.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:20.147+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:20.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:20.147+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:20.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:20.148+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:20.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:20.328+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:20.328810+0000) 2024-04-27T03:30:20.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:30:20.426+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64163 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:30:20.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:20.423+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:20.423931+0000) 2024-04-27T03:30:20.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:20.424+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:20.425513+0000) 2024-04-27T03:30:20.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:20.424+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:20.425176+0000) 2024-04-27T03:30:20.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:20.515+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:20.516598+0000) 2024-04-27T03:30:21.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:20.855+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:21.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:20.855+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:21.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:20.855+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:21.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:20.856+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:21.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:21.160+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:21.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:21.160+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:21.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:21.160+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:21.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:21.161+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:21.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:21.279+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:21.279932+0000) 2024-04-27T03:30:21.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:21.135+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:21.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:21.135+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:21.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:21.135+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:21.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:21.136+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:21.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:21.328+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:21.328989+0000) 2024-04-27T03:30:21.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:21.518+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:21.519439+0000) 2024-04-27T03:30:22.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:21.809+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:22.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:21.809+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:22.109 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:21.809+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:22.109 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:21.810+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:22.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:22.102+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:22.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:22.102+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:22.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:22.102+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:22.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:22.103+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:22.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:22.330+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:22.331612+0000) 2024-04-27T03:30:22.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:22.168+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:22.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:22.168+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:22.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:22.168+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:22.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:22.169+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:22.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:22.282+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:22.283084+0000) 2024-04-27T03:30:22.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:22.521+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:22.522391+0000) 2024-04-27T03:30:23.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:22.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:23.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:22.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:23.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:22.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:23.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:22.858+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:23.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:23.056+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:23.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:23.056+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:23.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:23.056+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:23.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:23.057+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:23.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:23.334+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:23.334817+0000) 2024-04-27T03:30:23.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:23.185+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:23.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:23.185+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:23.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:23.185+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:23.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:23.186+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:23.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:23.284+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:23.285722+0000) 2024-04-27T03:30:23.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:23.525+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:23.526669+0000) 2024-04-27T03:30:24.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:24.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:24.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:24.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:24.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:24.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:24.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:24.067+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:24.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:23.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:24.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:23.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:24.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:23.819+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:24.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:23.819+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:24.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:24.336+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:24.337457+0000) 2024-04-27T03:30:24.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:24.198+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:24.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:24.198+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:24.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:24.198+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:24.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:24.199+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:24.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:24.287+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:24.287936+0000) 2024-04-27T03:30:24.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:24.528+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:24.529240+0000) 2024-04-27T03:30:25.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:24.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:25.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:24.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:25.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:24.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:25.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:24.857+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:25.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:25.071+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:25.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:25.071+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:25.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:25.071+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:25.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:25.072+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:25.433 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:25.338+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:25.339330+0000) 2024-04-27T03:30:25.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:25.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:25.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:25.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:25.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:25.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:25.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:25.186+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:25.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:25.288+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:25.289793+0000) 2024-04-27T03:30:25.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:25.428+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:25.433029+0000) 2024-04-27T03:30:25.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:30:25.433+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64199 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:30:25.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:25.428+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:25.432774+0000) 2024-04-27T03:30:25.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:25.428+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:25.432372+0000) 2024-04-27T03:30:25.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:25.528+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:25.529440+0000) 2024-04-27T03:30:26.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:25.906+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:26.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:25.906+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:26.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:25.906+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:26.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:25.907+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:26.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:26.112+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:26.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:26.112+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:26.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:26.112+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:26.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:26.113+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:26.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:26.338+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:26.339528+0000) 2024-04-27T03:30:26.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:26.208+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:26.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:26.208+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:26.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:26.208+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:26.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:26.209+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:26.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:26.289+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:26.290020+0000) 2024-04-27T03:30:26.873 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:26.531+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:26.532283+0000) 2024-04-27T03:30:27.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:26.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:27.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:26.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:27.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:26.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:27.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:26.868+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:27.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:27.107+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:27.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:27.107+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:27.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:27.107+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:27.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:27.108+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:27.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:27.341+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:27.342361+0000) 2024-04-27T03:30:27.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:27.230+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:27.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:27.230+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:27.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:27.230+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:27.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:27.231+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:27.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:27.291+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:27.292691+0000) 2024-04-27T03:30:27.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:27.534+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:27.535527+0000) 2024-04-27T03:30:28.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:27.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:28.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:27.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:28.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:27.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:28.108 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:27.848+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:28.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:28.251+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:28.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:28.251+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:28.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:28.251+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:28.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:28.253+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:28.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:28.295+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:28.295963+0000) 2024-04-27T03:30:28.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:28.102+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:28.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:28.102+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:28.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:28.102+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:28.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:28.103+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:28.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:28.344+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:28.345463+0000) 2024-04-27T03:30:28.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:28.537+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:28.538402+0000) 2024-04-27T03:30:29.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:28.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:29.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:28.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:29.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:28.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:29.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:28.817+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:29.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:29.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:29.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:29.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:29.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:29.212+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:29.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:29.213+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:29.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:29.297+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:29.298658+0000) 2024-04-27T03:30:29.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:29.071+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:29.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:29.071+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:29.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:29.071+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:29.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:29.071+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:29.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:29.346+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:29.347753+0000) 2024-04-27T03:30:29.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:29.540+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:29.541218+0000) 2024-04-27T03:30:30.111 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:29.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:30.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:29.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:30.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:29.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:30.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:29.849+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:30.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:30.231+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:30.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:30.231+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:30.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:30.231+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:30.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:30.231+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:30.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:30.301+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:30.302354+0000) 2024-04-27T03:30:30.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:30.105+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:30.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:30.105+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:30.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:30.105+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:30.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:30.107+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:30.438 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:30.349+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:30.350377+0000) 2024-04-27T03:30:30.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:30.440+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:30.441591+0000) 2024-04-27T03:30:30.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:30:30.442+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64238 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:30:30.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:30.439+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:30.440258+0000) 2024-04-27T03:30:30.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:30.439+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:30.440542+0000) 2024-04-27T03:30:30.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:30.540+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:30.541476+0000) 2024-04-27T03:30:31.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:30.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:31.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:30.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:31.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:30.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:31.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:30.892+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:31.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:31.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:31.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:31.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:31.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:31.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:31.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:31.156+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:31.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:31.349+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:31.350541+0000) 2024-04-27T03:30:31.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:31.281+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:31.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:31.281+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:31.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:31.281+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:31.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:31.282+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:31.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:31.301+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:31.302567+0000) 2024-04-27T03:30:31.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:31.544+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:31.545742+0000) 2024-04-27T03:30:32.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:31.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:32.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:31.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:32.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:31.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:32.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:31.931+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:32.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:32.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:32.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:32.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:32.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:32.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:32.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:32.162+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:32.458 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:32.351+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:32.352552+0000) 2024-04-27T03:30:32.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:32.298+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:32.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:32.298+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:32.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:32.298+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:32.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:32.299+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:32.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:32.304+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:32.305431+0000) 2024-04-27T03:30:32.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:32.549+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:32.549975+0000) 2024-04-27T03:30:33.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:32.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:33.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:32.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:33.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:32.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:33.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:32.932+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:33.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:33.307+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:33.308285+0000) 2024-04-27T03:30:33.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:33.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:33.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:33.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:33.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:33.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:33.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:33.340+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:33.459 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:33.195+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:33.459 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:33.195+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:33.459 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:33.195+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:33.459 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:33.195+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:33.459 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:33.354+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:33.355589+0000) 2024-04-27T03:30:33.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:33.551+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:33.552170+0000) 2024-04-27T03:30:34.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:33.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:34.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:33.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:34.148 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:33.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:34.148 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:33.947+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:34.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:34.310+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:34.311265+0000) 2024-04-27T03:30:34.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:34.380+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:34.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:34.380+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:34.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:34.380+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:34.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:34.381+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:34.438 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:34.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:34.438 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:34.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:34.438 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:34.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:34.438 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:34.215+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:34.438 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:34.356+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:34.357718+0000) 2024-04-27T03:30:34.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:34.553+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:34.554245+0000) 2024-04-27T03:30:35.212 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:34.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:35.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:34.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:35.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:34.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:35.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:34.932+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:35.466 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:30:35.449+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64278 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:30:35.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:35.446+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:35.447726+0000) 2024-04-27T03:30:35.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:35.314+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:35.314921+0000) 2024-04-27T03:30:35.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:35.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:35.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:35.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:35.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:35.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:35.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:35.333+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:35.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:35.448+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:35.449123+0000) 2024-04-27T03:30:35.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:35.257+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:35.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:35.257+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:35.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:35.257+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:35.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:35.258+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:35.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:35.358+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:35.359723+0000) 2024-04-27T03:30:35.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:35.446+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:35.447494+0000) 2024-04-27T03:30:35.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:35.553+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:35.554531+0000) 2024-04-27T03:30:36.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:35.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:36.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:35.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:36.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:35.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:36.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:35.956+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:36.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:36.297+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:36.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:36.297+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:36.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:36.297+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:36.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:36.299+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:36.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:36.314+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:36.315105+0000) 2024-04-27T03:30:36.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:36.220+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:36.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:36.220+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:36.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:36.220+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:36.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:36.221+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:36.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:36.359+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:36.359921+0000) 2024-04-27T03:30:36.940 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:36.557+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:36.558743+0000) 2024-04-27T03:30:37.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:36.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:37.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:36.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:37.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:36.934+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:37.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:36.935+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:37.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:37.317+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:37.318222+0000) 2024-04-27T03:30:37.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:37.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:37.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:37.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:37.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:37.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:37.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:37.322+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:37.463 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:37.194+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:37.463 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:37.194+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:37.463 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:37.194+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:37.463 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:37.195+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:37.463 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:37.361+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:37.362011+0000) 2024-04-27T03:30:37.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:37.560+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:37.561059+0000) 2024-04-27T03:30:38.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:38.186+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:38.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:38.186+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:38.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:38.186+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:38.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:38.187+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:38.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:37.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:38.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:37.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:38.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:37.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:38.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:37.981+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:38.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:38.363+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:38.364160+0000) 2024-04-27T03:30:38.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:38.563+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:38.564170+0000) 2024-04-27T03:30:38.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:38.321+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:38.322438+0000) 2024-04-27T03:30:38.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:38.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:38.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:38.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:38.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:38.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:38.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:38.335+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:39.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:38.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:39.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:38.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:39.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:38.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:39.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:38.982+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:39.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:39.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:39.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:39.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:39.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:39.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:39.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:39.290+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:39.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:39.324+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:39.325226+0000) 2024-04-27T03:30:39.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:39.223+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:39.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:39.223+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:39.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:39.223+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:39.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:39.224+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:39.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:39.368+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:39.368949+0000) 2024-04-27T03:30:39.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:39.565+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:39.566769+0000) 2024-04-27T03:30:40.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:40.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:40.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:40.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:40.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:40.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:40.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:40.010+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:40.306 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:40.178+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:40.306 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:40.178+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:40.306 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:40.178+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:40.306 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:40.179+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:40.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:40.453+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:40.454572+0000) 2024-04-27T03:30:40.572 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:30:40.456+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64317 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:30:40.572 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:40.370+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:40.370821+0000) 2024-04-27T03:30:40.572 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:40.455+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:40.456366+0000) 2024-04-27T03:30:40.572 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:40.316+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:40.573 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:40.316+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:40.573 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:40.316+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:40.573 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:40.317+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:40.573 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:40.326+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:40.327695+0000) 2024-04-27T03:30:40.573 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:40.455+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:40.456124+0000) 2024-04-27T03:30:40.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:40.566+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:40.566986+0000) 2024-04-27T03:30:41.333 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:41.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:41.333 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:41.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:41.333 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:41.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:41.333 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:41.010+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:41.333 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:41.225+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:41.333 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:41.225+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:41.333 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:41.225+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:41.333 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:41.226+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:41.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:41.370+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:41.371004+0000) 2024-04-27T03:30:41.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:41.569+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:41.570165+0000) 2024-04-27T03:30:41.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:41.327+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:41.327892+0000) 2024-04-27T03:30:41.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:41.343+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:41.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:41.343+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:41.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:41.343+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:41.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:41.344+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:42.337 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:42.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:42.337 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:42.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:42.337 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:42.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:42.337 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:42.203+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:42.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:42.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:42.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:42.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:42.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:42.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:42.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:42.057+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:42.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:42.372+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:42.373662+0000) 2024-04-27T03:30:42.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:42.572+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:42.573209+0000) 2024-04-27T03:30:42.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:42.331+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:42.331944+0000) 2024-04-27T03:30:42.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:42.370+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:42.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:42.370+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:42.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:42.370+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:42.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:42.372+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:43.339 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:43.183+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:43.340 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:43.183+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:43.340 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:43.183+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:43.340 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:43.184+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:43.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:43.067+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:43.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:43.067+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:43.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:43.067+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:43.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:43.067+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:43.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:43.376+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:43.376843+0000) 2024-04-27T03:30:43.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:43.575+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:43.575824+0000) 2024-04-27T03:30:43.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:43.334+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:43.334826+0000) 2024-04-27T03:30:43.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:43.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:43.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:43.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:43.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:43.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:43.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:43.400+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:44.148 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:44.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:44.148 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:44.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:44.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:44.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:44.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:44.046+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:44.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:44.336+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:44.337134+0000) 2024-04-27T03:30:44.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:44.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:44.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:44.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:44.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:44.375+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:44.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:44.375+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:44.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:44.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:44.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:44.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:44.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:44.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:44.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:44.162+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:44.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:44.378+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:44.379025+0000) 2024-04-27T03:30:44.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:44.577+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:44.578642+0000) 2024-04-27T03:30:45.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:45.126+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:45.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:45.126+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:45.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:45.126+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:45.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:45.126+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:45.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:45.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:45.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:45.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:45.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:45.020+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:45.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:45.021+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:45.725 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:30:45.464+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64357 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:30:45.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:45.460+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:45.461659+0000) 2024-04-27T03:30:45.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:45.578+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:45.578841+0000) 2024-04-27T03:30:45.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:45.381+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:45.382412+0000) 2024-04-27T03:30:45.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:45.462+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:45.462965+0000) 2024-04-27T03:30:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:45.338+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:45.339595+0000) 2024-04-27T03:30:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:45.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:45.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:45.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:45.411+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:45.461+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:45.462641+0000) 2024-04-27T03:30:46.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:46.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:46.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:46.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:46.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:46.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:46.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:46.030+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:46.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:46.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:46.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:46.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:46.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:46.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:46.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:46.134+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:46.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:46.381+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:46.382677+0000) 2024-04-27T03:30:46.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:46.580+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:46.581013+0000) 2024-04-27T03:30:46.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:46.339+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:46.339819+0000) 2024-04-27T03:30:46.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:46.374+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:46.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:46.374+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:46.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:46.374+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:46.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:46.375+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:47.347 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:47.110+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:47.348 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:47.110+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:47.348 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:47.110+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:47.348 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:47.111+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:47.348 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:47.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:47.348 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:47.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:47.348 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:47.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:47.348 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:47.008+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:47.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:47.386+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:47.387392+0000) 2024-04-27T03:30:47.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:47.582+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:47.583242+0000) 2024-04-27T03:30:47.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:47.341+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:47.342495+0000) 2024-04-27T03:30:47.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:47.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:47.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:47.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:47.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:47.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:47.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:47.414+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:48.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:48.057+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:48.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:48.057+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:48.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:48.057+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:48.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:48.058+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:48.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:48.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:48.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:48.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:48.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:48.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:48.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:48.132+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:48.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:48.391+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:48.392121+0000) 2024-04-27T03:30:48.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:48.584+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:48.585488+0000) 2024-04-27T03:30:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:48.344+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:48.345477+0000) 2024-04-27T03:30:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:48.411+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:48.411+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:48.411+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:48.412+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:49.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:49.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:49.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:49.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:49.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:49.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:49.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:49.012+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:49.354 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:49.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:49.354 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:49.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:49.354 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:49.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:49.354 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:49.098+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:49.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:49.586+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:49.587666+0000) 2024-04-27T03:30:49.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:49.393+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:49.394337+0000) 2024-04-27T03:30:49.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:49.347+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:49.348565+0000) 2024-04-27T03:30:49.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:49.390+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:49.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:49.390+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:49.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:49.390+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:49.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:49.391+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:50.306 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:50.132+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:50.306 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:50.133+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:50.306 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:50.133+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:50.306 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:50.134+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:50.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:50.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:50.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:50.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:50.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:50.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:50.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:50.024+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:50.592 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:30:50.471+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64396 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:30:50.593 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:50.468+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:50.469635+0000) 2024-04-27T03:30:50.593 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:50.397+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:50.397899+0000) 2024-04-27T03:30:50.593 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:50.468+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:50.469741+0000) 2024-04-27T03:30:50.593 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:50.350+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:50.351632+0000) 2024-04-27T03:30:50.594 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:50.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:50.594 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:50.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:50.594 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:50.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:50.594 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:50.422+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:50.594 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:50.468+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:50.469507+0000) 2024-04-27T03:30:50.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:50.587+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:50.587825+0000) 2024-04-27T03:30:51.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:51.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:51.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:51.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:51.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:51.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:51.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:51.052+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:51.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:51.116+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:51.358 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:51.116+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:51.358 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:51.116+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:51.358 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:51.117+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:51.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:51.591+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:51.592635+0000) 2024-04-27T03:30:51.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:51.397+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:51.398151+0000) 2024-04-27T03:30:51.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:51.351+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:51.351824+0000) 2024-04-27T03:30:51.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:51.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:51.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:51.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:51.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:51.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:51.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:51.422+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:52.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:52.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:52.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:52.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:52.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:52.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:52.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:52.070+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:52.359 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:52.095+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:52.360 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:52.095+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:52.360 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:52.095+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:52.360 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:52.096+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:52.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:52.402+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:52.402906+0000) 2024-04-27T03:30:52.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:52.594+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:52.595407+0000) 2024-04-27T03:30:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:52.353+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:52.354142+0000) 2024-04-27T03:30:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:52.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:52.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:52.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:52.461+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:53.361 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:53.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:53.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:53.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:53.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:53.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:53.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:53.030+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:53.362 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:53.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:53.362 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:53.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:53.362 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:53.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:53.362 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:53.067+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:53.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:53.406+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:53.407727+0000) 2024-04-27T03:30:53.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:53.597+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:53.598304+0000) 2024-04-27T03:30:53.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:53.355+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:53.356635+0000) 2024-04-27T03:30:53.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:53.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:53.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:53.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:53.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:53.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:53.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:53.482+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:54.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:54.030+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:54.150 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:54.030+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:54.150 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:54.030+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:54.150 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:54.030+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:54.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:54.071+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:54.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:54.071+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:54.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:54.071+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:54.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:54.072+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:54.406 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:54.360+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:54.360858+0000) 2024-04-27T03:30:54.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:54.411+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:54.412532+0000) 2024-04-27T03:30:54.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:54.599+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:54.600355+0000) 2024-04-27T03:30:54.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:54.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:54.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:54.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:54.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:54.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:54.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:54.488+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:55.066 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:55.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:55.066 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:55.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:55.066 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:55.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:55.066 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:55.022+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:55.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:55.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:55.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:55.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:55.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:55.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:55.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:55.100+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:55.725 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:30:55.479+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64436 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:30:55.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:55.476+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:55.476959+0000) 2024-04-27T03:30:55.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:55.599+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:55.600571+0000) 2024-04-27T03:30:55.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:55.414+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:55.415722+0000) 2024-04-27T03:30:55.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:55.476+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:55.477386+0000) 2024-04-27T03:30:55.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:55.362+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:55.362874+0000) 2024-04-27T03:30:55.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:55.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:55.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:55.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:55.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:55.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:55.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:55.464+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:55.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:55.475+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:55.476711+0000) 2024-04-27T03:30:56.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:55.993+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:56.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:55.993+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:56.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:55.993+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:56.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:55.993+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:56.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:56.073+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:56.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:56.073+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:56.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:56.073+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:56.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:56.073+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:56.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:56.602+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:56.603490+0000) 2024-04-27T03:30:56.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:56.415+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:56.415903+0000) 2024-04-27T03:30:56.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:56.362+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:56.363068+0000) 2024-04-27T03:30:56.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:56.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:56.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:56.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:56.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:56.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:56.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:56.455+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:57.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:56.964+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:57.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:56.964+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:57.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:56.964+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:57.048 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:56.966+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:57.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:57.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:57.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:57.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:57.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:57.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:57.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:57.042+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:57.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:57.605+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:57.606612+0000) 2024-04-27T03:30:57.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:57.417+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:57.418140+0000) 2024-04-27T03:30:57.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:57.367+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:57.368037+0000) 2024-04-27T03:30:57.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:57.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:57.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:57.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:57.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:57.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:57.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:57.505+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:58.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:57.930+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:58.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:57.930+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:58.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:57.930+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:18.155506+0000 (oldest deadline 2024-04-27T03:20:38.155506+0000) 2024-04-27T03:30:58.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:57.930+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:58.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:58.079+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:58.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:58.079+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:58.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:58.079+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:58.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:58.079+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:58.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:58.420+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:58.421185+0000) 2024-04-27T03:30:58.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:58.370+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:58.370932+0000) 2024-04-27T03:30:58.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:58.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:58.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:58.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:58.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:58.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:58.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:58.459+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:58.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:58.607+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:58.608753+0000) 2024-04-27T03:30:59.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:58.970+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:59.378 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:59.063+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:59.378 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:59.063+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:59.378 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:59.063+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:21.876715+0000 (oldest deadline 2024-04-27T03:20:41.876715+0000) 2024-04-27T03:30:59.378 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:59.063+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:30:59.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:59.422+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:59.423329+0000) 2024-04-27T03:30:59.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:30:59.611+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:59.611822+0000) 2024-04-27T03:30:59.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:59.372+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:30:59.373034+0000) 2024-04-27T03:30:59.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:59.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:59.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:59.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:59.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:59.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:20:28.346269+0000 (oldest deadline 2024-04-27T03:20:48.346269+0000) 2024-04-27T03:30:59.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:30:59.453+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:00.111 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:30:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:30:59.927+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:00.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:00.106+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:00.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:31:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:31:00.486+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64475 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:31:00.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:00.483+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:00.484761+0000) 2024-04-27T03:31:00.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:00.611+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:00.612041+0000) 2024-04-27T03:31:00.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:00.376+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:00.377568+0000) 2024-04-27T03:31:00.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:00.483+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:00.484162+0000) 2024-04-27T03:31:00.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:00.488+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:00.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:00.424+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:00.425301+0000) 2024-04-27T03:31:00.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:00.483+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:00.484748+0000) 2024-04-27T03:31:01.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:00.944+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:01.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:01.137+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:01.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:01.424+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:01.425553+0000) 2024-04-27T03:31:01.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:01.376+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:01.377782+0000) 2024-04-27T03:31:01.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:01.481+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:01.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:01.616+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:01.617107+0000) 2024-04-27T03:31:02.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:02.187+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:02.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:01.966+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:02.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:02.621+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:02.622205+0000) 2024-04-27T03:31:02.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:02.427+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:02.427840+0000) 2024-04-27T03:31:02.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:02.381+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:02.381978+0000) 2024-04-27T03:31:02.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:02.463+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:03.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:02.936+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:03.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:03.385+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:03.386217+0000) 2024-04-27T03:31:03.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:03.209+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:03.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:03.429+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:03.430048+0000) 2024-04-27T03:31:03.851 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:03.479+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:03.851 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:03.624+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:03.624990+0000) 2024-04-27T03:31:04.150 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:03.896+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:04.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:04.188+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:04.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:04.388+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:04.388823+0000) 2024-04-27T03:31:04.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:04.518+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:04.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:04.431+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:04.432762+0000) 2024-04-27T03:31:04.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:04.626+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:04.627063+0000) 2024-04-27T03:31:05.216 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:04.943+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:05.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:05.390+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:05.391698+0000) 2024-04-27T03:31:05.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:05.434+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:05.435534+0000) 2024-04-27T03:31:05.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:05.217+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:05.954 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:31:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:31:05.495+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64515 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:31:05.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:05.490+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:05.491044+0000) 2024-04-27T03:31:05.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:05.559+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:05.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:05.492+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:05.493048+0000) 2024-04-27T03:31:05.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:05.949+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:05.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:05.492+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:05.493304+0000) 2024-04-27T03:31:05.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:05.626+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:05.627291+0000) 2024-04-27T03:31:06.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:06.391+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:06.391905+0000) 2024-04-27T03:31:06.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:06.434+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:06.435741+0000) 2024-04-27T03:31:06.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:06.235+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:06.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:06.567+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:06.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:06.629+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:06.629791+0000) 2024-04-27T03:31:07.257 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:06.995+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:07.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:07.393+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:07.393994+0000) 2024-04-27T03:31:07.545 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:07.251+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:07.545 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:07.437+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:07.438203+0000) 2024-04-27T03:31:07.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:07.538+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:07.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:07.631+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:07.632464+0000) 2024-04-27T03:31:08.216 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:07.951+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:08.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:08.394+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:08.396095+0000) 2024-04-27T03:31:08.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:08.228+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:08.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:08.439+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:08.441001+0000) 2024-04-27T03:31:08.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:08.494+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:08.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:08.633+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:08.634829+0000) 2024-04-27T03:31:09.403 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:09.216+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:09.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:08.981+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:09.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:09.635+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:09.637781+0000) 2024-04-27T03:31:09.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:09.396+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:09.398794+0000) 2024-04-27T03:31:09.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:09.528+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:09.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:09.442+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:09.444107+0000) 2024-04-27T03:31:10.306 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:10.021+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:10.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:10.209+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:10.558 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:31:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:31:10.503+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64554 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:31:10.558 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:10.499+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:10.501597+0000) 2024-04-27T03:31:10.559 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:10.399+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:10.401524+0000) 2024-04-27T03:31:10.559 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:10.499+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:10.501107+0000) 2024-04-27T03:31:10.559 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:10.551+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:10.559 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:10.444+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:10.446172+0000) 2024-04-27T03:31:10.559 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:10.498+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:10.500759+0000) 2024-04-27T03:31:10.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:10.636+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:10.638008+0000) 2024-04-27T03:31:11.406 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:11.204+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:11.407 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:11.020+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:11.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:11.638+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:11.640311+0000) 2024-04-27T03:31:11.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:11.399+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:11.401793+0000) 2024-04-27T03:31:11.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:11.560+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:11.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:11.444+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:11.446335+0000) 2024-04-27T03:31:12.408 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:12.239+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:12.409 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:11.994+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:12.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:12.447+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:12.449007+0000) 2024-04-27T03:31:12.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:12.640+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:12.642467+0000) 2024-04-27T03:31:12.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:12.402+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:12.403933+0000) 2024-04-27T03:31:12.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:12.566+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:13.285 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:13.032+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:13.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:13.450+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:13.452177+0000) 2024-04-27T03:31:13.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:13.404+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:13.406611+0000) 2024-04-27T03:31:13.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:13.568+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:13.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:13.278+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:13.852 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:13.642+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:13.644665+0000) 2024-04-27T03:31:14.154 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:14.072+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:14.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:14.312+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:14.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:14.644+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:14.646723+0000) 2024-04-27T03:31:14.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:14.454+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:14.456380+0000) 2024-04-27T03:31:14.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:14.406+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:14.408729+0000) 2024-04-27T03:31:14.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:14.540+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:15.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:15.096+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:15.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:31:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:31:15.511+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64594 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:31:15.572 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:15.411+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:15.413068+0000) 2024-04-27T03:31:15.572 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:15.508+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:15.510483+0000) 2024-04-27T03:31:15.572 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:15.458+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:15.460294+0000) 2024-04-27T03:31:15.572 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:15.507+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:15.509437+0000) 2024-04-27T03:31:15.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:15.291+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:15.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:15.508+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:15.510083+0000) 2024-04-27T03:31:15.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:15.645+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:15.647501+0000) 2024-04-27T03:31:15.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:15.565+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:16.296 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:16.061+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:16.594 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:16.289+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:16.594 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:16.411+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:16.413346+0000) 2024-04-27T03:31:16.594 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:16.587+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:16.595 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:16.458+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:16.460469+0000) 2024-04-27T03:31:16.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:16.650+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:16.652183+0000) 2024-04-27T03:31:17.336 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:17.108+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:17.661 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:17.463+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:17.465502+0000) 2024-04-27T03:31:17.662 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:17.329+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:17.662 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:17.414+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:17.416352+0000) 2024-04-27T03:31:17.662 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:17.572+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:17.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:17.655+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:17.656909+0000) 2024-04-27T03:31:18.297 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:18.139+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:18.574 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:18.467+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:18.469618+0000) 2024-04-27T03:31:18.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:18.290+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:18.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:18.418+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:18.420583+0000) 2024-04-27T03:31:18.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:18.566+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:18.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:18.658+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:18.660137+0000) 2024-04-27T03:31:19.312 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:19.129+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:19.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:19.423+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:19.423961+0000) 2024-04-27T03:31:19.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:19.525+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:19.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:19.306+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:19.655 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:19.470+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:19.472535+0000) 2024-04-27T03:31:19.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:19.660+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:19.662711+0000) 2024-04-27T03:31:20.287 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:20.166+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:20.546 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:31:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:31:20.520+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64633 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:31:20.547 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:20.427+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:20.427823+0000) 2024-04-27T03:31:20.547 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:20.517+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:20.518365+0000) 2024-04-27T03:31:20.547 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:20.473+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:20.474579+0000) 2024-04-27T03:31:20.547 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:20.517+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:20.518129+0000) 2024-04-27T03:31:20.547 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:20.281+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:20.547 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:20.517+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:20.518058+0000) 2024-04-27T03:31:20.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:20.662+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:20.662919+0000) 2024-04-27T03:31:20.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:20.544+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:21.297 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:21.132+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:21.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:21.132+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:21.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:21.132+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:21.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:21.133+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:21.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:21.474+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:21.474846+0000) 2024-04-27T03:31:21.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:21.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:21.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:21.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:21.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:21.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:21.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:21.292+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:21.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:21.427+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:21.428039+0000) 2024-04-27T03:31:21.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:21.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:21.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:21.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:21.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:21.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:21.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:21.528+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:21.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:21.664+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:21.665014+0000) 2024-04-27T03:31:22.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:22.251+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:22.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:22.251+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:22.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:22.251+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:22.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:22.251+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:22.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:22.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:22.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:22.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:22.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:22.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:22.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:22.134+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:22.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:22.666+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:22.667688+0000) 2024-04-27T03:31:22.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:22.476+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:22.476963+0000) 2024-04-27T03:31:22.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:22.430+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:22.431181+0000) 2024-04-27T03:31:22.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:22.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:22.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:22.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:22.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:22.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:22.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:22.479+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:23.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:23.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:23.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:23.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:23.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:23.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:23.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:23.236+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:23.440 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:23.111+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:23.440 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:23.111+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:23.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:23.111+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:23.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:23.112+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:23.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:23.479+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:23.479820+0000) 2024-04-27T03:31:23.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:23.669+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:23.669892+0000) 2024-04-27T03:31:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:23.434+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:23.435346+0000) 2024-04-27T03:31:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:23.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:23.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:23.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:23.436+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:24.155 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:24.086+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:24.155 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:24.086+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:24.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:24.086+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:24.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:24.087+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:24.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:24.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:24.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:24.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:24.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:24.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:24.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:24.215+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:24.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:24.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:24.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:24.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:24.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:24.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:24.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:24.418+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:24.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:24.438+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:24.439582+0000) 2024-04-27T03:31:24.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:24.481+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:24.481952+0000) 2024-04-27T03:31:24.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:24.671+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:24.672019+0000) 2024-04-27T03:31:25.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:25.047+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:25.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:25.047+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:25.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:25.047+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:25.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:25.048+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:25.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:25.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:25.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:25.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:25.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:25.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:25.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:25.204+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:25.725 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:31:25.530+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64673 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:31:25.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:25.525+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:25.526634+0000) 2024-04-27T03:31:25.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:25.671+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:25.672256+0000) 2024-04-27T03:31:25.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:25.483+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:25.483953+0000) 2024-04-27T03:31:25.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:25.524+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:25.525575+0000) 2024-04-27T03:31:25.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:25.440+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:25.441667+0000) 2024-04-27T03:31:25.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:25.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:25.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:25.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:25.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:25.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:25.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:25.467+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:25.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:25.526+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:25.527031+0000) 2024-04-27T03:31:26.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:26.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:26.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:26.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:26.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:26.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:26.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:26.170+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:26.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:26.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:26.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:26.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:26.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:26.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:26.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:26.092+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:26.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:26.483+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:26.484158+0000) 2024-04-27T03:31:26.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:26.440+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:26.441781+0000) 2024-04-27T03:31:26.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:26.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:26.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:26.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:26.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:26.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:26.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:26.506+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:26.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:26.673+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:26.674454+0000) 2024-04-27T03:31:27.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:27.136+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:27.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:27.136+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:27.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:27.136+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:27.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:27.137+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:27.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:27.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:27.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:27.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:27.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:27.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:27.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:27.167+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:27.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:27.487+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:27.488474+0000) 2024-04-27T03:31:27.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:27.676+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:27.677339+0000) 2024-04-27T03:31:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:27.443+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:27.443847+0000) 2024-04-27T03:31:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:27.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:27.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:27.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:27.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:27.472+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:28.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:28.152+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:28.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:28.152+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:28.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:28.152+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:28.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:28.154+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:28.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:28.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:28.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:28.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:28.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:28.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:28.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:28.215+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:28.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:28.489+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:28.490646+0000) 2024-04-27T03:31:28.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:28.679+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:28.680541+0000) 2024-04-27T03:31:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:28.445+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:28.446072+0000) 2024-04-27T03:31:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:28.473+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:28.473+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:28.473+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:28.473+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:29.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:29.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:29.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:29.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:29.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:29.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:29.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:29.204+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:29.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:29.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:29.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:29.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:29.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:29.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:29.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:29.171+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:29.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:29.492+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:29.492976+0000) 2024-04-27T03:31:29.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:29.683+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:29.684502+0000) 2024-04-27T03:31:29.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:29.449+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:29.450294+0000) 2024-04-27T03:31:29.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:29.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:29.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:29.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:29.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:29.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:29.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:29.472+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:30.308 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:30.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:30.308 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:30.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:30.308 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:30.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:30.308 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:30.157+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:30.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:30.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:30.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:30.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:30.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:30.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:30.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:30.132+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:30.561 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:30.535+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:30.536539+0000) 2024-04-27T03:31:30.561 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:31:30.539+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64712 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:31:30.561 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:30.494+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:30.495080+0000) 2024-04-27T03:31:30.562 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:30.535+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:30.536224+0000) 2024-04-27T03:31:30.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:30.451+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:30.452238+0000) 2024-04-27T03:31:30.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:30.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:30.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:30.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:30.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:30.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:30.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:30.456+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:30.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:30.536+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:30.537133+0000) 2024-04-27T03:31:30.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:30.683+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:30.684716+0000) 2024-04-27T03:31:31.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:31.169+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:31.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:31.169+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:31.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:31.169+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:31.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:31.169+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:31.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:31.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:31.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:31.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:31.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:31.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:31.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:31.161+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:31.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:31.451+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:31.452504+0000) 2024-04-27T03:31:31.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:31.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:31.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:31.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:31.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:31.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:31.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:31.501+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:31.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:31.494+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:31.495251+0000) 2024-04-27T03:31:31.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:31.688+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:31.688954+0000) 2024-04-27T03:31:32.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:32.210+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:32.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:32.210+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:32.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:32.210+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:32.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:32.211+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:32.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:32.184+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:32.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:32.184+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:32.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:32.184+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:32.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:32.185+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:32.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:32.497+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:32.498196+0000) 2024-04-27T03:31:32.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:32.453+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:32.454601+0000) 2024-04-27T03:31:32.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:32.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:32.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:32.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:32.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:32.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:32.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:32.536+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:32.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:32.691+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:32.692035+0000) 2024-04-27T03:31:33.306 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:33.153+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:33.307 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:33.153+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:33.307 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:33.153+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:33.307 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:33.154+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:33.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:33.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:33.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:33.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:33.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:33.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:33.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:33.228+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:33.635 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:33.499+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:33.500302+0000) 2024-04-27T03:31:33.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:33.455+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:33.456786+0000) 2024-04-27T03:31:33.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:33.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:33.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:33.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:33.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:33.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:33.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:33.529+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:33.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:33.693+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:33.694697+0000) 2024-04-27T03:31:34.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:34.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:34.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:34.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:34.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:34.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:34.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:34.228+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:34.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:34.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:34.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:34.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:34.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:34.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:34.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:34.144+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:34.703 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:34.502+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:34.503022+0000) 2024-04-27T03:31:34.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:34.459+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:34.460054+0000) 2024-04-27T03:31:34.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:34.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:34.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:34.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:34.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:34.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:34.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:34.518+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:34.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:34.697+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:34.698424+0000) 2024-04-27T03:31:35.263 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:35.164+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:35.263 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:35.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:35.264 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:35.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:35.264 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:35.165+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:35.541 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:35.504+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:35.504954+0000) 2024-04-27T03:31:35.541 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:35.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:35.542 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:35.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:35.542 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:35.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:35.542 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:35.258+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:35.542 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:35.463+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:35.464462+0000) 2024-04-27T03:31:35.542 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:35.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:35.542 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:35.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:35.542 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:35.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:35.542 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:35.487+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:35.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:35.544+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:35.545156+0000) 2024-04-27T03:31:35.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:31:35.547+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64752 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:31:35.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:35.543+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:35.544696+0000) 2024-04-27T03:31:35.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:35.544+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:35.545649+0000) 2024-04-27T03:31:35.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:35.697+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:35.698628+0000) 2024-04-27T03:31:36.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:36.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:36.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:36.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:36.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:36.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:36.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:36.150+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:36.705 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:36.504+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:36.505155+0000) 2024-04-27T03:31:36.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:36.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:36.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:36.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:36.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:36.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:36.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:36.442+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:36.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:36.463+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:36.464729+0000) 2024-04-27T03:31:36.706 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:36.293+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:36.706 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:36.293+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:36.706 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:36.293+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:36.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:36.294+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:36.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:36.700+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:36.700842+0000) 2024-04-27T03:31:37.334 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:37.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:37.334 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:37.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:37.334 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:37.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:37.334 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:37.123+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:37.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:37.507+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:37.507801+0000) 2024-04-27T03:31:37.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:37.468+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:37.469388+0000) 2024-04-27T03:31:37.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:37.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:37.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:37.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:37.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:37.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:37.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:37.479+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:37.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:37.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:37.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:37.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:37.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:37.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:37.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:37.328+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:37.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:37.704+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:37.704798+0000) 2024-04-27T03:31:38.288 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:38.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:38.289 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:38.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:38.289 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:38.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:38.289 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:38.161+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:38.545 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:38.509+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:38.509825+0000) 2024-04-27T03:31:38.546 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:38.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:38.546 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:38.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:38.546 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:38.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:38.546 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:38.460+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:38.546 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:38.473+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:38.474034+0000) 2024-04-27T03:31:38.546 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:38.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:38.547 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:38.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:38.547 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:38.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:38.547 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:38.283+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:38.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:38.706+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:38.707465+0000) 2024-04-27T03:31:39.287 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:39.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:39.287 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:39.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:39.287 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:39.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:39.287 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:39.161+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:39.546 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:39.511+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:39.511955+0000) 2024-04-27T03:31:39.546 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:39.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:39.546 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:39.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:39.546 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:39.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:39.546 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:39.282+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:39.547 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:39.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:39.547 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:39.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:39.547 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:39.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:39.547 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:39.450+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:39.548 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:39.475+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:39.476165+0000) 2024-04-27T03:31:39.825 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:39.708+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:39.709556+0000) 2024-04-27T03:31:40.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:40.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:40.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:40.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:40.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:40.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:40.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:40.260+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:40.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:40.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:40.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:40.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:40.162+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:40.715 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:31:40.555+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64791 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:31:40.715 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:40.553+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:40.554052+0000) 2024-04-27T03:31:40.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:40.479+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:40.480543+0000) 2024-04-27T03:31:40.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:40.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:40.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:40.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:40.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:40.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:40.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:40.483+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:40.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:40.552+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:40.553606+0000) 2024-04-27T03:31:40.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:40.515+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:40.515975+0000) 2024-04-27T03:31:40.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:40.552+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:40.553647+0000) 2024-04-27T03:31:40.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:40.708+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:40.709761+0000) 2024-04-27T03:31:41.297 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:41.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:41.297 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:41.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:41.297 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:41.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:41.297 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:41.140+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:41.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:41.290+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:41.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:41.290+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:41.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:41.290+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:41.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:41.291+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:41.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:41.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:41.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:41.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:41.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:41.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:41.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:41.476+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:41.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:41.479+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:41.480673+0000) 2024-04-27T03:31:41.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:41.515+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:41.516171+0000) 2024-04-27T03:31:41.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:41.713+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:41.714639+0000) 2024-04-27T03:31:42.290 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:42.121+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:42.290 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:42.121+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:42.290 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:42.121+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:42.291 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:42.122+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:42.551 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:42.518+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:42.519188+0000) 2024-04-27T03:31:42.551 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:42.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:42.551 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:42.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:42.551 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:42.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:42.551 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:42.285+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:42.551 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:42.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:42.551 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:42.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:42.552 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:42.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:42.552 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:42.443+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:42.552 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:42.482+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:42.482821+0000) 2024-04-27T03:31:42.851 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:42.716+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:42.717714+0000) 2024-04-27T03:31:43.401 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:43.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:43.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:43.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:43.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:43.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:43.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:43.316+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:43.402 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:43.115+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:43.402 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:43.115+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:43.402 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:43.115+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:43.402 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:43.116+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:43.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:43.521+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:43.521905+0000) 2024-04-27T03:31:43.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:43.396+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:43.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:43.396+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:43.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:43.396+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:43.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:43.397+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:43.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:43.484+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:43.485489+0000) 2024-04-27T03:31:44.012 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:43.720+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:43.720983+0000) 2024-04-27T03:31:44.326 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:44.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:44.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:44.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:44.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:44.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:44.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:44.119+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:44.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:44.525+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:44.526268+0000) 2024-04-27T03:31:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:44.411+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:44.411+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:44.411+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:44.411+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:44.488+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:44.489745+0000) 2024-04-27T03:31:44.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:44.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:44.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:44.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:44.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:44.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:44.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:44.321+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:45.012 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:44.722+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:44.723686+0000) 2024-04-27T03:31:45.347 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:45.138+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:45.347 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:45.138+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:45.347 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:45.138+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:45.347 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:45.139+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:45.686 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:31:45.562+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64831 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:31:45.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:45.528+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:45.529036+0000) 2024-04-27T03:31:45.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:45.560+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:45.560903+0000) 2024-04-27T03:31:45.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:45.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:45.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:45.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:45.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:45.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:45.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:45.341+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:45.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:45.559+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:45.560783+0000) 2024-04-27T03:31:45.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:45.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:45.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:45.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:45.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:45.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:45.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:45.431+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:45.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:45.492+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:45.492875+0000) 2024-04-27T03:31:45.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:45.559+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:45.560724+0000) 2024-04-27T03:31:45.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:45.723+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:45.723867+0000) 2024-04-27T03:31:46.335 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:46.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:46.336 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:46.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:46.336 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:46.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:46.336 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:46.161+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:46.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:46.528+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:46.529262+0000) 2024-04-27T03:31:46.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:46.329+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:46.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:46.329+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:46.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:46.329+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:46.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:46.330+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:46.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:46.387+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:46.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:46.387+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:46.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:46.387+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:46.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:46.388+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:46.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:46.492+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:46.493079+0000) 2024-04-27T03:31:47.012 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:46.725+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:46.726093+0000) 2024-04-27T03:31:47.328 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:47.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:47.328 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:47.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:47.328 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:47.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:47.328 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:47.147+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:47.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:47.530+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:47.531382+0000) 2024-04-27T03:31:47.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:47.374+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:47.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:47.374+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:47.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:47.374+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:47.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:47.375+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:47.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:47.495+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:47.496170+0000) 2024-04-27T03:31:47.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:47.322+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:47.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:47.322+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:47.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:47.322+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:47.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:47.323+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:48.013 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:47.727+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:47.728162+0000) 2024-04-27T03:31:48.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:48.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:48.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:48.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:48.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:48.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:48.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:48.099+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:48.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:48.532+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:48.533652+0000) 2024-04-27T03:31:48.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:48.349+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:48.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:48.349+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:48.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:48.349+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:48.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:48.350+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:48.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:48.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:48.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:48.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:48.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:48.387+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:48.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:48.498+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:48.498835+0000) 2024-04-27T03:31:49.013 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:48.730+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:48.730859+0000) 2024-04-27T03:31:49.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:49.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:49.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:49.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:49.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:49.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:49.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:49.126+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:49.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T03:31:49.610+0000 7ff0e4ed9700 0 monclient(hunting): authenticate timed out after 300 2024-04-27T03:31:49.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:49.534+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:49.535787+0000) 2024-04-27T03:31:49.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:49.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:49.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:49.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:49.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:49.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:49.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:49.397+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:49.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:49.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:49.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:49.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:49.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:49.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:49.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:49.369+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:49.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:49.500+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:49.501098+0000) 2024-04-27T03:31:50.013 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:49.734+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:49.735411+0000) 2024-04-27T03:31:50.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:50.102+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:50.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:50.102+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:50.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:50.102+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:50.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:50.102+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:50.570 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:50.539+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:50.540169+0000) 2024-04-27T03:31:50.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:50.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:50.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:50.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:50.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:50.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:50.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:50.442+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:50.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:50.341+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:50.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:50.341+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:50.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:50.341+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:50.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:50.343+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:50.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:50.502+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:50.503248+0000) 2024-04-27T03:31:50.975 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:31:50.571+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64870 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:31:50.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:50.567+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:50.568750+0000) 2024-04-27T03:31:50.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:50.568+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:50.569149+0000) 2024-04-27T03:31:50.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:50.567+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:50.568482+0000) 2024-04-27T03:31:50.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:50.734+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:50.735697+0000) 2024-04-27T03:31:51.390 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:51.133+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:51.391 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:51.133+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:51.391 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:51.133+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:51.391 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:51.134+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:51.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:51.539+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:51.540443+0000) 2024-04-27T03:31:51.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:51.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:51.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:51.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:51.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:51.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:51.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:51.432+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:51.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:51.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:51.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:51.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:51.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:51.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:51.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:51.386+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:51.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:51.502+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:51.503435+0000) 2024-04-27T03:31:52.013 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:51.737+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:51.737960+0000) 2024-04-27T03:31:52.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:52.158+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:52.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:52.158+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:52.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:52.158+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:52.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:52.158+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:52.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:52.541+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:52.542590+0000) 2024-04-27T03:31:52.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:52.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:52.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:52.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:52.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:52.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:52.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:52.409+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:52.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:52.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:52.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:52.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:52.426+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:52.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:52.504+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:52.505471+0000) 2024-04-27T03:31:53.013 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:52.739+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:52.740082+0000) 2024-04-27T03:31:53.386 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:53.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:53.386 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:53.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:53.386 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:53.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:53.386 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:53.143+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:53.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:53.543+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:53.544731+0000) 2024-04-27T03:31:53.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:53.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:53.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:53.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:53.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:53.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:53.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:53.381+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:53.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:53.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:53.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:53.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:53.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:53.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:53.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:53.454+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:53.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:53.506+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:53.507517+0000) 2024-04-27T03:31:54.013 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:53.741+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:53.742353+0000) 2024-04-27T03:31:54.382 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:54.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:54.382 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:54.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:54.382 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:54.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:54.382 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:54.163+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:54.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:54.546+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:54.546824+0000) 2024-04-27T03:31:54.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:54.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:54.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:54.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:54.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:54.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:54.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:54.376+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:54.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:54.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:54.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:54.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:54.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:54.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:54.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:54.435+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:54.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:54.508+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:54.509609+0000) 2024-04-27T03:31:55.013 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:54.744+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:54.745223+0000) 2024-04-27T03:31:55.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:55.167+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:55.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:55.167+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:55.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:55.167+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:55.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:55.168+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:55.592 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:31:55.579+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64910 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:31:55.592 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:55.549+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:55.550018+0000) 2024-04-27T03:31:55.592 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:55.577+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:55.577875+0000) 2024-04-27T03:31:55.593 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:55.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:55.593 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:55.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:55.593 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:55.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:55.593 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:55.462+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:55.593 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:55.510+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:55.511595+0000) 2024-04-27T03:31:55.593 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:55.576+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:55.576801+0000) 2024-04-27T03:31:55.593 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:55.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:55.594 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:55.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:55.594 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:55.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:55.594 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:55.336+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:55.594 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:55.575+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:55.576579+0000) 2024-04-27T03:31:55.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:55.744+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:55.745422+0000) 2024-04-27T03:31:56.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:56.137+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:56.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:56.137+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:56.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:56.137+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:56.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:56.138+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:56.555 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:56.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:56.555 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:56.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:56.555 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:56.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:56.555 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:56.443+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:56.555 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:56.511+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:56.511794+0000) 2024-04-27T03:31:56.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:56.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:56.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:56.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:56.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:56.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:56.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:56.292+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:56.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:56.549+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:56.550219+0000) 2024-04-27T03:31:56.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:56.749+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:56.750476+0000) 2024-04-27T03:31:57.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:57.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:57.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:57.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:57.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:57.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:57.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:57.093+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:57.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:57.250+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:57.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:57.250+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:57.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:57.250+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:57.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:57.251+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:57.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:57.551+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:57.552725+0000) 2024-04-27T03:31:57.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:57.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:57.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:57.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:57.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:57.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:57.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:57.476+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:57.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:57.515+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:57.516023+0000) 2024-04-27T03:31:58.014 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:57.754+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:57.755583+0000) 2024-04-27T03:31:58.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:58.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:58.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:58.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:58.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:58.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:58.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:58.217+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:58.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:58.061+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:58.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:58.061+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:58.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:58.061+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:58.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:58.062+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:58.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:58.556+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:58.557433+0000) 2024-04-27T03:31:58.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:58.520+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:58.521079+0000) 2024-04-27T03:31:58.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:58.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:58.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:58.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:58.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:58.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:58.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:58.522+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:59.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:58.759+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:58.760693+0000) 2024-04-27T03:31:59.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:59.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:59.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:59.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:59.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:59.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:31:59.447 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:59.247+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:59.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:59.058+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:59.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:59.058+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:59.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:59.058+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:31:59.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:59.059+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:59.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:31:59.558+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:59.559542+0000) 2024-04-27T03:31:59.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:59.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:59.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:59.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:59.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:59.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:31:59.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:59.517+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:31:59.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:31:59.525+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:59.526193+0000) 2024-04-27T03:32:00.026 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:31:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:31:59.762+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:31:59.762848+0000) 2024-04-27T03:32:00.290 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:00.049+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:00.291 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:00.049+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:00.291 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:00.049+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:00.291 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:00.049+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:00.566 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:00.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:00.567 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:00.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:00.567 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:00.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:00.567 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:00.285+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:00.567 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:00.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:00.567 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:00.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:00.567 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:00.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:00.567 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:00.493+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:00.567 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:00.529+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:00.530760+0000) 2024-04-27T03:32:00.975 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:32:00.587+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64949 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:32:00.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:00.584+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:00.585585+0000) 2024-04-27T03:32:00.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:00.560+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:00.561572+0000) 2024-04-27T03:32:00.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:00.583+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:00.584143+0000) 2024-04-27T03:32:00.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:00.583+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:00.584046+0000) 2024-04-27T03:32:00.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:00.762+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:00.763026+0000) 2024-04-27T03:32:01.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:01.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:01.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:01.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:01.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:01.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:01.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:01.094+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:01.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:01.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:01.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:01.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:01.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:01.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:01.536 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:01.463+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:01.536 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:01.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:01.536 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:01.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:01.536 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:01.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:01.536 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:01.289+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:01.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:01.561+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:01.561847+0000) 2024-04-27T03:32:01.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:01.530+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:01.530951+0000) 2024-04-27T03:32:01.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:01.766+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:01.767233+0000) 2024-04-27T03:32:02.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:02.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:02.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:02.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:02.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:02.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:02.328 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:02.125+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:02.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:02.565+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:02.566068+0000) 2024-04-27T03:32:02.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:02.321+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:02.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:02.321+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:02.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:02.321+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:02.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:02.322+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:02.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:02.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:02.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:02.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:02.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:02.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:02.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:02.430+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:02.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:02.534+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:02.535107+0000) 2024-04-27T03:32:03.073 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:02.769+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:02.770032+0000) 2024-04-27T03:32:03.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:03.140+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:03.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:03.140+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:03.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:03.140+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:03.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:03.140+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:03.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:03.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:03.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:03.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:03.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:03.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:03.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:03.277+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:03.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:03.567+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:03.568248+0000) 2024-04-27T03:32:03.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:03.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:03.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:03.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:03.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:03.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:03.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:03.465+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:03.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:03.538+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:03.539380+0000) 2024-04-27T03:32:04.073 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:03.771+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:03.772586+0000) 2024-04-27T03:32:04.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:04.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:04.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:04.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:04.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:04.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:04.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:04.144+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:04.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:04.287+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:04.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:04.287+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:04.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:04.287+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:04.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:04.287+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:04.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:04.571+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:04.572432+0000) 2024-04-27T03:32:04.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:04.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:04.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:04.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:04.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:04.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:04.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:04.421+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:04.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:04.540+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:04.541612+0000) 2024-04-27T03:32:05.074 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:04.773+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:04.774700+0000) 2024-04-27T03:32:05.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:05.103+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:05.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:05.103+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:05.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:05.103+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:05.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:05.104+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:05.579 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:05.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:05.579 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:05.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:05.579 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:05.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:05.579 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:05.426+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:05.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:05.542+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:05.543748+0000) 2024-04-27T03:32:05.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:05.321+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:05.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:05.321+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:05.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:05.321+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:05.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:05.321+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:05.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:32:05.595+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 64989 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:32:05.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:05.592+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:05.593482+0000) 2024-04-27T03:32:05.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:05.573+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:05.574441+0000) 2024-04-27T03:32:05.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:05.592+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:05.592813+0000) 2024-04-27T03:32:05.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:05.592+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:05.593765+0000) 2024-04-27T03:32:05.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:05.774+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:05.774933+0000) 2024-04-27T03:32:06.285 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:06.069+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:06.285 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:06.069+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:06.285 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:06.069+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:06.285 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:06.070+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:06.549 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:06.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:06.549 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:06.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:06.549 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:06.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:06.549 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:06.432+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:06.549 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:06.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:06.550 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:06.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:06.550 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:06.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:06.550 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:06.279+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:06.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:06.543+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:06.543973+0000) 2024-04-27T03:32:06.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:06.779+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:06.780034+0000) 2024-04-27T03:32:06.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:06.573+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:06.574666+0000) 2024-04-27T03:32:07.394 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:07.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:07.394 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:07.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:07.394 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:07.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:07.394 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:07.256+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:07.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:07.023+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:07.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:07.023+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:07.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:07.023+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:07.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:07.023+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:07.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:07.575+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:07.576793+0000) 2024-04-27T03:32:07.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:07.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:07.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:07.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:07.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:07.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:07.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:07.389+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:07.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:07.545+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:07.546351+0000) 2024-04-27T03:32:08.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:07.784+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:07.785020+0000) 2024-04-27T03:32:08.442 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:08.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:08.442 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:08.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:08.442 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:08.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:08.442 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:08.043+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:08.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:08.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:08.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:08.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:08.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:08.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:08.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:08.277+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:08.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:08.578+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:08.579060+0000) 2024-04-27T03:32:08.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:08.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:08.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:08.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:08.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:08.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:08.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:08.437+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:08.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:08.547+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:08.548530+0000) 2024-04-27T03:32:09.074 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:08.786+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:08.787759+0000) 2024-04-27T03:32:09.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:09.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:09.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:09.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:09.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:09.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:09.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:09.277+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:09.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:09.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:09.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:09.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:09.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:09.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:09.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:09.076+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:09.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:09.580+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:09.581195+0000) 2024-04-27T03:32:09.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:09.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:09.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:09.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:09.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:09.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:09.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:09.440+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:09.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:09.549+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:09.550704+0000) 2024-04-27T03:32:10.074 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:09.790+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:09.792545+0000) 2024-04-27T03:32:10.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:10.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:10.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:10.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:10.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:10.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:10.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:10.123+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:10.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:10.277+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:10.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:10.277+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:10.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:10.277+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:10.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:10.278+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:10.725 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:32:10.602+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65028 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:32:10.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:10.581+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:10.583295+0000) 2024-04-27T03:32:10.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:10.599+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:10.600824+0000) 2024-04-27T03:32:10.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:10.599+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:10.601558+0000) 2024-04-27T03:32:10.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:10.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:10.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:10.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:10.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:10.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:10.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:10.430+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:10.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:10.550+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:10.552695+0000) 2024-04-27T03:32:10.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:10.601+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:10.603400+0000) 2024-04-27T03:32:11.015 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:10.791+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:10.792837+0000) 2024-04-27T03:32:11.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:11.171+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:11.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:11.171+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:11.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:11.171+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:11.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:11.172+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:11.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:11.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:11.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:11.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:11.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:11.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:11.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:11.246+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:11.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:11.581+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:11.583501+0000) 2024-04-27T03:32:11.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:11.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:11.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:11.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:11.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:11.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:11.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:11.449+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:11.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:11.551+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:11.552922+0000) 2024-04-27T03:32:12.074 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:11.793+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:11.794940+0000) 2024-04-27T03:32:12.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:12.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:12.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:12.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:12.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:12.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:12.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:12.250+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:12.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:12.186+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:12.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:12.186+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:12.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:12.186+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:12.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:12.187+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:12.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:12.584+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:12.586565+0000) 2024-04-27T03:32:12.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:12.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:12.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:12.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:12.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:12.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:12.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:12.492+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:12.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:12.553+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:12.554948+0000) 2024-04-27T03:32:13.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:12.795+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:12.797622+0000) 2024-04-27T03:32:13.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:13.243+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:13.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:13.243+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:13.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:13.243+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:13.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:13.244+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:13.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:13.189+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:13.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:13.189+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:13.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:13.189+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:13.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:13.189+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:13.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:13.586+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:13.588711+0000) 2024-04-27T03:32:13.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:13.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:13.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:13.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:13.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:13.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:13.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:13.520+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:13.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:13.555+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:13.557102+0000) 2024-04-27T03:32:14.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:13.800+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:13.801891+0000) 2024-04-27T03:32:14.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:14.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:14.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:14.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:14.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:14.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:14.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:14.208+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:14.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:14.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:14.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:14.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:14.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:14.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:14.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:14.171+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:14.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:14.591+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:14.593049+0000) 2024-04-27T03:32:14.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:14.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:14.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:14.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:14.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:14.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:14.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:14.537+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:14.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:14.559+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:14.561327+0000) 2024-04-27T03:32:15.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:14.804+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:14.806665+0000) 2024-04-27T03:32:15.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:15.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:15.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:15.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:15.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:15.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:15.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:15.218+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:15.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:15.163+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:15.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:15.163+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:15.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:15.163+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:15.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:15.164+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:15.725 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:32:15.610+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65068 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:32:15.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:15.607+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:15.609115+0000) 2024-04-27T03:32:15.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:15.596+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:15.597997+0000) 2024-04-27T03:32:15.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:15.607+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:15.608950+0000) 2024-04-27T03:32:15.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:15.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:15.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:15.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:15.550+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:15.562+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:15.563911+0000) 2024-04-27T03:32:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:15.607+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:15.609149+0000) 2024-04-27T03:32:16.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:15.805+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:15.806941+0000) 2024-04-27T03:32:16.266 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:16.152+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:16.266 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:16.152+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:16.267 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:16.152+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:16.267 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:16.153+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:16.547 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:16.259+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:16.547 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:16.259+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:16.547 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:16.259+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:16.547 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:16.260+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:16.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:16.596+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:16.598268+0000) 2024-04-27T03:32:16.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:16.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:16.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:16.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:16.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:16.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:16.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:16.541+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:16.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:16.562+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:16.564076+0000) 2024-04-27T03:32:17.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:16.807+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:16.809270+0000) 2024-04-27T03:32:17.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:17.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:17.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:17.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:17.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:17.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:17.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:17.282+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:17.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:17.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:17.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:17.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:17.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:17.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:17.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:17.201+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:17.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:17.598+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:17.600531+0000) 2024-04-27T03:32:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:17.539+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:17.539+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:17.539+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:17.540+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:17.564+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:17.566120+0000) 2024-04-27T03:32:18.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:17.809+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:17.811740+0000) 2024-04-27T03:32:18.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:18.189+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:18.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:18.189+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:18.333 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:18.189+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:18.333 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:18.190+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:18.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:18.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:18.592 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:18.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:18.592 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:18.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:18.592 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:18.326+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:18.592 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:18.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:18.592 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:18.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:18.592 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:18.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:18.592 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:18.497+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:18.592 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:18.566+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:18.568250+0000) 2024-04-27T03:32:18.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:18.601+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:18.603404+0000) 2024-04-27T03:32:18.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:18.813+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:18.814957+0000) 2024-04-27T03:32:19.294 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:19.232+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:19.294 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:19.232+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:19.294 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:19.232+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:19.294 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:19.233+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:19.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:19.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:19.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:19.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:19.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:19.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:19.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:19.461+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:19.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:19.287+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:19.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:19.287+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:19.576 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:19.287+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:19.577 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:19.287+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:19.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:19.569+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:19.571217+0000) 2024-04-27T03:32:19.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:19.815+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:19.817588+0000) 2024-04-27T03:32:19.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:19.604+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:19.606579+0000) 2024-04-27T03:32:20.579 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:20.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:20.579 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:20.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:20.579 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:20.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:20.579 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:20.468+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:20.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:20.268+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:20.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:20.268+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:20.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:20.268+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:20.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:20.268+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:20.580 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:20.281+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:20.580 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:20.281+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:20.580 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:20.281+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:20.580 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:20.282+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:20.975 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:32:20.619+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65107 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:32:20.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:20.572+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:20.574050+0000) 2024-04-27T03:32:20.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:20.614+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:20.616674+0000) 2024-04-27T03:32:20.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:20.607+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:20.609307+0000) 2024-04-27T03:32:20.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:20.615+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:20.617695+0000) 2024-04-27T03:32:20.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:20.616+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:20.618517+0000) 2024-04-27T03:32:20.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:20.815+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:20.817790+0000) 2024-04-27T03:32:21.579 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:21.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:21.579 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:21.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:21.579 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:21.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:21.579 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:21.315+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:21.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:21.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:21.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:21.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:21.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:21.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:21.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:21.435+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:21.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:21.572+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:21.574299+0000) 2024-04-27T03:32:21.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:21.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:21.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:21.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:21.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:21.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:21.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:21.317+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:21.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:21.607+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:21.609480+0000) 2024-04-27T03:32:21.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:21.819+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:21.820998+0000) 2024-04-27T03:32:22.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:22.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:22.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:22.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:22.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:22.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:22.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:22.456+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:22.584 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:22.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:22.584 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:22.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:22.584 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:22.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:22.584 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:22.310+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:22.584 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:22.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:22.585 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:22.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:22.585 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:22.335+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:22.585 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:22.335+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:22.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:22.577+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:22.578517+0000) 2024-04-27T03:32:22.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:22.824+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:22.825319+0000) 2024-04-27T03:32:22.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:22.610+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:22.611786+0000) 2024-04-27T03:32:23.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:23.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:23.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:23.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:23.588 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:23.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:23.588 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:23.462+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:23.588 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:23.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:23.588 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:23.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:23.588 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:23.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:23.588 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:23.292+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:23.588 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:23.293+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:23.588 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:23.293+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:23.589 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:23.293+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:23.589 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:23.293+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:23.856 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:23.581+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:23.582605+0000) 2024-04-27T03:32:23.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:23.614+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:23.615013+0000) 2024-04-27T03:32:23.856 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:23.827+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:23.828115+0000) 2024-04-27T03:32:24.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:24.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:24.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:24.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:24.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:24.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:24.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:24.279+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:24.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:24.321+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:24.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:24.321+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:24.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:24.321+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:24.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:24.322+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:24.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:24.617+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:24.617958+0000) 2024-04-27T03:32:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:24.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:24.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:24.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:24.452+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:24.584+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:24.584942+0000) 2024-04-27T03:32:25.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:24.829+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:24.830286+0000) 2024-04-27T03:32:25.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:25.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:25.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:25.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:25.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:25.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:25.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:25.298+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:25.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:25.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:25.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:25.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:25.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:25.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:25.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:25.312+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:25.725 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:32:25.628+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65147 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:32:25.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:25.620+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:25.621067+0000) 2024-04-27T03:32:25.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:25.624+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:25.625363+0000) 2024-04-27T03:32:25.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:25.624+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:25.625278+0000) 2024-04-27T03:32:25.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:25.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:25.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:25.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:25.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:25.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:25.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:25.449+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:25.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:25.586+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:25.587576+0000) 2024-04-27T03:32:25.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:25.624+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:25.625385+0000) 2024-04-27T03:32:26.017 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:25.829+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:25.830509+0000) 2024-04-27T03:32:26.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:26.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:26.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:26.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:26.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:26.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:26.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:26.432+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:26.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:26.587+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:26.587834+0000) 2024-04-27T03:32:26.600 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:26.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:26.600 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:26.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:26.600 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:26.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:26.600 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:26.348+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:26.601 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:26.340+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:26.601 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:26.340+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:26.601 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:26.340+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:26.601 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:26.340+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:26.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:26.620+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:26.621306+0000) 2024-04-27T03:32:26.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:26.832+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:26.833200+0000) 2024-04-27T03:32:27.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:27.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:27.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:27.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:27.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:27.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:27.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:27.338+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:27.598 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:27.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:27.598 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:27.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:27.598 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:27.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:27.598 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:27.432+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:27.598 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:27.368+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:27.598 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:27.368+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:27.598 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:27.368+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:27.598 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:27.368+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:27.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:27.623+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:27.623843+0000) 2024-04-27T03:32:27.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:27.835+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:27.836167+0000) 2024-04-27T03:32:27.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:27.591+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:27.592220+0000) 2024-04-27T03:32:28.632 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:28.389+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:28.632 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:28.389+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:28.632 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:28.389+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:28.632 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:28.390+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:28.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:28.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:28.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:28.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:28.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:28.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:28.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:28.365+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:28.633 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:28.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:28.633 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:28.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:28.633 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:28.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:28.633 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:28.472+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:28.633 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:28.593+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:28.594667+0000) 2024-04-27T03:32:28.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:28.626+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:28.627072+0000) 2024-04-27T03:32:28.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:28.838+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:28.839200+0000) 2024-04-27T03:32:29.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:29.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:29.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:29.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:29.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:29.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:29.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:29.363+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:29.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:29.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:29.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:29.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:29.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:29.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:29.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:29.444+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:29.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:29.596+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:29.597413+0000) 2024-04-27T03:32:29.636 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:29.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:29.636 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:29.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:29.636 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:29.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:29.637 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:29.419+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:29.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:29.629+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:29.630255+0000) 2024-04-27T03:32:29.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:29.841+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:29.841899+0000) 2024-04-27T03:32:30.604 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:30.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:30.605 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:30.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:30.605 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:30.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:30.605 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:30.430+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:30.605 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:30.431+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:30.605 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:30.431+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:30.605 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:30.431+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:30.605 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:30.432+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:30.606 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:30.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:30.606 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:30.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:30.606 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:30.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:30.606 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:30.391+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:30.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:32:30.636+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65187 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:32:30.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:30.598+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:30.599644+0000) 2024-04-27T03:32:30.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:30.633+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:30.634136+0000) 2024-04-27T03:32:30.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:30.633+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:30.633860+0000) 2024-04-27T03:32:30.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:30.841+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:30.842100+0000) 2024-04-27T03:32:30.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:30.634+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:30.635237+0000) 2024-04-27T03:32:30.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:30.634+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:30.635261+0000) 2024-04-27T03:32:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:31.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:31.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:31.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:31.401+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:31.599+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:31.599822+0000) 2024-04-27T03:32:31.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:31.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:31.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:31.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:31.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:31.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:31.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:31.475+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:31.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:31.638+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:31.639059+0000) 2024-04-27T03:32:31.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:31.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:31.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:31.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:31.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:31.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:31.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:31.429+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:32.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:31.843+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:31.844279+0000) 2024-04-27T03:32:32.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:32.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:32.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:32.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:32.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:32.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:32.448 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:32.368+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:32.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:32.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:32.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:32.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:32.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:32.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:32.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:32.406+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:32.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:32.602+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:32.603726+0000) 2024-04-27T03:32:32.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:32.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:32.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:32.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:32.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:32.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:32.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:32.443+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:32.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:32.643+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:32.644042+0000) 2024-04-27T03:32:33.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:32.846+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:32.847394+0000) 2024-04-27T03:32:33.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:33.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:33.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:33.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:33.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:33.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:33.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:33.387+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:33.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:33.606+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:33.607009+0000) 2024-04-27T03:32:33.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:33.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:33.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:33.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:33.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:33.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:33.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:33.438+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:33.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:33.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:33.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:33.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:33.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:33.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:33.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:33.441+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:33.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:33.648+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:33.649199+0000) 2024-04-27T03:32:34.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:33.848+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:33.849737+0000) 2024-04-27T03:32:34.429 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:34.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:34.429 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:34.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:34.430 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:34.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:34.430 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:34.379+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:34.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:34.608+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:34.609264+0000) 2024-04-27T03:32:34.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:34.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:34.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:34.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:34.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:34.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:34.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:34.474+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:34.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:34.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:34.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:34.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:34.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:34.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:34.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:34.424+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:34.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:34.653+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:34.654218+0000) 2024-04-27T03:32:35.163 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:34.851+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:34.852034+0000) 2024-04-27T03:32:35.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:35.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:35.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:35.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:35.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:35.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:35.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:35.346+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:35.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:35.610+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:35.611562+0000) 2024-04-27T03:32:35.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:35.641+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:35.642178+0000) 2024-04-27T03:32:35.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:32:35.643+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65227 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:32:35.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:35.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:35.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:35.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:35.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:35.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:35.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:35.456+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:35.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:35.641+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:35.641983+0000) 2024-04-27T03:32:35.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:35.469+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:35.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:35.469+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:35.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:35.469+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:35.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:35.470+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:35.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:35.641+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:35.642672+0000) 2024-04-27T03:32:35.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:35.653+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:35.654492+0000) 2024-04-27T03:32:36.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:35.851+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:35.852308+0000) 2024-04-27T03:32:36.662 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:36.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:36.662 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:36.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:36.662 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:36.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:36.662 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:36.475+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:36.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:36.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:36.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:36.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:36.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:36.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:36.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:36.366+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:36.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:36.610+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:36.611777+0000) 2024-04-27T03:32:36.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:36.491+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:36.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:36.491+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:36.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:36.491+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:36.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:36.492+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:36.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:36.656+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:36.657341+0000) 2024-04-27T03:32:36.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:36.854+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:36.855199+0000) 2024-04-27T03:32:37.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:37.325+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:37.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:37.325+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:37.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:37.325+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:37.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:37.326+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:37.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:37.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:37.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:37.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:37.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:37.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:37.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:37.500+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:37.617 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:37.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:37.617 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:37.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:37.617 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:37.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:37.617 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:37.438+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:37.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:37.658+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:37.659714+0000) 2024-04-27T03:32:37.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:37.857+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:37.857887+0000) 2024-04-27T03:32:37.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:37.613+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:37.614034+0000) 2024-04-27T03:32:38.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:38.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:38.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:38.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:38.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:38.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:38.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:38.477+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:38.618 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:38.435+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:38.618 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:38.435+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:38.618 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:38.435+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:38.618 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:38.437+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:38.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:38.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:38.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:38.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:38.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:38.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:38.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:38.355+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:38.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:38.615+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:38.616260+0000) 2024-04-27T03:32:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:38.662+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:38.662978+0000) 2024-04-27T03:32:38.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:38.861+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:38.862132+0000) 2024-04-27T03:32:39.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:39.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:39.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:39.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:39.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:39.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:39.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:39.347+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:39.618 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:39.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:39.618 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:39.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:39.619 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:39.475+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:39.619 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:39.476+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:39.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:39.509+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:39.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:39.509+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:39.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:39.509+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:39.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:39.509+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:39.869 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:39.618+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:39.619226+0000) 2024-04-27T03:32:39.869 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:39.664+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:39.665026+0000) 2024-04-27T03:32:40.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:39.863+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:39.864554+0000) 2024-04-27T03:32:40.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:40.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:40.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:40.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:40.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:40.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:40.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:40.322+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:40.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:32:40.652+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65266 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:32:40.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:40.620+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:40.621370+0000) 2024-04-27T03:32:40.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:40.648+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:40.649745+0000) 2024-04-27T03:32:40.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:40.479+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:40.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:40.479+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:40.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:40.479+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:40.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:40.480+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:40.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:40.648+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:40.649325+0000) 2024-04-27T03:32:40.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:40.451+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:40.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:40.451+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:40.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:40.451+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:40.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:40.452+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:40.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:40.648+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:40.649009+0000) 2024-04-27T03:32:40.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:40.664+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:40.665242+0000) 2024-04-27T03:32:41.078 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:40.864+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:40.864817+0000) 2024-04-27T03:32:41.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:41.360+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:41.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:41.360+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:41.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:41.360+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:41.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:41.362+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:41.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:41.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:41.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:41.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:41.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:41.480+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:41.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:41.480+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:41.619 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:41.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:41.619 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:41.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:41.619 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:41.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:41.619 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:41.410+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:41.874 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:41.669+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:41.670311+0000) 2024-04-27T03:32:41.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:41.620+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:41.621562+0000) 2024-04-27T03:32:42.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:41.868+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:41.869510+0000) 2024-04-27T03:32:42.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:42.374+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:42.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:42.374+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:42.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:42.374+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:42.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:42.375+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:42.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:42.398+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:42.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:42.398+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:42.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:42.398+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:42.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:42.399+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:42.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:42.674+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:42.675388+0000) 2024-04-27T03:32:42.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:42.622+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:42.623634+0000) 2024-04-27T03:32:42.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:42.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:42.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:42.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:42.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:42.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:42.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:42.437+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:43.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:42.873+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:42.874092+0000) 2024-04-27T03:32:43.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:43.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:43.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:43.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:43.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:43.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:43.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:43.406+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:43.440 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:43.384+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:43.440 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:43.384+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:43.440 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:43.384+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:43.440 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:43.385+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:43.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:43.679+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:43.680500+0000) 2024-04-27T03:32:43.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:43.625+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:43.625837+0000) 2024-04-27T03:32:43.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:43.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:43.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:43.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:43.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:43.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:43.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:43.434+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:44.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:43.875+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:43.876387+0000) 2024-04-27T03:32:44.451 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:44.389+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:44.452 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:44.389+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:44.452 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:44.389+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:44.452 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:44.390+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:44.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:44.684+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:44.685589+0000) 2024-04-27T03:32:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:44.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:44.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:44.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:44.453+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:44.629+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:44.630065+0000) 2024-04-27T03:32:44.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:44.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:44.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:44.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:44.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:44.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:44.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:44.452+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:45.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:44.877+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:44.878613+0000) 2024-04-27T03:32:45.434 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:45.345+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:45.434 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:45.345+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:45.434 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:45.345+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:45.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:45.347+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:45.685 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:32:45.660+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65306 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:32:45.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:45.656+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:45.657462+0000) 2024-04-27T03:32:45.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:45.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:45.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:45.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:45.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:45.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:45.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:45.465+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:45.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:45.631+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:45.632152+0000) 2024-04-27T03:32:45.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:45.657+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:45.658486+0000) 2024-04-27T03:32:45.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:45.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:45.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:45.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:45.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:45.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:45.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:45.429+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:45.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:45.657+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:45.658123+0000) 2024-04-27T03:32:45.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:45.685+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:45.685839+0000) 2024-04-27T03:32:45.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:45.878+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:45.878815+0000) 2024-04-27T03:32:46.624 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:46.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:46.625 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:46.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:46.625 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:46.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:46.625 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:46.468+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:46.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:46.386+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:46.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:46.386+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:46.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:46.386+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:46.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:46.388+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:46.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:46.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:46.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:46.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:46.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:46.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:46.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:46.437+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:46.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:46.631+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:46.632355+0000) 2024-04-27T03:32:46.886 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:46.687+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:46.687958+0000) 2024-04-27T03:32:47.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:46.880+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:46.881014+0000) 2024-04-27T03:32:47.451 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:47.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:47.451 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:47.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:47.452 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:47.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:47.452 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:47.365+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:47.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:47.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:47.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:47.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:47.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:47.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:47.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:47.392+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:47.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:47.689+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:47.690058+0000) 2024-04-27T03:32:47.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:47.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:47.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:47.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:47.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:47.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:47.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:47.447+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:47.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:47.633+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:47.634485+0000) 2024-04-27T03:32:48.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:47.882+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:47.883359+0000) 2024-04-27T03:32:48.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:48.377+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:48.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:48.377+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:48.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:48.377+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:48.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:48.378+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:48.452 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:48.340+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:48.452 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:48.340+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:48.453 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:48.340+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:48.453 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:48.340+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:48.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:48.693+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:48.694434+0000) 2024-04-27T03:32:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:48.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:48.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:48.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:48.496+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:48.635+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:48.636660+0000) 2024-04-27T03:32:49.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:48.887+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:48.888150+0000) 2024-04-27T03:32:49.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:49.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:49.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:49.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:49.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:49.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:49.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:49.365+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:49.452 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:49.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:49.452 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:49.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:49.453 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:49.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:49.453 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:49.292+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:49.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:49.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:49.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:49.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:49.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:49.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:49.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:49.482+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:49.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:49.640+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:49.640895+0000) 2024-04-27T03:32:49.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:49.698+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:49.698815+0000) 2024-04-27T03:32:49.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:49.892+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:49.892931+0000) 2024-04-27T03:32:50.582 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:50.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:50.582 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:50.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:50.582 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:50.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:50.582 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:50.408+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:50.583 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:50.326+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:50.583 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:50.326+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:50.583 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:50.326+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:50.583 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:50.326+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:50.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:50.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:50.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:50.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:50.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:50.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:50.583 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:50.518+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:50.898 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:32:50.668+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65345 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:32:50.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:50.665+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:50.666146+0000) 2024-04-27T03:32:50.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:50.645+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:50.646026+0000) 2024-04-27T03:32:50.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:50.666+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:50.667098+0000) 2024-04-27T03:32:50.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:50.665+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:50.666577+0000) 2024-04-27T03:32:50.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:50.698+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:50.699013+0000) 2024-04-27T03:32:51.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:50.892+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:50.893222+0000) 2024-04-27T03:32:51.452 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:51.300+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:51.452 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:51.300+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:51.453 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:51.300+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:51.453 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:51.301+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:51.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:51.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:51.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:51.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:51.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:51.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:51.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:51.393+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:51.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:51.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:51.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:51.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:51.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:51.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:51.707 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:51.502+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:51.707 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:51.645+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:51.646317+0000) 2024-04-27T03:32:51.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:51.894+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:51.895757+0000) 2024-04-27T03:32:51.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:51.700+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:51.701225+0000) 2024-04-27T03:32:52.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:52.252+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:52.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:52.252+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:52.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:52.252+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:52.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:52.253+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:52.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:52.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:52.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:52.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:52.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:52.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:52.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:52.515+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:52.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:52.648+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:52.648908+0000) 2024-04-27T03:32:52.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:52.421+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:52.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:52.421+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:52.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:52.421+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:52.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:52.421+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:52.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:52.897+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:52.897799+0000) 2024-04-27T03:32:52.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:52.704+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:52.705705+0000) 2024-04-27T03:32:53.411 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:53.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:53.411 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:53.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:53.411 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:53.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:53.411 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:53.219+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:53.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:53.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:53.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:53.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:53.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:53.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:53.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:53.405+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:53.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:53.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:53.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:53.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:53.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:53.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:53.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:53.525+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:53.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:53.651+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:53.652069+0000) 2024-04-27T03:32:53.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:53.899+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:53.900596+0000) 2024-04-27T03:32:53.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:53.708+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:53.709205+0000) 2024-04-27T03:32:54.438 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:54.214+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:54.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:54.214+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:54.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:54.214+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:54.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:54.215+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:54.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:54.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:54.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:54.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:54.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:54.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:54.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:54.548+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:54.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:54.655+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:54.656305+0000) 2024-04-27T03:32:54.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:54.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:54.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:54.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:54.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:54.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:54.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:54.433+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:54.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:54.903+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:54.903822+0000) 2024-04-27T03:32:54.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:54.710+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:54.711627+0000) 2024-04-27T03:32:55.452 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:55.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:55.453 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:55.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:55.453 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:55.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:55.453 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:55.216+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:55.703 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:32:55.679+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65385 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:32:55.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:55.457+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:55.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:55.457+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:55.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:55.457+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:55.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:55.458+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:55.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:55.674+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:55.674981+0000) 2024-04-27T03:32:55.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:55.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:55.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:55.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:55.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:55.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:55.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:55.586+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:55.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:55.657+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:55.658608+0000) 2024-04-27T03:32:55.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:55.672+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:55.672917+0000) 2024-04-27T03:32:55.704 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:55.672+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:55.673617+0000) 2024-04-27T03:32:55.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:55.711+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:55.711901+0000) 2024-04-27T03:32:55.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:55.903+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:55.904050+0000) 2024-04-27T03:32:56.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:56.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:56.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:56.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:56.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:56.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:56.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:56.189+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:56.628 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:56.437+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:56.628 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:56.437+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:56.628 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:56.437+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:56.628 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:56.438+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:56.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:56.713+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:56.713960+0000) 2024-04-27T03:32:56.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:56.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:56.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:56.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:56.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:56.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:56.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:56.624+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:56.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:56.658+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:56.658846+0000) 2024-04-27T03:32:57.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:56.906+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:56.907243+0000) 2024-04-27T03:32:57.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:57.221+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:57.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:57.221+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:57.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:57.221+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:57.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:57.222+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:57.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:57.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:57.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:57.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:57.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:57.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:57.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:57.444+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:57.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:57.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:57.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:57.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:57.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:57.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:57.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:57.580+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:57.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:57.660+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:57.660898+0000) 2024-04-27T03:32:57.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:57.715+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:57.716055+0000) 2024-04-27T03:32:57.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:57.908+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:57.909724+0000) 2024-04-27T03:32:58.547 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:58.267+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:58.548 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:58.267+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:58.548 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:58.267+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:58.548 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:58.268+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:58.548 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:58.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:58.548 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:58.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:58.548 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:58.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:58.548 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:58.419+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:58.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:58.717+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:58.718707+0000) 2024-04-27T03:32:58.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:58.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:58.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:58.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:58.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:58.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:58.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:58.543+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:58.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:58.664+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:58.665738+0000) 2024-04-27T03:32:59.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:58.911+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:58.912075+0000) 2024-04-27T03:32:59.517 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:59.265+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:59.518 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:59.265+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:59.518 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:59.265+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:32:59.518 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:59.265+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:59.518 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:59.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:59.518 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:59.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:59.518 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:59.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:32:59.518 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:59.430+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:59.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:32:59.722+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:59.723044+0000) 2024-04-27T03:32:59.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:59.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:59.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:59.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:59.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:59.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:32:59.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:59.513+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:32:59.922 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:32:59.667+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:59.667840+0000) 2024-04-27T03:33:00.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:32:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:32:59.916+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:32:59.916839+0000) 2024-04-27T03:33:00.487 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:00.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:00.487 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:00.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:00.487 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:00.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:00.488 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:00.450+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:00.488 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:00.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:00.488 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:00.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:00.488 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:00.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:00.488 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:00.313+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:00.814 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:33:00.686+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65426 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:33:00.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:00.683+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:00.684517+0000) 2024-04-27T03:33:00.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:00.722+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:00.723366+0000) 2024-04-27T03:33:00.815 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:00.684+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:00.684918+0000) 2024-04-27T03:33:00.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:00.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:00.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:00.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:00.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:00.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:00.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:00.482+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:00.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:00.669+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:00.669987+0000) 2024-04-27T03:33:00.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:00.684+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:00.685087+0000) 2024-04-27T03:33:01.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:00.916+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:00.917162+0000) 2024-04-27T03:33:01.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:01.283+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:01.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:01.283+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:01.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:01.283+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:01.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:01.284+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:01.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:01.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:01.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:01.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:01.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:01.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:01.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:01.507+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:01.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:01.669+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:01.670209+0000) 2024-04-27T03:33:01.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:01.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:01.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:01.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:01.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:01.438+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:01.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:01.439+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:02.021 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:01.921+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:01.921929+0000) 2024-04-27T03:33:02.021 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:01.725+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:01.726294+0000) 2024-04-27T03:33:02.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:02.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:02.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:02.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:02.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:02.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:02.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:02.486+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:02.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:02.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:02.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:02.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:02.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:02.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:02.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:02.409+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:02.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:02.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:02.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:02.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:02.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:02.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:02.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:02.308+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:02.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:02.671+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:02.672430+0000) 2024-04-27T03:33:02.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:02.728+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:02.728994+0000) 2024-04-27T03:33:03.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:02.923+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:02.924182+0000) 2024-04-27T03:33:03.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:03.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:03.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:03.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:03.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:03.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:03.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:03.460+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:03.648 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:03.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:03.648 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:03.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:03.648 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:03.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:03.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:03.394+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:03.649 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:03.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:03.649 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:03.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:03.649 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:03.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:03.649 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:03.347+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:03.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:03.673+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:03.674711+0000) 2024-04-27T03:33:03.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:03.731+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:03.732137+0000) 2024-04-27T03:33:04.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:03.927+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:03.928255+0000) 2024-04-27T03:33:04.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:04.390+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:04.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:04.390+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:04.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:04.390+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:04.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:04.391+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:04.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:04.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:04.649 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:04.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:04.649 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:04.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:04.649 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:04.451+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:04.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:04.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:04.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:04.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:04.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:04.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:04.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:04.410+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:04.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:04.733+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:04.734235+0000) 2024-04-27T03:33:04.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:04.678+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:04.678988+0000) 2024-04-27T03:33:05.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:04.929+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:04.930546+0000) 2024-04-27T03:33:05.489 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:05.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:05.489 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:05.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:05.489 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:05.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:05.489 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:05.382+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:05.489 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:05.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:05.489 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:05.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:05.490 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:05.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:05.490 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:05.426+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:05.739 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:33:05.694+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65466 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:33:05.739 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:05.692+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:05.693074+0000) 2024-04-27T03:33:05.740 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:05.690+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:05.691372+0000) 2024-04-27T03:33:05.740 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:05.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:05.740 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:05.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:05.740 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:05.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:05.740 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:05.483+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:05.740 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:05.682+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:05.683199+0000) 2024-04-27T03:33:05.740 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:05.692+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:05.693203+0000) 2024-04-27T03:33:06.022 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:05.733+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:05.734449+0000) 2024-04-27T03:33:06.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:05.930+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:05.930826+0000) 2024-04-27T03:33:06.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:06.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:06.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:06.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:06.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:06.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:06.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:06.475+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:06.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:06.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:06.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:06.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:06.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:06.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:06.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:06.406+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:06.649 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:06.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:06.649 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:06.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:06.649 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:06.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:06.649 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:06.466+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:06.939 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:06.682+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:06.683459+0000) 2024-04-27T03:33:06.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:06.736+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:06.737533+0000) 2024-04-27T03:33:07.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:06.933+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:06.934129+0000) 2024-04-27T03:33:07.651 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:07.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:07.651 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:07.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:07.651 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:07.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:07.651 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:07.453+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:07.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:07.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:07.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:07.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:07.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:07.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:07.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:07.385+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:07.652 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:07.424+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:07.652 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:07.424+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:07.652 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:07.424+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:07.652 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:07.425+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:07.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:07.684+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:07.685694+0000) 2024-04-27T03:33:07.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:07.738+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:07.739785+0000) 2024-04-27T03:33:08.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:07.937+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:07.938213+0000) 2024-04-27T03:33:08.486 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:08.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:08.487 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:08.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:08.487 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:08.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:08.487 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:08.464+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:08.487 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:08.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:08.487 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:08.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:08.487 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:08.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:08.487 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:08.338+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:08.747 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:08.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:08.747 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:08.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:08.747 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:08.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:08.747 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:08.482+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:08.747 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:08.687+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:08.687915+0000) 2024-04-27T03:33:09.022 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:08.741+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:08.741894+0000) 2024-04-27T03:33:09.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:08.939+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:08.940730+0000) 2024-04-27T03:33:09.653 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:09.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:09.653 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:09.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:09.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:09.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:09.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:09.464+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:09.654 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:09.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:09.655 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:09.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:09.655 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:09.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:09.655 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:09.473+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:09.655 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:09.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:09.655 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:09.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:09.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:09.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:09.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:09.329+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:09.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:09.689+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:09.690199+0000) 2024-04-27T03:33:09.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:09.743+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:09.744694+0000) 2024-04-27T03:33:10.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:09.942+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:09.943152+0000) 2024-04-27T03:33:10.653 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:10.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:10.653 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:10.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:10.653 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:10.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:10.653 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:10.282+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:10.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:10.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:10.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:10.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:10.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:10.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:10.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:10.456+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:10.654 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:10.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:10.654 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:10.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:10.654 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:10.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:10.654 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:10.458+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:10.948 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:33:10.701+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65505 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:33:10.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:10.698+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:10.699473+0000) 2024-04-27T03:33:10.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:10.693+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:10.694459+0000) 2024-04-27T03:33:10.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:10.697+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:10.698430+0000) 2024-04-27T03:33:10.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:10.698+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:10.698913+0000) 2024-04-27T03:33:10.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:10.744+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:10.744931+0000) 2024-04-27T03:33:11.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:10.942+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:10.943407+0000) 2024-04-27T03:33:11.500 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:11.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:11.501 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:11.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:11.501 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:11.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:11.501 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:11.286+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:11.501 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:11.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:11.501 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:11.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:11.501 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:11.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:11.501 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:11.461+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:11.752 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:11.692+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:11.694746+0000) 2024-04-27T03:33:11.752 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:11.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:11.753 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:11.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:11.753 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:11.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:11.753 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:11.496+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:12.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:11.943+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:11.945785+0000) 2024-04-27T03:33:12.022 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:11.745+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:11.747579+0000) 2024-04-27T03:33:12.655 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:12.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:12.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:12.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:12.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:12.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:12.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:12.427+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:12.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:12.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:12.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:12.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:12.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:12.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:12.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:12.282+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:12.657 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:12.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:12.657 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:12.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:12.657 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:12.458+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:12.657 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:12.458+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:12.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:12.695+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:12.696973+0000) 2024-04-27T03:33:12.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:12.750+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:12.751800+0000) 2024-04-27T03:33:13.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:12.947+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:12.949037+0000) 2024-04-27T03:33:13.657 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:13.421+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:13.657 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:13.421+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:13.657 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:13.421+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:13.657 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:13.421+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:13.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:13.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:13.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:13.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:13.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:13.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:13.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:13.263+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:13.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:13.471+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:13.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:13.471+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:13.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:13.471+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:13.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:13.472+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:13.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:13.697+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:13.699274+0000) 2024-04-27T03:33:13.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:13.752+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:13.753926+0000) 2024-04-27T03:33:14.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:13.949+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:13.951251+0000) 2024-04-27T03:33:14.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:14.217+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:14.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:14.217+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:14.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:14.217+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:14.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:14.218+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:14.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:14.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:14.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:14.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:14.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:14.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:14.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:14.379+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:14.761 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:14.699+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:14.701566+0000) 2024-04-27T03:33:14.761 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:14.477+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:14.761 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:14.477+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:14.761 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:14.477+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:14.761 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:14.477+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:15.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:14.754+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:14.756075+0000) 2024-04-27T03:33:15.012 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:14.952+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:14.953823+0000) 2024-04-27T03:33:15.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:15.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:15.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:15.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:15.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:15.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:15.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:15.232+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:15.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:15.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:15.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:15.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:15.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:15.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:15.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:15.406+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:15.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:15.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:15.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:15.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:15.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:15.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:15.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:15.456+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:15.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:33:15.708+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65545 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:33:15.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:15.705+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:15.707041+0000) 2024-04-27T03:33:15.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:15.705+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:15.707357+0000) 2024-04-27T03:33:15.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:15.705+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:15.706931+0000) 2024-04-27T03:33:15.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:15.754+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:15.756482+0000) 2024-04-27T03:33:15.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:15.705+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:15.707158+0000) 2024-04-27T03:33:16.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:15.952+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:15.954216+0000) 2024-04-27T03:33:16.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:16.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:16.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:16.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:16.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:16.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:16.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:16.443+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:16.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:16.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:16.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:16.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:16.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:16.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:16.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:16.400+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:16.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:16.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:16.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:16.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:16.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:16.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:16.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:16.241+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:16.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:16.708+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:16.709904+0000) 2024-04-27T03:33:16.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:16.759+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:16.761098+0000) 2024-04-27T03:33:17.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:16.957+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:16.958824+0000) 2024-04-27T03:33:17.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:17.192+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:17.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:17.192+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:17.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:17.192+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:17.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:17.193+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:17.660 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:17.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:17.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:17.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:17.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:17.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:17.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:17.407+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:17.661 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:17.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:17.661 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:17.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:17.661 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:17.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:17.661 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:17.403+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:17.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:17.710+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:17.712090+0000) 2024-04-27T03:33:17.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:17.763+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:17.765237+0000) 2024-04-27T03:33:18.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:17.959+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:17.961691+0000) 2024-04-27T03:33:18.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:18.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:18.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:18.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:18.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:18.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:18.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:18.181+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:18.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:18.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:18.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:18.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:18.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:18.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:18.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:18.419+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:18.663 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:18.422+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:18.663 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:18.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:18.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:18.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:18.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:18.424+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:18.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:18.712+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:18.714373+0000) 2024-04-27T03:33:18.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:18.765+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:18.767478+0000) 2024-04-27T03:33:19.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:18.962+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:18.964546+0000) 2024-04-27T03:33:19.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:19.185+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:19.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:19.185+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:19.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:19.185+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:19.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:19.186+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:19.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:19.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:19.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:19.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:19.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:19.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:19.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:19.419+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:19.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:19.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:19.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:19.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:19.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:19.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:19.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:19.472+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:19.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:19.714+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:19.716694+0000) 2024-04-27T03:33:19.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:19.767+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:19.769761+0000) 2024-04-27T03:33:20.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:19.967+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:19.968885+0000) 2024-04-27T03:33:20.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:20.181+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:20.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:20.181+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:20.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:20.181+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:20.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:20.182+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:20.489 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:20.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:20.489 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:20.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:20.489 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:20.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:20.489 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:20.456+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:20.743 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:33:20.717+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65584 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:33:20.744 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:20.714+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:20.716136+0000) 2024-04-27T03:33:20.744 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:20.715+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:20.717005+0000) 2024-04-27T03:33:20.744 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:20.714+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:20.716377+0000) 2024-04-27T03:33:20.745 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:20.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:20.745 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:20.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:20.745 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:20.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:20.745 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:20.483+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:20.745 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:20.712+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:20.714498+0000) 2024-04-27T03:33:21.023 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:20.768+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:20.770085+0000) 2024-04-27T03:33:21.023 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:20.967+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:20.969288+0000) 2024-04-27T03:33:21.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:21.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:21.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:21.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:21.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:21.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:21.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:21.172+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:21.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:21.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:21.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:21.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:21.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:21.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:21.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:21.448+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:21.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:21.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:21.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:21.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:21.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:21.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:21.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:21.474+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:21.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:21.771+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:21.773120+0000) 2024-04-27T03:33:21.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:21.718+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:21.719936+0000) 2024-04-27T03:33:22.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:21.969+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:21.971463+0000) 2024-04-27T03:33:22.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:22.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:22.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:22.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:22.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:22.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:22.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:22.169+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:22.509 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:22.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:22.509 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:22.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:22.509 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:22.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:22.509 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:22.444+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:22.780 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:22.721+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:22.723693+0000) 2024-04-27T03:33:22.780 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:22.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:22.780 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:22.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:22.780 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:22.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:22.780 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:22.503+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:22.781 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:22.773+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:22.775250+0000) 2024-04-27T03:33:23.083 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:22.971+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:22.973709+0000) 2024-04-27T03:33:23.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:23.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:23.456 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:23.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:23.456 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:23.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:23.456 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:23.163+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:23.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:23.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:23.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:23.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:23.466+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:23.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:23.453+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:23.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:23.453+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:23.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:23.453+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:23.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:23.454+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:24.023 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:23.976+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:23.976951+0000) 2024-04-27T03:33:24.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:23.727+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:23.728006+0000) 2024-04-27T03:33:24.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:23.778+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:23.779661+0000) 2024-04-27T03:33:24.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:24.197+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:24.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:24.197+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:24.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:24.197+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:24.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:24.198+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:24.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:24.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:24.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:24.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:24.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:24.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:24.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:24.439+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:24.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:24.408+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:24.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:24.408+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:24.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:24.408+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:24.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:24.409+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:24.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:24.730+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:24.731112+0000) 2024-04-27T03:33:24.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:24.782+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:24.782945+0000) 2024-04-27T03:33:25.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:24.978+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:24.979649+0000) 2024-04-27T03:33:25.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:25.224+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:25.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:25.224+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:25.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:25.224+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:25.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:25.225+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:25.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:25.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:25.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:25.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:25.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:25.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:25.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:25.488+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:25.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:25.389+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:25.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:25.389+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:25.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:25.389+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:25.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:25.390+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:25.973 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:33:25.727+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65624 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:33:25.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:25.723+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:25.724619+0000) 2024-04-27T03:33:25.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:25.730+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:25.731566+0000) 2024-04-27T03:33:25.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:25.723+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:25.724420+0000) 2024-04-27T03:33:25.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:25.724+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:25.725420+0000) 2024-04-27T03:33:25.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:25.782+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:25.783607+0000) 2024-04-27T03:33:26.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:25.979+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:25.980062+0000) 2024-04-27T03:33:26.532 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:26.394+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:26.532 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:26.394+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:26.532 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:26.394+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:26.532 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:26.395+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:26.532 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:26.242+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:26.532 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:26.242+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:26.533 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:26.242+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:26.533 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:26.242+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:26.790 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:26.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:26.791 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:26.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:26.791 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:26.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:26.791 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:26.527+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:26.791 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:26.734+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:26.735104+0000) 2024-04-27T03:33:27.083 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:26.982+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:26.983515+0000) 2024-04-27T03:33:27.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:26.785+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:26.785823+0000) 2024-04-27T03:33:27.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:27.288+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:27.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:27.288+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:27.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:27.288+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:27.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:27.288+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:27.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:27.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:27.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:27.508+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:27.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:27.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:27.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:27.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:27.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:27.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:27.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:27.441+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:28.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:27.736+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:27.737308+0000) 2024-04-27T03:33:28.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:27.787+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:27.787981+0000) 2024-04-27T03:33:28.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:27.984+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:27.985664+0000) 2024-04-27T03:33:28.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:28.275+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:28.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:28.275+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:28.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:28.275+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:28.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:28.276+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:28.543 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:28.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:28.544 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:28.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:28.544 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:28.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:28.544 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:28.396+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:28.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:28.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:28.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:28.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:28.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:28.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:28.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:28.538+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:28.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:28.739+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:28.740292+0000) 2024-04-27T03:33:29.083 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:28.989+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:28.990430+0000) 2024-04-27T03:33:29.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:28.792+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:28.792812+0000) 2024-04-27T03:33:29.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:29.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:29.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:29.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:29.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:29.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:29.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:29.280+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:29.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:29.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:29.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:29.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:29.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:29.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:29.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:29.526+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:29.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:29.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:29.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:29.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:29.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:29.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:29.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:29.422+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:29.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:29.741+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:29.742389+0000) 2024-04-27T03:33:29.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:29.794+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:29.795652+0000) 2024-04-27T03:33:30.314 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:29.992+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:29.993402+0000) 2024-04-27T03:33:30.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:30.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:30.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:30.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:30.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:30.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:30.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:30.530+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:30.680 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:30.318+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:30.680 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:30.318+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:30.680 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:30.318+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:30.680 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:30.319+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:30.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:30.376+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:30.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:30.376+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:30.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:30.376+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:30.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:30.377+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:30.975 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:33:30.735+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65662 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:33:30.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:30.732+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:30.733284+0000) 2024-04-27T03:33:30.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:30.741+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:30.742707+0000) 2024-04-27T03:33:30.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:30.732+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:30.733376+0000) 2024-04-27T03:33:30.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:30.732+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:30.733369+0000) 2024-04-27T03:33:30.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:30.795+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:30.795932+0000) 2024-04-27T03:33:31.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:30.992+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:30.993752+0000) 2024-04-27T03:33:31.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:31.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:31.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:31.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:31.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:31.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:31.682 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:31.331+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:31.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:31.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:31.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:31.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:31.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:31.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:31.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:31.547+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:31.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:31.419+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:31.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:31.419+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:31.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:31.419+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:31.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:31.420+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:31.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:31.744+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:31.745572+0000) 2024-04-27T03:33:31.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:31.799+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:31.800307+0000) 2024-04-27T03:33:32.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:31.995+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:31.996480+0000) 2024-04-27T03:33:32.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:32.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:32.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:32.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:32.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:32.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:32.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:32.297+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:32.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:32.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:32.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:32.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:32.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:32.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:32.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:32.513+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:32.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:32.424+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:32.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:32.424+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:32.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:32.424+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:32.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:32.425+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:32.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:32.747+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:32.747819+0000) 2024-04-27T03:33:32.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:32.801+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:32.802460+0000) 2024-04-27T03:33:33.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:33.000+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:33.001202+0000) 2024-04-27T03:33:33.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:33.322+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:33.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:33.322+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:33.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:33.322+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:33.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:33.322+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:33.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:33.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:33.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:33.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:33.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:33.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:33.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:33.472+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:33.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:33.430+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:33.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:33.430+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:33.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:33.430+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:33.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:33.431+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:33.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:33.749+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:33.750091+0000) 2024-04-27T03:33:33.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:33.806+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:33.806835+0000) 2024-04-27T03:33:34.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:34.004+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:34.005506+0000) 2024-04-27T03:33:34.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:34.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:34.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:34.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:34.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:34.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:34.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:34.502+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:34.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:34.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:34.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:34.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:34.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:34.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:34.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:34.331+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:34.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:34.416+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:34.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:34.416+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:34.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:34.416+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:34.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:34.417+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:34.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:34.751+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:34.752182+0000) 2024-04-27T03:33:34.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:34.808+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:34.808914+0000) 2024-04-27T03:33:35.361 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:35.007+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:35.007856+0000) 2024-04-27T03:33:35.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:35.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:35.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:35.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:35.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:35.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:35.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:35.535+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:35.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:35.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:35.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:35.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:35.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:35.425+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:35.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:35.426+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:35.688 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:35.355+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:35.688 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:35.355+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:35.688 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:35.355+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:35.688 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:35.356+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:35.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:35.740+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:35.741151+0000) 2024-04-27T03:33:35.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:33:35.743+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65703 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:33:35.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:35.740+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:35.741609+0000) 2024-04-27T03:33:35.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:35.751+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:35.752548+0000) 2024-04-27T03:33:35.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:35.739+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:35.740546+0000) 2024-04-27T03:33:35.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:35.808+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:35.809183+0000) 2024-04-27T03:33:36.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:36.007+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:36.008164+0000) 2024-04-27T03:33:36.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:36.362+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:36.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:36.362+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:36.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:36.362+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:36.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:36.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:36.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:36.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:36.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:36.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:36.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:36.553+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:36.688 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:36.363+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:36.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:36.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:36.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:36.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:36.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:36.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:36.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:36.458+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:36.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:36.754+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:36.755306+0000) 2024-04-27T03:33:36.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:36.810+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:36.811436+0000) 2024-04-27T03:33:37.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:37.009+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:37.010263+0000) 2024-04-27T03:33:37.606 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:37.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:37.606 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:37.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:37.606 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:37.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:37.606 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:37.445+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:37.606 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:37.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:37.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:37.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:37.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:37.319+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:37.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:37.320+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:37.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:37.813+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:37.814393+0000) 2024-04-27T03:33:37.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:37.600+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:37.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:37.600+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:37.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:37.600+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:37.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:37.601+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:37.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:37.757+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:37.758208+0000) 2024-04-27T03:33:38.313 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:38.012+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:38.013233+0000) 2024-04-27T03:33:38.313 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:38.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:38.313 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:38.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:38.313 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:38.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:38.566 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:38.308+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:38.566 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:38.441+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:38.566 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:38.441+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:38.566 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:38.441+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:38.567 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:38.442+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:38.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:38.560+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:38.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:38.560+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:38.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:38.560+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:38.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:38.561+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:38.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:38.759+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:38.760445+0000) 2024-04-27T03:33:39.085 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:39.015+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:39.016227+0000) 2024-04-27T03:33:39.085 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:38.816+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:38.817330+0000) 2024-04-27T03:33:39.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:39.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:39.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:39.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:39.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:39.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:39.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:39.308+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:39.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:39.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:39.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:39.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:39.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:39.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:39.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:39.592+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:39.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:39.477+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:39.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:39.478+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:39.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:39.478+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:39.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:39.478+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:40.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:39.762+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:39.763338+0000) 2024-04-27T03:33:40.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:39.818+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:39.819776+0000) 2024-04-27T03:33:40.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:40.018+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:40.019249+0000) 2024-04-27T03:33:40.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:40.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:40.578 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:40.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:40.578 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:40.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:40.578 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:40.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:40.578 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:40.525+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:40.579 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:40.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:40.579 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:40.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:40.579 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:40.287+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:40.876 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:33:40.752+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65741 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:33:40.877 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:40.748+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:40.749170+0000) 2024-04-27T03:33:40.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:40.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:40.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:40.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:40.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:40.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:40.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:40.573+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:40.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:40.748+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:40.749129+0000) 2024-04-27T03:33:40.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:40.763+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:40.763827+0000) 2024-04-27T03:33:40.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:40.748+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:40.749520+0000) 2024-04-27T03:33:40.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:40.819+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:40.820257+0000) 2024-04-27T03:33:41.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:41.018+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:41.019651+0000) 2024-04-27T03:33:41.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:41.271+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:41.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:41.271+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:41.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:41.271+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:41.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:41.271+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:41.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:41.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:41.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:41.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:41.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:41.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:41.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:41.596+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:41.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:41.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:41.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:41.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:41.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:41.540+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:41.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:41.541+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:42.025 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:41.821+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:41.822315+0000) 2024-04-27T03:33:42.026 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:41.767+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:41.768045+0000) 2024-04-27T03:33:42.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:42.023+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:42.024745+0000) 2024-04-27T03:33:42.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:42.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:42.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:42.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:42.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:42.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:42.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:42.255+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:42.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:42.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:42.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:42.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:42.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:42.543+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:42.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:42.543+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:42.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:42.596+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:42.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:42.596+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:42.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:42.596+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:42.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:42.597+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:43.026 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:42.769+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:42.770721+0000) 2024-04-27T03:33:43.026 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:42.824+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:42.825043+0000) 2024-04-27T03:33:43.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:43.028+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:43.028984+0000) 2024-04-27T03:33:43.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:43.290+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:43.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:43.290+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:43.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:43.290+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:43.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:43.291+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:43.643 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:43.529+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:43.643 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:43.529+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:43.643 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:43.529+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:43.643 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:43.530+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:43.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:43.829+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:43.830167+0000) 2024-04-27T03:33:43.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:43.637+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:43.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:43.637+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:43.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:43.637+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:43.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:43.638+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:43.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:43.774+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:43.775683+0000) 2024-04-27T03:33:44.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:44.030+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:44.031250+0000) 2024-04-27T03:33:44.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:44.242+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:44.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:44.242+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:44.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:44.242+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:44.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:44.243+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:44.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:44.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:44.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:44.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:44.660+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:44.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:44.493+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:44.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:44.493+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:44.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:44.493+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:44.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:44.494+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:45.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:44.776+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:44.777749+0000) 2024-04-27T03:33:45.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:44.834+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:44.835213+0000) 2024-04-27T03:33:45.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:45.032+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:45.033349+0000) 2024-04-27T03:33:45.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:45.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:45.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:45.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:45.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:45.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:45.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:45.239+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:45.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:45.642+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:45.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:45.642+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:45.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:45.642+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:45.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:45.643+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:45.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:45.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:45.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:45.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:45.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:45.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:45.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:45.525+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:46.026 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:45.758+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:45.759339+0000) 2024-04-27T03:33:46.026 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:33:45.760+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65782 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:33:46.026 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:45.756+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:45.757246+0000) 2024-04-27T03:33:46.027 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:45.777+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:45.778178+0000) 2024-04-27T03:33:46.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:45.757+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:45.758025+0000) 2024-04-27T03:33:46.027 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:45.834+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:45.835610+0000) 2024-04-27T03:33:46.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:46.032+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:46.033783+0000) 2024-04-27T03:33:46.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:46.263+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:46.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:46.263+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:46.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:46.263+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:46.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:46.264+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:46.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:46.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:46.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:46.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:46.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:46.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:46.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:46.491+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:46.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:46.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:46.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:46.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:46.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:46.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:46.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:46.678+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:46.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:46.781+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:46.782388+0000) 2024-04-27T03:33:46.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:46.837+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:46.838388+0000) 2024-04-27T03:33:47.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:47.036+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:47.037498+0000) 2024-04-27T03:33:47.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:47.273+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:47.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:47.273+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:47.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:47.273+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:47.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:47.274+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:47.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:47.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:47.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:47.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:47.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:47.491+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:47.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:47.492+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:47.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:47.840+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:47.841153+0000) 2024-04-27T03:33:47.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:47.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:47.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:47.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:47.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:47.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:47.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:47.687+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:47.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:47.784+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:47.785313+0000) 2024-04-27T03:33:48.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:48.040+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:48.040845+0000) 2024-04-27T03:33:48.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:48.316+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:48.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:48.316+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:48.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:48.316+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:48.669 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:48.317+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:48.669 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:48.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:48.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:48.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:48.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:48.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:48.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:48.519+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:48.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:48.842+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:48.843233+0000) 2024-04-27T03:33:48.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:48.663+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:48.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:48.663+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:48.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:48.663+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:48.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:48.665+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:48.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:48.788+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:48.789278+0000) 2024-04-27T03:33:49.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:49.042+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:49.043116+0000) 2024-04-27T03:33:49.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:49.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:49.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:49.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:49.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:49.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:49.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:49.335+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:49.660 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:49.479+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:49.660 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:49.479+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:49.660 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:49.479+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:49.660 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:49.480+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:49.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:49.845+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:49.846253+0000) 2024-04-27T03:33:49.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:49.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:49.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:49.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:49.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:49.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:49.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:49.655+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:49.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:49.790+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:49.791438+0000) 2024-04-27T03:33:50.315 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:50.044+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:50.045414+0000) 2024-04-27T03:33:50.623 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:50.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:50.623 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:50.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:50.624 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:50.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:50.624 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:50.467+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:50.624 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:50.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:50.624 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:50.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:50.624 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:50.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:50.624 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:50.353+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:50.894 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:33:50.768+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65820 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:33:50.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:50.765+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:50.766771+0000) 2024-04-27T03:33:50.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:50.764+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:50.765526+0000) 2024-04-27T03:33:50.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:50.845+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:50.846667+0000) 2024-04-27T03:33:50.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:50.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:50.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:50.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:50.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:50.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:50.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:50.618+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:50.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:50.765+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:50.766449+0000) 2024-04-27T03:33:50.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:50.790+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:50.791750+0000) 2024-04-27T03:33:51.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:51.045+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:51.045845+0000) 2024-04-27T03:33:51.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:51.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:51.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:51.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:51.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:51.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:51.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:51.353+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:51.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:51.663+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:51.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:51.663+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:51.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:51.663+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:51.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:51.664+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:51.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:51.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:51.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:51.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:51.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:51.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:51.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:51.462+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:52.053 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:51.794+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:51.795010+0000) 2024-04-27T03:33:52.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:51.848+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:51.849408+0000) 2024-04-27T03:33:52.364 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:52.047+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:52.048090+0000) 2024-04-27T03:33:52.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:52.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:52.669 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:52.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:52.669 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:52.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:52.669 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:52.359+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:52.669 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:52.459+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:52.669 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:52.459+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:52.669 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:52.459+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:52.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:52.460+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:52.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:52.850+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:52.851537+0000) 2024-04-27T03:33:52.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:52.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:52.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:52.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:52.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:52.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:52.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:52.664+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:52.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:52.796+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:52.797098+0000) 2024-04-27T03:33:53.346 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:53.051+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:53.052220+0000) 2024-04-27T03:33:53.347 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:53.340+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:53.347 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:53.340+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:53.347 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:53.340+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:53.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:53.341+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:53.707 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:53.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:53.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:53.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:53.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:53.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:53.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:53.502+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:53.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:53.855+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:53.856580+0000) 2024-04-27T03:33:53.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:53.708+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:53.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:53.708+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:53.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:53.708+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:53.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:53.709+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:53.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:53.801+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:53.802244+0000) 2024-04-27T03:33:54.305 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:54.053+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:54.054375+0000) 2024-04-27T03:33:54.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:54.300+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:54.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:54.300+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:54.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:54.300+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:54.682 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:54.300+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:54.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:54.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:54.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:54.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:54.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:54.531+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:54.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:54.531+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:54.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:54.860+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:54.860956+0000) 2024-04-27T03:33:54.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:54.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:54.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:54.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:54.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:54.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:54.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:54.678+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:54.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:54.803+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:54.804580+0000) 2024-04-27T03:33:55.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:55.057+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:55.058679+0000) 2024-04-27T03:33:55.524 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:55.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:55.524 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:55.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:55.525 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:55.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:55.525 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:55.287+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:55.775 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:55.715+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:55.776 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:55.715+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:55.776 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:55.715+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:55.776 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:55.717+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:55.776 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:55.518+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:55.776 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:55.518+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:55.776 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:55.518+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:55.776 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:55.520+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:56.027 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:33:55.776+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65861 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:33:56.027 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:55.773+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:55.774392+0000) 2024-04-27T03:33:56.028 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:55.804+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:55.804909+0000) 2024-04-27T03:33:56.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:55.772+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:55.772893+0000) 2024-04-27T03:33:56.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:55.860+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:55.861374+0000) 2024-04-27T03:33:56.028 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:55.773+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:55.773868+0000) 2024-04-27T03:33:56.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:56.058+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:56.059172+0000) 2024-04-27T03:33:56.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:56.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:56.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:56.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:56.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:56.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:56.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:56.249+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:56.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:56.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:56.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:56.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:56.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:56.533+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:56.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:56.534+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:56.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:56.864+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:56.865704+0000) 2024-04-27T03:33:56.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:56.677+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:56.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:56.677+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:56.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:56.677+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:56.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:56.679+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:56.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:56.808+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:56.809121+0000) 2024-04-27T03:33:57.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:57.062+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:57.063422+0000) 2024-04-27T03:33:57.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:57.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:57.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:57.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:57.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:57.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:57.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:57.219+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:57.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:57.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:57.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:57.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:57.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:57.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:57.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:57.551+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:57.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:57.869+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:57.869997+0000) 2024-04-27T03:33:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:57.716+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:57.716+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:57.716+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:57.716+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:57.810+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:57.811288+0000) 2024-04-27T03:33:58.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:58.064+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:58.065533+0000) 2024-04-27T03:33:58.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:58.189+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:58.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:58.189+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:58.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:58.189+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:58.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:58.189+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:58.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:58.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:58.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:58.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:58.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:58.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:58.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:58.523+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:58.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:58.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:58.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:58.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:58.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:58.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:58.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:58.668+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:58.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:58.813+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:58.814251+0000) 2024-04-27T03:33:58.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:58.871+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:58.872075+0000) 2024-04-27T03:33:59.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:59.069+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:59.069819+0000) 2024-04-27T03:33:59.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:59.229+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:59.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:59.229+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:59.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:59.229+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:33:59.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:33:59.230+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:59.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:59.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:59.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:59.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:59.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:59.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:33:59.712 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:59.527+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:59.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:59.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:59.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:59.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:59.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:59.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:33:59.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:59.625+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:33:59.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:33:59.815+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:59.816377+0000) 2024-04-27T03:33:59.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:33:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:33:59.873+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:33:59.874426+0000) 2024-04-27T03:34:00.316 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:00.072+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:00.073153+0000) 2024-04-27T03:34:00.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:00.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:00.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:00.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:00.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:00.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:00.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:00.265+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:00.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:00.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:00.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:00.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:00.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:00.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:00.659 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:00.483+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:00.910 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:34:00.784+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65899 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:34:00.910 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:00.780+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:00.781716+0000) 2024-04-27T03:34:00.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:00.781+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:00.782750+0000) 2024-04-27T03:34:00.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:00.873+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:00.874760+0000) 2024-04-27T03:34:00.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:00.652+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:00.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:00.652+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:00.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:00.652+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:00.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:00.654+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:00.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:00.781+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:00.782214+0000) 2024-04-27T03:34:00.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:00.815+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:00.816724+0000) 2024-04-27T03:34:01.173 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:01.072+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:01.073789+0000) 2024-04-27T03:34:01.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:01.290+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:01.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:01.290+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:01.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:01.290+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:01.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:01.290+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:01.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:01.657+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:01.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:01.657+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:01.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:01.657+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:01.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:01.658+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:01.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:01.459+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:01.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:01.459+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:01.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:01.459+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:01.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:01.460+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:01.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:01.819+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:01.819910+0000) 2024-04-27T03:34:01.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:01.876+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:01.877535+0000) 2024-04-27T03:34:02.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:02.075+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:02.075881+0000) 2024-04-27T03:34:02.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:02.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:02.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:02.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:02.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:02.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:02.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:02.270+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:02.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:02.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:02.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:02.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:02.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:02.698+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:02.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:02.698+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:02.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:02.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:02.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:02.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:02.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:02.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:02.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:02.468+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:02.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:02.822+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:02.823033+0000) 2024-04-27T03:34:02.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:02.878+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:02.879658+0000) 2024-04-27T03:34:03.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:03.079+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:03.080159+0000) 2024-04-27T03:34:03.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:03.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:03.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:03.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:03.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:03.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:03.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:03.221+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:03.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:03.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:03.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:03.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:03.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:03.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:03.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:03.483+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:03.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:03.882+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:03.882849+0000) 2024-04-27T03:34:03.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:03.710+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:03.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:03.710+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:03.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:03.710+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:03.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:03.711+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:03.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:03.826+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:03.827277+0000) 2024-04-27T03:34:04.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:04.082+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:04.083364+0000) 2024-04-27T03:34:04.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:04.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:04.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:04.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:04.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:04.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:04.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:04.172+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:04.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:04.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:04.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:04.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:04.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:04.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:04.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:04.437+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:04.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:04.885+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:04.886075+0000) 2024-04-27T03:34:04.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:04.713+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:04.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:04.713+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:04.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:04.713+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:04.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:04.714+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:04.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:04.829+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:04.830284+0000) 2024-04-27T03:34:05.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:05.086+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:05.087365+0000) 2024-04-27T03:34:05.408 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:05.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:05.408 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:05.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:05.408 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:05.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:05.408 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:05.141+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:05.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:05.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:05.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:05.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:05.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:05.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:05.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:05.403+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:05.975 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:34:05.792+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65940 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:34:05.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:05.789+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:05.789925+0000) 2024-04-27T03:34:05.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:05.788+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:05.789722+0000) 2024-04-27T03:34:05.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:05.885+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:05.886576+0000) 2024-04-27T03:34:05.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:05.691+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:05.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:05.691+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:05.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:05.691+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:05.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:05.692+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:05.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:05.788+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:05.789430+0000) 2024-04-27T03:34:05.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:05.830+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:05.830908+0000) 2024-04-27T03:34:06.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:06.086+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:06.087727+0000) 2024-04-27T03:34:06.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:06.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:06.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:06.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:06.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:06.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:06.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:06.180+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:06.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:06.394+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:06.675 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:06.394+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:06.675 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:06.394+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:06.675 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:06.395+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:06.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:06.889+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:06.889977+0000) 2024-04-27T03:34:06.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:06.668+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:06.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:06.669+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:06.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:06.669+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:06.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:06.670+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:06.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:06.832+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:06.833166+0000) 2024-04-27T03:34:07.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:07.089+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:07.089861+0000) 2024-04-27T03:34:07.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:07.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:07.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:07.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:07.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:07.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:07.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:07.215+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:07.637 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:07.376+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:07.638 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:07.376+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:07.638 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:07.376+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:07.638 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:07.378+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:07.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:07.631+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:07.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:07.631+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:07.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:07.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:07.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:07.632+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:07.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:07.834+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:07.835249+0000) 2024-04-27T03:34:08.174 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:08.093+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:08.094060+0000) 2024-04-27T03:34:08.174 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:07.893+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:07.894300+0000) 2024-04-27T03:34:08.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:08.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:08.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:08.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:08.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:08.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:08.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:08.207+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:08.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:08.394+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:08.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:08.394+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:08.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:08.394+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:08.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:08.396+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:08.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:08.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:08.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:08.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:08.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:08.586+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:08.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:08.587+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:08.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:08.895+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:08.896672+0000) 2024-04-27T03:34:08.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:08.836+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:08.837525+0000) 2024-04-27T03:34:09.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:09.095+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:09.096429+0000) 2024-04-27T03:34:09.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:09.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:09.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:09.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:09.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:09.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:09.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:09.171+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:09.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:09.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:09.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:09.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:09.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:09.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:09.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:09.576+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:09.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:09.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:09.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:09.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:09.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:09.409+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:09.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:09.410+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:09.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:09.839+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:09.840597+0000) 2024-04-27T03:34:09.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:09.898+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:09.899402+0000) 2024-04-27T03:34:10.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:10.098+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:10.099672+0000) 2024-04-27T03:34:10.308 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:10.176+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:10.308 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:10.176+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:10.308 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:10.176+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:10.308 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:10.177+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:10.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:10.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:10.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:10.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:10.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:10.448+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:10.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:10.449+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:10.905 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:34:10.800+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 65978 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:34:10.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:10.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:10.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:10.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:10.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:10.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:10.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:10.625+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:10.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:10.797+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:10.797889+0000) 2024-04-27T03:34:10.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:10.840+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:10.841094+0000) 2024-04-27T03:34:10.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:10.797+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:10.798305+0000) 2024-04-27T03:34:10.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:10.796+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:10.797314+0000) 2024-04-27T03:34:11.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:11.099+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:11.100060+0000) 2024-04-27T03:34:11.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:11.165+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:11.171 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:10.899+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:10.899904+0000) 2024-04-27T03:34:11.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:11.165+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:11.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:11.165+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:11.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:11.165+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:11.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:11.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:11.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:11.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:11.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:11.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:11.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:11.659+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:11.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:11.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:11.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:11.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:11.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:11.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:11.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:11.462+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:11.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:11.842+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:11.843787+0000) 2024-04-27T03:34:11.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:11.901+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:11.902475+0000) 2024-04-27T03:34:12.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:12.102+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:12.103121+0000) 2024-04-27T03:34:12.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:12.212+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:12.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:12.212+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:12.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:12.212+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:12.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:12.213+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:12.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:12.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:12.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:12.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:12.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:12.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:12.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:12.684+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:12.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:12.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:12.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:12.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:12.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:12.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:12.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:12.486+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:13.029 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:12.846+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:12.847024+0000) 2024-04-27T03:34:13.029 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:12.903+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:12.904708+0000) 2024-04-27T03:34:13.379 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:13.106+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:13.107367+0000) 2024-04-27T03:34:13.379 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:13.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:13.379 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:13.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:13.379 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:13.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:13.379 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:13.180+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:13.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:13.515+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:13.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:13.515+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:13.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:13.515+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:13.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:13.516+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:13.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:13.905+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:13.907419+0000) 2024-04-27T03:34:13.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:13.693+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:13.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:13.693+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:13.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:13.693+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:13.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:13.693+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:13.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:13.847+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:13.849167+0000) 2024-04-27T03:34:14.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:14.107+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:14.109557+0000) 2024-04-27T03:34:14.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:14.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:14.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:14.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:14.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:14.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:14.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:14.204+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:14.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:14.516+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:14.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:14.516+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:14.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:14.516+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:14.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:14.516+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:15.029 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:14.910+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:14.911795+0000) 2024-04-27T03:34:15.029 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:14.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:15.029 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:14.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:15.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:14.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:15.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:14.732+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:15.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:14.850+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:14.851819+0000) 2024-04-27T03:34:15.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:15.110+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:15.111952+0000) 2024-04-27T03:34:15.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:15.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:15.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:15.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:15.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:15.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:15.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:15.205+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:15.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:15.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:15.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:15.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:15.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:15.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:15.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:15.471+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:16.030 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:34:15.807+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66019 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:34:16.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:15.804+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:15.806283+0000) 2024-04-27T03:34:16.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:15.805+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:15.807113+0000) 2024-04-27T03:34:16.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:15.910+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:15.912436+0000) 2024-04-27T03:34:16.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:15.750+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:16.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:15.750+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:16.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:15.750+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:16.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:15.751+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:16.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:15.803+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:15.805329+0000) 2024-04-27T03:34:16.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:15.850+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:15.852188+0000) 2024-04-27T03:34:16.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:16.111+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:16.112828+0000) 2024-04-27T03:34:16.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:16.247+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:16.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:16.247+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:16.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:16.247+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:16.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:16.247+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:16.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:16.477+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:16.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:16.477+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:16.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:16.477+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:16.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:16.478+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:17.027 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:16.753+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:17.028 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:16.753+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:17.028 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:16.753+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:17.028 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:16.754+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:17.028 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:16.852+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:16.854448+0000) 2024-04-27T03:34:17.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:16.913+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:16.915299+0000) 2024-04-27T03:34:17.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:17.114+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:17.116008+0000) 2024-04-27T03:34:17.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:17.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:17.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:17.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:17.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:17.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:17.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:17.250+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:17.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:17.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:17.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:17.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:17.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:17.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:17.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:17.456+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:18.029 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:17.915+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:17.917410+0000) 2024-04-27T03:34:18.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:17.734+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:18.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:17.734+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:18.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:17.734+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:18.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:17.735+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:18.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:17.854+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:17.856517+0000) 2024-04-27T03:34:18.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:18.116+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:18.118145+0000) 2024-04-27T03:34:18.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:18.243+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:18.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:18.243+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:18.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:18.243+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:18.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:18.244+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:18.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:18.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:18.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:18.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:18.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:18.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:18.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:18.500+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:18.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:18.919+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:18.921780+0000) 2024-04-27T03:34:18.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:18.717+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:18.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:18.717+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:18.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:18.717+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:18.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:18.717+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:18.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:18.856+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:18.858673+0000) 2024-04-27T03:34:19.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:19.119+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:19.120823+0000) 2024-04-27T03:34:19.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:19.226+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:19.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:19.226+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:19.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:19.226+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:19.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:19.226+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:19.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:19.471+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:19.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:19.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:19.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:19.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:19.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:19.472+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:19.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:19.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:19.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:19.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:19.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:19.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:19.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:19.678+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:20.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:19.861+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:19.862891+0000) 2024-04-27T03:34:20.030 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:19.922+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:19.924507+0000) 2024-04-27T03:34:20.318 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:20.123+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:20.125635+0000) 2024-04-27T03:34:20.318 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:20.237+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:20.318 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:20.237+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:20.318 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:20.237+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:20.318 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:20.238+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:20.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:20.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:20.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:20.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:20.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:20.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:20.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:20.492+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:20.975 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:34:20.815+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66057 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:34:20.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:20.812+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:20.813995+0000) 2024-04-27T03:34:20.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:20.923+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:20.924827+0000) 2024-04-27T03:34:20.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:20.812+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:20.813801+0000) 2024-04-27T03:34:20.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:20.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:20.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:20.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:20.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:20.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:20.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:20.708+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:20.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:20.812+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:20.813892+0000) 2024-04-27T03:34:20.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:20.861+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:20.863250+0000) 2024-04-27T03:34:21.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:21.124+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:21.126043+0000) 2024-04-27T03:34:21.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:21.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:21.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:21.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:21.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:21.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:21.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:21.269+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:21.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:21.507+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:21.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:21.507+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:21.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:21.507+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:21.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:21.508+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:22.030 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:21.928+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:21.929824+0000) 2024-04-27T03:34:22.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:21.741+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:22.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:21.741+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:22.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:21.741+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:22.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:21.742+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:22.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:21.863+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:21.865533+0000) 2024-04-27T03:34:22.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:22.129+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:22.131003+0000) 2024-04-27T03:34:22.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:22.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:22.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:22.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:22.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:22.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:22.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:22.225+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:22.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:22.703+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:22.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:22.703+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:22.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:22.703+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:22.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:22.704+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:22.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:22.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:22.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:22.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:22.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:22.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:22.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:22.546+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:23.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:22.866+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:22.867825+0000) 2024-04-27T03:34:23.030 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:22.932+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:22.934609+0000) 2024-04-27T03:34:23.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:23.134+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:23.136064+0000) 2024-04-27T03:34:23.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:23.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:23.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:23.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:23.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:23.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:23.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:23.262+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:23.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:23.503+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:23.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:23.503+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:23.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:23.503+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:23.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:23.504+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:23.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:23.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:23.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:23.675+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:24.030 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:23.937+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:23.939742+0000) 2024-04-27T03:34:24.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:23.869+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:23.871039+0000) 2024-04-27T03:34:24.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:24.139+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:24.141182+0000) 2024-04-27T03:34:24.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:24.217+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:24.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:24.217+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:24.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:24.217+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:24.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:24.218+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:24.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:24.541+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:24.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:24.541+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:24.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:24.541+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:24.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:24.542+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:24.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:24.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:24.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:24.635+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:25.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:24.871+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:24.873199+0000) 2024-04-27T03:34:25.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:24.943+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:24.944824+0000) 2024-04-27T03:34:25.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:25.144+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:25.146316+0000) 2024-04-27T03:34:25.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:25.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:25.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:25.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:25.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:25.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:25.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:25.225+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:25.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:25.561+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:25.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:25.561+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:25.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:25.561+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:25.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:25.562+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:25.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:25.602+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:25.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:25.602+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:25.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:25.602+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:25.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:25.602+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:26.030 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:34:25.827+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66098 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:34:26.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:25.822+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:25.823402+0000) 2024-04-27T03:34:26.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:25.872+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:25.873701+0000) 2024-04-27T03:34:26.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:25.820+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:25.821353+0000) 2024-04-27T03:34:26.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:25.821+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:25.822786+0000) 2024-04-27T03:34:26.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:25.944+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:25.945538+0000) 2024-04-27T03:34:26.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:26.145+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:26.146710+0000) 2024-04-27T03:34:26.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:26.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:26.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:26.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:26.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:26.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:26.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:26.185+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:26.566+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:26.566+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:26.566+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:26.567+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:26.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:26.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:26.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:26.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:26.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:26.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:26.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:26.576+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:27.030 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:26.875+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:26.876415+0000) 2024-04-27T03:34:27.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:26.947+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:26.948419+0000) 2024-04-27T03:34:27.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:27.148+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:27.148938+0000) 2024-04-27T03:34:27.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:27.234+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:27.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:27.234+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:27.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:27.234+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:27.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:27.235+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:27.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:27.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:27.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:27.536+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:27.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:27.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:27.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:27.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:27.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:27.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:27.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:27.578+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:28.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:27.879+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:27.880638+0000) 2024-04-27T03:34:28.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:27.949+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:27.950572+0000) 2024-04-27T03:34:28.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:28.153+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:28.153813+0000) 2024-04-27T03:34:28.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:28.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:28.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:28.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:28.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:28.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:28.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:28.263+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:28.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:28.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:28.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:28.501+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:28.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:28.536+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:28.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:28.536+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:28.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:28.536+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:28.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:28.537+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:29.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:28.881+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:28.882749+0000) 2024-04-27T03:34:29.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:28.951+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:28.952725+0000) 2024-04-27T03:34:29.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:29.155+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:29.155952+0000) 2024-04-27T03:34:29.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:29.275+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:29.558 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:29.275+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:29.558 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:29.275+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:29.558 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:29.276+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:29.559 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:29.498+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:29.559 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:29.498+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:29.559 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:29.498+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:29.559 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:29.499+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:29.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:29.552+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:29.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:29.552+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:29.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:29.552+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:29.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:29.553+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:30.166 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:29.954+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:29.954980+0000) 2024-04-27T03:34:30.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:29.884+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:29.885044+0000) 2024-04-27T03:34:30.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:30.160+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:30.161048+0000) 2024-04-27T03:34:30.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:30.250+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:30.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:30.250+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:30.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:30.250+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:30.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:30.251+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:30.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:30.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:30.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:30.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:30.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:30.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:30.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:30.560+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:30.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:30.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:30.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:30.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:30.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:30.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:30.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:30.464+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:31.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:30.831+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:30.832584+0000) 2024-04-27T03:34:31.031 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:34:30.835+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66136 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:34:31.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:30.832+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:30.833407+0000) 2024-04-27T03:34:31.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:30.884+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:30.885617+0000) 2024-04-27T03:34:31.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:30.832+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:30.833100+0000) 2024-04-27T03:34:31.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:30.954+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:30.955512+0000) 2024-04-27T03:34:31.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:31.161+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:31.161826+0000) 2024-04-27T03:34:31.449 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:31.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:31.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:31.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:31.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:31.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:31.450 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:31.240+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:31.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:31.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:31.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:31.445+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:31.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:31.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:31.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:31.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:31.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:31.578+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:31.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:31.579+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:32.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:31.887+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:31.888318+0000) 2024-04-27T03:34:32.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:31.957+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:31.958126+0000) 2024-04-27T03:34:32.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:32.163+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:32.164629+0000) 2024-04-27T03:34:32.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:32.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:32.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:32.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:32.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:32.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:32.455 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:32.249+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:32.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:32.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:32.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:32.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:32.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:32.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:32.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:32.449+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:32.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:32.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:32.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:32.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:32.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:32.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:32.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:32.587+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:33.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:32.889+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:32.890470+0000) 2024-04-27T03:34:33.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:32.961+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:32.962474+0000) 2024-04-27T03:34:33.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:33.166+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:33.166870+0000) 2024-04-27T03:34:33.553 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:33.416+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:33.553 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:33.416+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:33.553 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:33.416+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:33.553 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:33.417+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:33.553 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:33.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:33.553 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:33.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:33.553 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:33.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:33.553 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:33.277+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:33.864 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:33.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:33.864 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:33.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:33.864 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:33.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:33.864 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:33.547+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:34.175 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:33.964+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:33.964849+0000) 2024-04-27T03:34:34.175 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:33.892+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:33.892920+0000) 2024-04-27T03:34:34.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:34.169+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:34.170102+0000) 2024-04-27T03:34:34.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:34.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:34.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:34.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:34.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:34.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:34.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:34.240+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:34.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:34.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:34.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:34.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:34.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:34.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:34.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:34.463+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:34.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:34.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:34.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:34.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:34.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:34.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:34.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:34.555+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:35.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:34.895+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:34.896121+0000) 2024-04-27T03:34:35.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:34.967+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:34.968150+0000) 2024-04-27T03:34:35.285 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:35.172+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:35.173343+0000) 2024-04-27T03:34:35.285 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:35.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:35.285 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:35.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:35.285 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:35.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:35.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:35.279+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:35.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:35.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:35.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:35.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:35.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:35.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:35.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:35.431+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:35.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:35.538+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:35.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:35.538+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:35.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:35.538+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:35.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:35.539+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:36.177 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:34:35.844+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66177 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:34:36.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:35.840+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:35.840834+0000) 2024-04-27T03:34:36.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:35.895+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:35.896731+0000) 2024-04-27T03:34:36.178 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:35.841+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:35.841912+0000) 2024-04-27T03:34:36.178 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:35.967+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:35.968539+0000) 2024-04-27T03:34:36.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:35.840+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:35.841321+0000) 2024-04-27T03:34:36.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:36.173+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:36.173971+0000) 2024-04-27T03:34:36.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:36.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:36.435 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:36.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:36.435 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:36.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:36.435 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:36.295+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:36.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:36.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:36.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:36.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:36.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:36.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:36.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:36.429+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:36.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:36.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:36.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:36.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:36.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:36.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:36.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:36.556+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:37.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:36.898+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:36.899420+0000) 2024-04-27T03:34:37.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:36.970+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:36.971677+0000) 2024-04-27T03:34:37.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:37.175+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:37.176039+0000) 2024-04-27T03:34:37.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:37.267+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:37.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:37.267+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:37.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:37.267+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:37.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:37.268+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:37.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:37.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:37.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:37.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:37.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:37.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:37.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:37.603+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:37.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:37.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:37.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:37.398+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:37.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:37.398+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:37.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:37.399+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:38.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:37.900+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:37.901717+0000) 2024-04-27T03:34:38.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:37.975+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:37.975979+0000) 2024-04-27T03:34:38.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:38.177+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:38.178738+0000) 2024-04-27T03:34:38.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:38.233+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:38.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:38.233+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:38.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:38.233+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:38.446 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:38.234+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:38.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:38.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:38.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:38.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:38.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:38.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:38.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:38.440+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:38.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:38.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:38.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:38.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:38.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:38.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:38.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:38.628+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:39.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:38.903+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:38.903895+0000) 2024-04-27T03:34:39.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:38.977+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:38.978674+0000) 2024-04-27T03:34:39.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:39.182+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:39.182985+0000) 2024-04-27T03:34:39.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:39.270+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:39.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:39.270+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:39.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:39.270+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:39.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:39.271+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:39.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:39.608+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:39.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:39.608+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:39.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:39.608+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:39.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:39.609+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:39.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:39.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:39.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:39.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:39.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:39.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:39.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:39.452+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:40.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:39.907+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:39.908160+0000) 2024-04-27T03:34:40.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:39.982+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:39.982978+0000) 2024-04-27T03:34:40.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:40.184+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:40.185399+0000) 2024-04-27T03:34:40.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:40.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:40.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:40.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:40.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:40.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:40.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:40.270+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:40.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:40.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:40.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:40.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:40.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:40.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:40.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:40.435+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:40.877 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:34:40.852+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66215 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:34:40.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:40.848+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:40.849569+0000) 2024-04-27T03:34:40.878 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:40.850+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:40.851151+0000) 2024-04-27T03:34:40.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:40.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:40.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:40.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:40.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:40.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:40.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:40.615+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:40.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:40.848+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:40.849561+0000) 2024-04-27T03:34:41.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:40.982+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:40.983544+0000) 2024-04-27T03:34:41.178 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:40.907+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:40.908546+0000) 2024-04-27T03:34:41.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:41.393+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:41.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:41.393+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:41.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:41.393+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:41.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:41.395+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:41.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:41.185+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:41.185841+0000) 2024-04-27T03:34:41.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:41.290+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:41.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:41.290+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:41.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:41.290+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:41.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:41.290+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:41.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:41.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:41.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:41.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:41.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:41.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:41.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:41.607+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:41.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:41.912+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:41.913519+0000) 2024-04-27T03:34:42.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:42.190+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:42.190903+0000) 2024-04-27T03:34:42.261 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:41.984+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:41.985749+0000) 2024-04-27T03:34:42.633 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:42.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:42.633 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:42.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:42.633 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:42.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:42.633 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:42.445+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:42.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:42.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:42.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:42.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:42.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:42.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:42.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:42.255+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:42.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:42.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:42.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:42.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:42.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:42.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:42.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:42.628+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:43.178 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:42.914+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:42.915721+0000) 2024-04-27T03:34:43.178 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:42.989+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:42.990093+0000) 2024-04-27T03:34:43.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:43.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:43.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:43.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:43.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:43.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:43.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:43.407+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:43.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:43.192+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:43.192999+0000) 2024-04-27T03:34:43.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:43.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:43.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:43.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:43.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:43.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:43.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:43.206+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:43.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:43.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:43.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:43.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:43.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:43.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:43.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:43.678+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:44.033 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:43.918+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:43.919019+0000) 2024-04-27T03:34:44.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:43.991+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:43.992288+0000) 2024-04-27T03:34:44.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:44.194+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:44.195760+0000) 2024-04-27T03:34:44.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:44.227+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:44.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:44.227+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:44.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:44.227+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:44.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:44.228+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:44.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:44.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:44.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:44.449+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:44.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:44.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:44.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:44.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:44.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:44.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:44.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:44.663+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:45.033 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:44.920+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:44.921234+0000) 2024-04-27T03:34:45.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:44.995+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:44.996669+0000) 2024-04-27T03:34:45.422 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:45.199+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:45.200033+0000) 2024-04-27T03:34:45.422 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:45.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:45.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:45.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:45.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:45.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:45.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:45.254+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:45.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:45.416+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:45.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:45.416+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:45.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:45.416+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:45.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:45.417+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:45.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:45.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:45.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:45.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:45.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:45.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:45.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:45.644+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:46.033 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:34:45.861+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66256 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:34:46.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:45.857+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:45.857823+0000) 2024-04-27T03:34:46.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:45.996+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:45.997237+0000) 2024-04-27T03:34:46.034 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:45.856+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:45.857717+0000) 2024-04-27T03:34:46.034 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:45.858+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:45.859698+0000) 2024-04-27T03:34:46.034 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:45.920+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:45.921713+0000) 2024-04-27T03:34:46.305 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:46.200+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:46.200906+0000) 2024-04-27T03:34:46.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:46.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:46.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:46.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:46.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:46.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:46.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:46.451+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:46.636 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:46.300+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:46.636 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:46.300+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:46.636 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:46.300+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:46.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:46.300+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:46.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:46.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:46.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:46.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:46.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:46.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:46.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:46.630+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:47.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:46.923+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:46.923953+0000) 2024-04-27T03:34:47.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:46.999+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:46.999827+0000) 2024-04-27T03:34:47.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:47.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:47.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:47.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:47.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:47.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:47.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:47.407+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:47.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:47.202+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:47.202992+0000) 2024-04-27T03:34:47.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:47.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:47.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:47.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:47.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:47.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:47.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:47.286+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:47.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:47.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:47.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:47.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:47.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:47.661+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:47.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:47.661+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:48.010 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:47.925+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:47.926489+0000) 2024-04-27T03:34:48.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:48.206+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:48.206949+0000) 2024-04-27T03:34:48.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:48.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:48.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:48.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:48.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:48.239+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:48.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:48.239+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:48.459 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:48.003+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:48.004215+0000) 2024-04-27T03:34:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:48.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:48.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:48.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:48.454+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:48.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:48.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:48.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:48.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:48.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:48.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:48.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:48.661+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:49.034 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:48.928+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:48.929645+0000) 2024-04-27T03:34:49.034 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:49.006+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:49.007048+0000) 2024-04-27T03:34:49.431 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:49.208+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:49.209661+0000) 2024-04-27T03:34:49.431 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:49.245+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:49.431 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:49.245+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:49.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:49.245+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:49.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:49.246+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:49.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:49.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:49.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:49.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:49.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:49.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:49.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:49.426+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:49.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:49.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:49.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:49.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:49.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:49.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:49.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:49.660+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:50.033 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:49.931+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:49.932143+0000) 2024-04-27T03:34:50.034 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:50.009+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:50.009984+0000) 2024-04-27T03:34:50.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:50.212+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:50.212976+0000) 2024-04-27T03:34:50.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:50.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:50.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:50.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:50.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:50.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:50.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:50.270+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:50.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:50.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:50.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:50.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:50.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:50.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:50.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:50.435+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:50.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:50.867+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:50.867981+0000) 2024-04-27T03:34:50.892 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:34:50.869+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66294 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:34:50.893 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:50.865+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:50.866381+0000) 2024-04-27T03:34:50.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:50.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:50.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:50.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:50.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:50.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:50.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:50.636+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:50.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:50.865+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:50.866322+0000) 2024-04-27T03:34:51.179 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:50.932+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:50.932947+0000) 2024-04-27T03:34:51.179 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:51.010+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:51.010902+0000) 2024-04-27T03:34:51.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:51.212+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:51.213419+0000) 2024-04-27T03:34:51.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:51.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:51.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:51.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:51.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:51.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:51.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:51.241+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:51.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:51.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:51.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:51.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:51.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:51.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:51.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:51.407+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:51.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:51.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:51.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:51.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:51.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:51.664+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:51.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:51.665+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:52.033 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:51.934+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:51.935602+0000) 2024-04-27T03:34:52.034 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:52.012+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:52.013019+0000) 2024-04-27T03:34:52.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:52.215+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:52.215937+0000) 2024-04-27T03:34:52.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:52.252+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:52.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:52.252+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:52.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:52.252+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:52.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:52.252+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:52.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:52.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:52.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:52.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:52.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:52.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:52.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:52.661+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:52.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:52.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:52.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:52.428+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:53.034 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:52.936+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:52.937735+0000) 2024-04-27T03:34:53.034 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:53.014+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:53.015107+0000) 2024-04-27T03:34:53.315 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:53.217+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:53.218062+0000) 2024-04-27T03:34:53.315 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:53.277+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:53.315 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:53.277+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:53.315 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:53.277+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:53.315 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:53.278+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:53.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:53.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:53.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:53.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:53.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:53.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:53.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:53.390+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:53.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:53.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:53.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:53.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:53.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:53.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:53.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:53.623+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:54.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:53.939+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:53.939845+0000) 2024-04-27T03:34:54.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:54.017+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:54.018167+0000) 2024-04-27T03:34:54.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:54.219+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:54.220210+0000) 2024-04-27T03:34:54.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:54.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:54.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:54.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:54.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:54.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:54.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:54.311+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:54.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:54.355+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:54.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:54.355+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:54.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:54.355+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:54.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:54.356+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:54.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:54.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:54.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:54.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:54.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:54.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:54.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:54.628+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:55.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:55.022+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:55.022935+0000) 2024-04-27T03:34:55.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:54.941+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:54.942268+0000) 2024-04-27T03:34:55.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:55.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:55.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:55.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:55.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:55.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:55.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:55.380+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:55.587 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:55.221+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:55.222524+0000) 2024-04-27T03:34:55.587 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:55.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:55.587 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:55.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:55.588 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:55.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:55.588 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:55.285+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:55.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:55.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:55.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:55.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:55.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:55.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:55.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:55.581+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:56.179 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:34:55.878+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66335 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:34:56.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:55.874+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:55.875018+0000) 2024-04-27T03:34:56.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:55.873+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:55.874788+0000) 2024-04-27T03:34:56.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:55.942+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:55.942919+0000) 2024-04-27T03:34:56.180 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:55.875+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:55.875928+0000) 2024-04-27T03:34:56.180 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:56.022+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:56.023388+0000) 2024-04-27T03:34:56.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:56.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:56.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:56.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:56.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:56.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:56.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:56.402+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:56.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:56.222+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:56.223426+0000) 2024-04-27T03:34:56.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:56.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:56.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:56.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:56.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:56.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:56.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:56.237+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:56.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:56.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:56.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:56.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:56.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:56.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:56.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:56.596+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:57.034 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:56.945+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:56.946080+0000) 2024-04-27T03:34:57.034 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:57.024+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:57.025521+0000) 2024-04-27T03:34:57.394 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:57.227+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:57.228169+0000) 2024-04-27T03:34:57.394 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:57.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:57.394 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:57.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:57.394 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:57.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:57.394 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:57.253+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:57.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:57.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:57.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:57.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:57.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:57.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:57.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:57.389+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:57.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:57.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:57.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:57.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:57.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:57.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:57.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:57.598+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:58.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:57.947+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:57.948729+0000) 2024-04-27T03:34:58.094 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:58.026+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:58.027707+0000) 2024-04-27T03:34:58.409 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:58.230+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:58.230818+0000) 2024-04-27T03:34:58.409 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:58.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:58.409 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:58.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:58.409 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:58.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:58.409 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:58.286+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:58.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:58.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:58.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:58.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:58.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:58.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:58.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:58.404+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:58.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:58.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:58.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:58.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:58.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:58.610+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:58.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:58.611+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:59.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:58.950+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:58.951008+0000) 2024-04-27T03:34:59.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:59.232+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:59.233032+0000) 2024-04-27T03:34:59.334 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:59.031+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:59.032040+0000) 2024-04-27T03:34:59.636 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:59.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:59.636 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:59.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:59.636 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:59.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:34:59.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:34:59.328+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:59.637 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:59.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:59.637 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:59.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:59.637 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:59.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:34:59.637 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:59.398+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:34:59.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:59.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:59.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:59.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:59.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:59.630+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:34:59.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:34:59.631+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:00.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:34:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:34:59.953+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:34:59.954116+0000) 2024-04-27T03:35:00.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:00.033+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:00.034262+0000) 2024-04-27T03:35:00.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:00.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:00.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:00.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:00.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:00.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:00.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:00.419+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:00.639 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:00.236+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:00.237790+0000) 2024-04-27T03:35:00.640 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:00.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:00.640 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:00.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:00.640 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:00.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:00.640 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:00.348+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:00.640 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:00.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:00.641 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:00.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:00.641 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:00.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:00.889 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:00.882+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:00.883264+0000) 2024-04-27T03:35:00.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:00.633+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:00.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:00.882+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:00.883210+0000) 2024-04-27T03:35:01.180 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:35:00.886+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66373 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:35:01.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:00.884+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:00.885077+0000) 2024-04-27T03:35:01.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:00.953+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:00.954598+0000) 2024-04-27T03:35:01.180 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:01.033+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:01.034711+0000) 2024-04-27T03:35:01.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:01.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:01.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:01.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:01.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:01.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:01.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:01.399+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:01.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:01.237+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:01.238689+0000) 2024-04-27T03:35:01.467 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:01.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:01.467 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:01.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:01.467 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:01.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:01.467 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:01.349+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:01.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:01.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:01.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:01.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:01.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:01.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:01.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:01.605+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:02.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:01.955+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:01.956639+0000) 2024-04-27T03:35:02.364 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:02.242+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:02.242891+0000) 2024-04-27T03:35:02.365 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:02.038+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:02.038971+0000) 2024-04-27T03:35:02.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:02.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:02.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:02.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:02.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:02.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:02.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:02.584+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:02.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:02.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:02.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:02.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:02.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:02.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:02.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:02.359+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:02.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:02.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:02.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:02.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:02.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:02.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:02.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:02.380+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:03.046 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:02.958+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:02.959479+0000) 2024-04-27T03:35:03.346 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:03.244+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:03.245053+0000) 2024-04-27T03:35:03.346 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:03.040+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:03.041590+0000) 2024-04-27T03:35:03.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:03.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:03.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:03.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:03.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:03.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:03.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:03.341+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:03.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:03.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:03.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:03.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:03.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:03.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:03.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:03.403+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:03.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:03.612+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:03.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:03.612+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:03.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:03.612+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:03.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:03.613+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:04.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:03.963+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:03.964619+0000) 2024-04-27T03:35:04.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:04.043+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:04.044365+0000) 2024-04-27T03:35:04.650 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:04.359+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:04.650 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:04.359+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:04.650 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:04.359+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:04.650 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:04.359+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:04.650 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:04.246+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:04.247174+0000) 2024-04-27T03:35:04.650 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:04.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:04.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:04.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:04.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:04.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:04.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:04.370+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:04.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:04.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:04.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:04.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:04.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:04.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:04.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:04.645+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:05.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:04.968+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:04.969702+0000) 2024-04-27T03:35:05.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:05.047+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:05.048652+0000) 2024-04-27T03:35:05.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:05.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:05.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:05.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:05.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:05.338+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:05.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:05.339+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:05.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:05.249+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:05.250322+0000) 2024-04-27T03:35:05.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:05.385+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:05.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:05.385+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:05.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:05.385+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:05.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:05.385+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:05.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:05.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:05.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:05.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:05.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:05.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:05.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:05.631+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:06.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:05.892+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:05.893267+0000) 2024-04-27T03:35:06.181 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:35:05.894+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66414 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:35:06.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:05.890+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:05.891286+0000) 2024-04-27T03:35:06.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:06.048+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:06.049389+0000) 2024-04-27T03:35:06.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:05.892+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:05.892872+0000) 2024-04-27T03:35:06.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:05.969+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:05.970248+0000) 2024-04-27T03:35:06.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:06.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:06.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:06.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:06.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:06.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:06.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:06.378+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:06.467 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:06.250+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:06.251063+0000) 2024-04-27T03:35:06.467 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:06.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:06.467 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:06.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:06.467 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:06.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:06.467 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:06.344+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:06.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:06.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:06.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:06.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:06.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:06.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:06.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:06.622+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:07.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:06.971+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:06.972379+0000) 2024-04-27T03:35:07.326 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:07.252+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:07.253177+0000) 2024-04-27T03:35:07.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:07.051+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:07.052346+0000) 2024-04-27T03:35:07.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:07.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:07.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:07.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:07.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:07.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:07.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:07.340+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:07.621 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:07.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:07.622 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:07.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:07.622 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:07.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:07.622 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:07.321+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:07.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:07.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:07.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:07.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:07.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:07.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:07.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:07.616+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:08.241 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:07.975+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:07.976062+0000) 2024-04-27T03:35:08.241 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:08.054+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:08.054903+0000) 2024-04-27T03:35:08.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:08.333+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:08.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:08.333+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:08.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:08.333+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:08.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:08.333+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:08.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:08.256+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:08.257500+0000) 2024-04-27T03:35:08.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:08.300+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:08.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:08.300+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:08.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:08.300+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:08.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:08.301+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:08.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:08.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:08.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:08.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:08.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:08.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:08.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:08.653+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:09.241 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:08.979+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:08.980296+0000) 2024-04-27T03:35:09.241 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:09.057+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:09.058632+0000) 2024-04-27T03:35:09.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:09.330+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:09.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:09.330+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:09.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:09.330+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:09.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:09.331+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:09.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:09.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:09.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:09.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:09.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:09.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:09.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:09.260+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:09.261748+0000) 2024-04-27T03:35:09.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:09.261+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:09.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:09.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:09.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:09.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:09.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:09.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:09.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:09.639+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:10.241 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:09.982+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:09.982956+0000) 2024-04-27T03:35:10.241 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:10.061+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:10.061822+0000) 2024-04-27T03:35:10.242 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:10.223+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:10.242 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:10.223+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:10.242 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:10.223+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:10.242 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:10.223+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:10.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:10.263+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:10.264785+0000) 2024-04-27T03:35:10.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:10.337+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:10.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:10.337+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:10.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:10.337+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:10.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:10.338+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:10.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:10.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:10.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:10.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:10.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:10.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:10.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:10.633+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:11.181 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:35:10.902+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66452 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:35:11.181 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:10.899+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:10.899933+0000) 2024-04-27T03:35:11.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:10.899+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:10.899819+0000) 2024-04-27T03:35:11.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:10.983+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:10.983904+0000) 2024-04-27T03:35:11.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:10.899+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:10.900457+0000) 2024-04-27T03:35:11.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:11.061+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:11.062267+0000) 2024-04-27T03:35:11.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:11.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:11.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:11.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:11.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:11.379+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:11.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:11.380+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:11.467 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:11.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:11.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:11.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:11.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:11.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:11.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:11.247+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:11.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:11.264+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:11.265714+0000) 2024-04-27T03:35:11.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:11.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:11.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:11.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:11.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:11.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:11.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:11.639+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:12.008 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:11.985+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:11.986530+0000) 2024-04-27T03:35:12.273 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:12.066+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:12.067230+0000) 2024-04-27T03:35:12.273 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:12.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:12.273 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:12.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:12.273 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:12.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:12.273 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:12.208+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:12.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:12.267+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:12.267879+0000) 2024-04-27T03:35:12.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:12.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:12.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:12.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:12.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:12.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:12.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:12.428+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:12.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:12.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:12.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:12.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:12.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:12.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:12.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:12.672+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:13.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:12.988+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:12.989256+0000) 2024-04-27T03:35:13.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:13.068+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:13.069556+0000) 2024-04-27T03:35:13.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:13.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:13.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:13.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:13.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:13.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:13.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:13.194+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:13.680 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:13.269+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:13.270141+0000) 2024-04-27T03:35:13.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:13.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:13.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:13.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:13.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:13.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:13.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:13.460+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:13.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:13.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:13.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:13.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:13.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:13.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:13.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:13.675+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:14.247 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:13.990+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:13.991347+0000) 2024-04-27T03:35:14.247 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:14.071+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:14.071825+0000) 2024-04-27T03:35:14.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:14.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:14.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:14.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:14.515 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:14.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:14.515 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:14.241+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:14.515 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:14.271+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:14.272610+0000) 2024-04-27T03:35:14.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:14.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:14.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:14.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:14.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:14.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:14.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:14.510+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:14.803 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:14.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:14.803 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:14.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:14.803 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:14.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:14.803 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:14.655+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:15.079 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:14.992+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:14.993566+0000) 2024-04-27T03:35:15.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:15.073+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:15.074478+0000) 2024-04-27T03:35:15.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:15.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:15.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:15.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:15.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:15.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:15.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:15.228+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:15.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:15.275+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:15.276843+0000) 2024-04-27T03:35:15.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:15.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:15.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:15.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:15.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:15.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:15.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:15.624+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:15.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:15.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:15.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:15.495+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:15.495+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:16.079 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:35:15.911+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66493 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:35:16.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:15.906+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:15.908480+0000) 2024-04-27T03:35:16.080 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:15.905+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:15.907578+0000) 2024-04-27T03:35:16.080 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:15.992+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:15.994564+0000) 2024-04-27T03:35:16.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:15.907+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:15.908832+0000) 2024-04-27T03:35:16.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:16.073+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:16.074909+0000) 2024-04-27T03:35:16.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:16.185+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:16.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:16.185+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:16.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:16.185+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:16.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:16.185+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:16.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:16.275+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:16.277358+0000) 2024-04-27T03:35:16.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:16.584+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:16.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:16.584+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:16.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:16.584+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:16.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:16.584+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:16.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:16.469+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:16.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:16.469+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:16.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:16.469+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:16.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:16.470+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:17.083 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:16.994+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:16.996784+0000) 2024-04-27T03:35:17.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:17.076+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:17.078025+0000) 2024-04-27T03:35:17.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:17.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:17.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:17.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:17.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:17.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:17.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:17.148+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:17.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:17.278+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:17.280260+0000) 2024-04-27T03:35:17.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:17.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:17.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:17.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:17.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:17.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:17.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:17.631+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:17.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:17.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:17.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:17.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:17.475+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:18.086 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:17.998+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:18.000029+0000) 2024-04-27T03:35:18.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:18.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:18.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:18.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:18.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:18.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:18.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:18.108+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:18.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:18.282+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:18.284459+0000) 2024-04-27T03:35:18.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:18.079+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:18.081012+0000) 2024-04-27T03:35:18.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:18.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:18.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:18.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:18.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:18.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:18.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:18.469+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:18.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:18.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:18.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:18.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:18.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:18.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:18.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:18.619+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:19.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:19.003+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:19.005142+0000) 2024-04-27T03:35:19.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:19.078+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:19.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:19.078+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:19.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:19.078+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:19.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:19.079+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:19.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:19.285+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:19.286842+0000) 2024-04-27T03:35:19.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:19.082+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:19.083922+0000) 2024-04-27T03:35:19.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:19.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:19.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:19.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:19.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:19.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:19.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:19.454+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:19.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:19.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:19.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:19.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:19.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:19.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:19.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:19.620+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:20.091 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:20.008+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:20.010205+0000) 2024-04-27T03:35:20.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:20.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:20.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:20.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:20.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:20.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:20.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:20.084+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:20.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:20.288+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:20.290054+0000) 2024-04-27T03:35:20.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:20.085+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:20.087234+0000) 2024-04-27T03:35:20.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:20.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:20.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:20.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:20.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:20.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:20.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:20.502+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:20.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:20.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:20.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:20.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:20.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:20.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:20.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:20.587+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:21.092 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:35:20.919+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66531 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:35:21.093 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:20.915+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:20.917668+0000) 2024-04-27T03:35:21.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:20.916+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:20.918470+0000) 2024-04-27T03:35:21.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:21.008+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:21.010760+0000) 2024-04-27T03:35:21.093 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:20.915+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:20.917121+0000) 2024-04-27T03:35:21.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:21.086+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:21.087829+0000) 2024-04-27T03:35:21.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:21.116+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:21.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:21.116+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:21.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:21.116+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:21.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:21.117+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:21.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:21.289+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:21.290825+0000) 2024-04-27T03:35:21.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:21.539+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:21.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:21.539+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:21.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:21.539+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:21.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:21.540+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:21.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:21.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:21.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:21.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:21.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:21.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:21.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:21.638+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:22.091 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:22.012+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:22.013951+0000) 2024-04-27T03:35:22.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:22.088+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:22.090266+0000) 2024-04-27T03:35:22.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:22.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:22.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:22.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:22.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:22.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:22.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:22.105+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:22.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:22.293+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:22.294989+0000) 2024-04-27T03:35:22.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:22.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:22.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:22.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:22.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:22.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:22.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:22.511+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:22.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:22.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:22.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:22.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:22.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:22.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:22.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:22.616+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:23.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:23.014+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:23.016611+0000) 2024-04-27T03:35:23.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:23.094+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:23.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:23.094+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:23.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:23.094+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:23.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:23.095+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:23.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:23.295+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:23.297729+0000) 2024-04-27T03:35:23.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:23.091+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:23.093222+0000) 2024-04-27T03:35:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:23.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:23.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:23.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:23.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:23.514+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:23.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:23.665+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:23.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:23.665+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:23.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:23.665+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:23.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:23.666+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:24.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:24.016+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:24.018693+0000) 2024-04-27T03:35:24.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:24.096+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:24.098136+0000) 2024-04-27T03:35:24.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:24.098+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:24.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:24.098+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:24.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:24.098+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:24.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:24.098+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:24.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:24.298+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:24.300565+0000) 2024-04-27T03:35:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:24.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:24.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:24.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:24.472+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:24.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:24.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:24.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:24.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:24.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:24.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:24.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:24.623+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:25.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:25.021+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:25.023670+0000) 2024-04-27T03:35:25.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:25.100+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:25.102549+0000) 2024-04-27T03:35:25.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:25.067+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:25.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:25.067+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:25.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:25.067+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:25.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:25.068+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:25.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:25.300+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:25.302704+0000) 2024-04-27T03:35:25.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:25.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:25.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:25.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:25.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:25.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:25.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:25.647+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:25.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:25.488+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:25.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:25.488+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:25.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:25.488+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:25.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:25.489+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:26.067 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:35:25.927+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66572 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:35:26.067 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:25.923+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:25.925584+0000) 2024-04-27T03:35:26.067 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:26.022+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:26.024721+0000) 2024-04-27T03:35:26.067 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:25.923+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:25.925582+0000) 2024-04-27T03:35:26.067 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:26.060+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:26.068 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:26.060+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:26.068 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:26.060+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:26.068 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:25.924+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:25.926056+0000) 2024-04-27T03:35:26.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:26.101+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:26.103199+0000) 2024-04-27T03:35:26.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:26.060+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:26.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:26.301+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:26.303654+0000) 2024-04-27T03:35:26.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:26.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:26.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:26.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:26.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:26.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:26.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:26.521+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:26.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:26.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:26.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:26.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:26.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:26.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:26.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:26.673+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:27.314 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:27.024+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:27.026793+0000) 2024-04-27T03:35:27.314 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:27.103+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:27.105426+0000) 2024-04-27T03:35:27.314 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:27.076+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:27.314 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:27.076+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:27.315 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:27.076+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:27.315 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:27.077+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:27.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:27.307+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:27.308735+0000) 2024-04-27T03:35:27.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:27.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:27.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:27.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:27.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:27.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:27.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:27.489+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:27.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:27.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:27.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:27.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:27.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:27.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:27.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:27.658+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:28.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:28.028+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:28.029555+0000) 2024-04-27T03:35:28.317 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:28.109+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:28.110496+0000) 2024-04-27T03:35:28.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:28.094+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:28.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:28.094+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:28.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:28.094+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:28.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:28.095+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:28.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:28.310+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:28.311642+0000) 2024-04-27T03:35:28.645 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:28.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:28.645 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:28.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:28.646 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:28.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:28.646 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:28.538+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:28.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:28.639+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:28.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:28.639+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:28.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:28.639+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:28.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:28.640+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:29.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:29.031+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:29.032664+0000) 2024-04-27T03:35:29.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:29.112+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:29.113293+0000) 2024-04-27T03:35:29.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:29.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:29.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:29.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:29.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:29.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:29.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:29.112+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:29.640 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:29.313+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:29.314524+0000) 2024-04-27T03:35:29.640 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:29.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:29.640 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:29.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:29.640 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:29.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:29.640 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:29.540+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:29.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:29.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:29.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:29.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:29.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:29.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:29.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:29.635+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:30.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:30.034+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:30.035474+0000) 2024-04-27T03:35:30.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:30.115+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:30.115911+0000) 2024-04-27T03:35:30.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:30.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:30.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:30.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:30.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:30.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:30.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:30.125+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:30.601 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:30.316+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:30.317497+0000) 2024-04-27T03:35:30.601 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:30.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:30.602 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:30.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:30.602 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:30.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:30.602 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:30.546+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:30.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:30.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:30.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:30.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:30.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:30.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:30.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:30.596+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:31.226 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:35:30.936+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66610 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:35:31.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:30.933+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:30.934673+0000) 2024-04-27T03:35:31.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:31.035+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:31.036463+0000) 2024-04-27T03:35:31.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:30.933+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:30.933949+0000) 2024-04-27T03:35:31.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:31.115+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:31.116564+0000) 2024-04-27T03:35:31.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:30.932+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:30.933672+0000) 2024-04-27T03:35:31.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:31.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:31.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:31.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:31.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:31.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:31.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:31.132+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:31.519 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:31.317+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:31.318435+0000) 2024-04-27T03:35:31.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:31.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:31.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:31.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:31.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:31.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:31.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:31.515+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:31.818 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:31.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:31.818 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:31.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:31.818 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:31.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:31.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:31.584+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:32.098 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:32.037+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:32.038561+0000) 2024-04-27T03:35:32.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:32.120+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:32.121562+0000) 2024-04-27T03:35:32.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:32.152+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:32.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:32.152+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:32.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:32.152+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:32.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:32.153+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:32.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:32.319+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:32.320684+0000) 2024-04-27T03:35:32.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:32.542+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:32.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:32.542+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:32.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:32.542+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:32.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:32.543+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:32.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:32.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:32.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:32.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:32.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:32.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:32.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:32.555+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:33.098 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:33.040+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:33.041678+0000) 2024-04-27T03:35:33.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:33.125+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:33.125856+0000) 2024-04-27T03:35:33.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:33.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:33.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:33.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:33.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:33.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:33.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:33.134+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:33.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:33.324+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:33.324896+0000) 2024-04-27T03:35:33.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:33.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:33.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:33.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:33.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:33.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:33.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:33.551+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:33.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:33.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:33.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:33.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:33.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:33.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:33.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:33.517+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:34.098 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:34.042+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:34.043762+0000) 2024-04-27T03:35:34.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:34.127+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:34.127979+0000) 2024-04-27T03:35:34.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:34.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:34.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:34.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:34.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:34.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:34.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:34.175+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:34.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:34.328+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:34.329118+0000) 2024-04-27T03:35:34.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:34.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:34.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:34.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:34.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:34.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:34.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:34.487+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:34.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:34.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:34.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:34.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:34.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:34.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:34.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:34.583+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:35.098 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:35.045+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:35.045844+0000) 2024-04-27T03:35:35.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:35.129+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:35.130094+0000) 2024-04-27T03:35:35.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:35.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:35.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:35.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:35.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:35.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:35.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:35.145+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:35.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:35.330+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:35.331380+0000) 2024-04-27T03:35:35.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:35.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:35.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:35.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:35.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:35.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:35.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:35.530+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:35.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:35.608+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:35.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:35.608+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:35.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:35.608+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:35.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:35.609+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:36.098 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:35.941+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:35.942712+0000) 2024-04-27T03:35:36.099 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:35:35.944+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66651 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:35:36.099 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:35.941+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:35.942234+0000) 2024-04-27T03:35:36.099 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:36.045+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:36.046397+0000) 2024-04-27T03:35:36.099 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:35.941+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:35.942335+0000) 2024-04-27T03:35:36.372 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:36.130+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:36.131024+0000) 2024-04-27T03:35:36.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:36.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:36.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:36.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:36.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:36.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:36.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:36.193+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:36.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:36.331+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:36.332016+0000) 2024-04-27T03:35:36.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:36.484+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:36.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:36.484+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:36.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:36.484+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:36.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:36.484+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:36.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:36.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:36.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:36.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:36.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:36.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:36.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:36.659+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:37.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:37.047+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:37.048521+0000) 2024-04-27T03:35:37.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:37.132+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:37.133134+0000) 2024-04-27T03:35:37.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:37.153+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:37.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:37.153+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:37.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:37.153+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:37.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:37.154+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:37.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:37.335+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:37.336239+0000) 2024-04-27T03:35:37.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:37.488+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:37.657 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:37.488+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:37.657 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:37.488+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:37.657 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:37.489+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:37.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:37.650+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:37.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:37.650+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:37.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:37.650+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:37.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:37.652+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:38.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:38.049+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:38.050595+0000) 2024-04-27T03:35:38.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:38.134+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:38.135290+0000) 2024-04-27T03:35:38.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:38.189+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:38.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:38.189+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:38.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:38.189+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:38.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:38.189+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:38.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:38.338+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:38.338987+0000) 2024-04-27T03:35:38.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:38.499+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:38.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:38.499+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:38.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:38.499+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:38.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:38.500+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:38.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:38.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:38.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:38.675+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:39.348 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:39.052+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:39.052844+0000) 2024-04-27T03:35:39.348 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:39.136+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:39.137606+0000) 2024-04-27T03:35:39.349 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:39.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:39.349 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:39.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:39.349 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:39.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:39.349 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:39.161+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:39.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:39.342+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:39.343368+0000) 2024-04-27T03:35:39.674 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:39.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:39.674 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:39.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:39.674 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:39.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:39.674 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:39.526+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:39.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:39.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:39.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:39.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:39.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:39.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:39.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:39.669+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:40.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:40.054+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:40.054954+0000) 2024-04-27T03:35:40.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:40.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:40.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:40.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:40.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:40.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:40.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:40.210+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:40.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:40.139+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:40.139823+0000) 2024-04-27T03:35:40.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:40.345+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:40.346092+0000) 2024-04-27T03:35:40.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:40.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:40.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:40.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:40.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:40.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:40.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:40.520+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:40.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:40.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:40.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:40.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:40.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:40.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:40.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:40.699+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:40.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:40.949+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:40.950076+0000) 2024-04-27T03:35:41.225 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:35:40.953+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66690 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:35:41.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:41.139+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:41.140367+0000) 2024-04-27T03:35:41.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:40.949+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:40.950460+0000) 2024-04-27T03:35:41.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:41.054+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:41.055485+0000) 2024-04-27T03:35:41.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:40.950+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:40.951168+0000) 2024-04-27T03:35:41.532 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:41.233+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:41.532 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:41.233+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:41.532 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:41.233+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:41.532 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:41.234+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:41.532 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:41.346+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:41.346989+0000) 2024-04-27T03:35:41.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:41.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:41.831 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:41.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:41.831 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:41.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:41.831 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:41.527+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:41.831 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:41.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:41.831 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:41.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:41.831 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:41.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:41.831 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:41.681+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:42.102 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:42.059+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:42.060277+0000) 2024-04-27T03:35:42.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:42.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:42.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:42.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:42.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:42.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:42.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:42.209+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:42.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:42.141+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:42.142580+0000) 2024-04-27T03:35:42.652 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:42.348+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:42.349102+0000) 2024-04-27T03:35:42.653 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:42.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:42.653 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:42.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:42.653 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:42.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:42.653 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:42.563+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:42.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:42.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:42.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:42.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:42.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:42.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:42.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:42.647+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:43.356 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:43.061+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:43.062399+0000) 2024-04-27T03:35:43.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:43.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:43.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:43.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:43.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:43.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:43.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:43.184+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:43.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:43.143+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:43.144708+0000) 2024-04-27T03:35:43.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:43.350+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:43.351675+0000) 2024-04-27T03:35:43.633 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:43.515+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:43.633 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:43.515+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:43.634 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:43.515+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:43.634 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:43.516+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:43.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:43.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:43.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:43.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:43.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:43.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:43.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:43.629+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:44.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:44.064+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:44.065242+0000) 2024-04-27T03:35:44.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:44.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:44.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:44.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:44.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:44.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:44.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:44.189+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:44.360 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:44.147+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:44.148022+0000) 2024-04-27T03:35:44.669 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:44.353+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:44.354091+0000) 2024-04-27T03:35:44.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:44.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:44.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:44.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:44.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:44.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:44.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:44.549+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:44.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:44.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:44.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:44.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:44.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:44.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:44.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:44.664+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:45.363 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:45.066+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:45.067502+0000) 2024-04-27T03:35:45.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:45.149+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:45.150105+0000) 2024-04-27T03:35:45.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:45.234+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:45.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:45.234+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:45.364 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:45.234+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:45.364 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:45.234+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:45.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:45.523+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:45.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:45.523+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:45.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:45.523+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:45.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:45.524+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:45.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:45.357+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:45.358118+0000) 2024-04-27T03:35:45.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:45.958+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:45.959464+0000) 2024-04-27T03:35:45.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:45.957+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:45.958399+0000) 2024-04-27T03:35:45.974 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:35:45.960+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66730 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:35:45.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:45.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:45.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:45.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:45.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:45.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:45.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:45.714+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:45.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:45.957+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:45.958575+0000) 2024-04-27T03:35:46.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:46.067+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:46.068223+0000) 2024-04-27T03:35:46.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:46.149+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:46.150677+0000) 2024-04-27T03:35:46.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:46.258+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:46.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:46.258+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:46.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:46.258+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:46.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:46.259+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:46.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:46.358+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:46.359242+0000) 2024-04-27T03:35:46.838 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:46.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:46.838 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:46.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:46.838 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:46.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:46.838 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:46.566+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:46.838 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:46.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:46.839 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:46.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:46.839 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:46.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:46.839 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:46.741+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:47.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:47.069+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:47.070335+0000) 2024-04-27T03:35:47.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:47.153+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:47.153920+0000) 2024-04-27T03:35:47.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:47.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:47.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:47.303+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:47.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:47.303+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:47.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:47.303+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:47.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:47.360+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:47.361381+0000) 2024-04-27T03:35:47.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:47.551+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:47.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:47.551+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:47.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:47.551+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:47.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:47.552+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:48.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:47.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:48.040 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:47.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:48.040 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:47.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:48.040 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:47.771+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:48.326 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:48.071+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:48.072456+0000) 2024-04-27T03:35:48.326 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:48.155+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:48.155994+0000) 2024-04-27T03:35:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:48.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:48.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:48.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:48.504+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:48.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:48.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:48.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:48.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:48.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:48.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:48.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:48.320+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:48.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:48.364+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:48.365350+0000) 2024-04-27T03:35:49.040 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:48.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:49.040 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:48.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:49.040 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:48.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:49.040 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:48.738+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:49.358 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:49.074+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:49.075307+0000) 2024-04-27T03:35:49.358 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:49.157+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:49.158729+0000) 2024-04-27T03:35:49.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:49.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:49.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:49.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:49.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:49.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:49.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:49.352+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:49.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:49.368+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:49.369662+0000) 2024-04-27T03:35:49.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:49.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:49.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:49.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:49.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:49.505+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:49.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:49.506+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:49.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:49.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:49.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:49.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:49.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:49.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:49.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:49.746+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:50.324 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:50.077+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:50.077948+0000) 2024-04-27T03:35:50.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:50.160+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:50.161045+0000) 2024-04-27T03:35:50.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:50.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:50.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:50.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:50.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:50.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:50.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:50.535+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:50.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:50.372+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:50.372937+0000) 2024-04-27T03:35:50.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:50.398+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:50.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:50.398+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:50.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:50.398+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:50.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:50.398+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:51.040 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:35:50.968+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66769 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:35:51.040 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:50.965+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:50.966159+0000) 2024-04-27T03:35:51.040 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:50.964+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:50.965731+0000) 2024-04-27T03:35:51.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:50.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:51.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:50.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:51.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:50.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:51.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:50.767+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:51.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:50.965+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:50.965839+0000) 2024-04-27T03:35:51.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:51.077+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:51.078485+0000) 2024-04-27T03:35:51.379 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:51.160+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:51.161550+0000) 2024-04-27T03:35:51.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:51.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:51.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:51.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:51.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:51.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:51.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:51.580+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:51.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:51.372+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:51.373488+0000) 2024-04-27T03:35:51.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:51.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:51.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:51.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:51.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:51.411+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:51.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:51.411+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:51.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:51.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:51.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:51.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:51.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:51.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:51.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:51.726+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:52.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:52.079+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:52.080778+0000) 2024-04-27T03:35:52.381 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:52.162+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:52.163760+0000) 2024-04-27T03:35:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:52.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:52.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:52.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:52.591+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:52.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:52.375+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:52.376083+0000) 2024-04-27T03:35:52.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:52.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:52.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:52.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:52.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:52.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:52.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:52.379+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:53.040 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:52.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:53.040 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:52.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:53.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:52.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:53.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:52.771+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:53.384 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:53.083+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:53.084001+0000) 2024-04-27T03:35:53.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:53.165+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:53.166487+0000) 2024-04-27T03:35:53.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:53.378+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:53.379288+0000) 2024-04-27T03:35:53.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:53.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:53.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:53.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:53.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:53.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:53.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:53.423+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:53.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:53.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:53.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:53.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:53.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:53.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:53.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:53.580+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:54.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:53.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:54.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:53.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:54.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:53.793+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:54.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:53.794+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:54.387 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:54.085+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:54.086159+0000) 2024-04-27T03:35:54.387 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:54.167+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:54.168651+0000) 2024-04-27T03:35:54.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:54.381+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:54.382346+0000) 2024-04-27T03:35:54.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:54.473+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:54.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:54.473+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:54.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:54.473+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:54.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:54.474+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:54.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:54.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:54.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:54.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:54.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:54.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:54.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:54.621+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:55.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:54.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:55.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:54.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:55.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:54.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:55.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:54.746+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:55.391 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:55.089+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:55.090058+0000) 2024-04-27T03:35:55.391 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:55.172+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:55.173083+0000) 2024-04-27T03:35:55.666 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:55.385+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:55.386370+0000) 2024-04-27T03:35:55.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:55.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:55.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:55.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:55.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:55.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:55.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:55.467+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:55.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:55.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:55.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:55.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:55.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:55.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:55.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:55.783+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:55.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:55.660+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:55.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:55.660+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:55.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:55.660+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:55.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:55.661+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:56.226 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:35:55.976+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66809 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:35:56.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:55.974+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:55.974978+0000) 2024-04-27T03:35:56.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:55.974+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:55.975139+0000) 2024-04-27T03:35:56.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:56.089+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:56.090598+0000) 2024-04-27T03:35:56.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:55.974+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:55.975036+0000) 2024-04-27T03:35:56.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:56.172+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:56.173755+0000) 2024-04-27T03:35:56.664 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:56.386+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:56.387290+0000) 2024-04-27T03:35:56.664 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:56.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:56.664 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:56.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:56.664 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:56.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:56.664 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:56.421+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:56.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:56.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:56.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:56.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:56.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:56.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:56.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:56.659+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:56.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:56.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:56.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:56.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:56.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:56.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:56.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:56.770+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:57.395 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:57.093+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:57.093844+0000) 2024-04-27T03:35:57.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:57.176+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:57.176993+0000) 2024-04-27T03:35:57.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:57.389+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:57.390313+0000) 2024-04-27T03:35:57.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:57.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:57.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:57.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:57.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:57.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:57.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:57.422+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:57.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:57.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:57.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:57.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:57.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:57.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:57.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:57.726+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:57.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:57.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:57.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:57.708+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:58.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:58.095+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:58.096138+0000) 2024-04-27T03:35:58.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:58.178+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:58.179262+0000) 2024-04-27T03:35:58.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:58.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:58.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:58.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:58.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:58.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:58.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:58.390+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:58.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:58.393+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:58.394457+0000) 2024-04-27T03:35:59.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:58.723+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:59.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:58.723+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:59.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:58.723+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:59.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:58.725+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:59.041 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:58.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:59.041 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:58.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:59.041 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:58.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:59.042 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:58.745+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:59.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:59.097+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:59.098213+0000) 2024-04-27T03:35:59.402 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:59.180+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:59.181380+0000) 2024-04-27T03:35:59.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:59.395+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:35:59.396660+0000) 2024-04-27T03:35:59.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:59.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:59.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:59.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:59.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:59.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:35:59.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:35:59.413+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:59.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:59.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:59.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:59.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:59.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:59.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:35:59.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:35:59.708+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:35:59.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:59.695+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:59.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:59.695+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:59.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:59.695+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:35:59.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:35:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:35:59.696+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:00.325 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:00.100+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:00.101215+0000) 2024-04-27T03:36:00.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:00.183+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:00.184377+0000) 2024-04-27T03:36:00.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:00.400+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:00.401372+0000) 2024-04-27T03:36:00.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:00.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:00.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:00.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:00.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:00.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:00.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:00.443+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:00.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:00.702+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:00.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:00.702+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:00.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:00.702+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:00.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:00.702+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:00.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:00.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:00.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:00.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:00.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:00.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:00.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:00.693+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:01.246 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:00.981+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:00.982211+0000) 2024-04-27T03:36:01.247 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:36:00.985+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66849 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:36:01.248 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:00.981+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:00.982254+0000) 2024-04-27T03:36:01.248 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:01.101+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:01.102143+0000) 2024-04-27T03:36:01.248 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:00.981+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:00.981971+0000) 2024-04-27T03:36:01.248 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:01.184+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:01.184976+0000) 2024-04-27T03:36:01.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:01.401+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:01.401984+0000) 2024-04-27T03:36:01.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:01.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:01.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:01.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:01.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:01.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:01.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:01.444+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:01.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:01.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:01.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:01.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:01.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:01.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:01.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:01.731+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:01.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:01.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:01.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:01.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:01.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:01.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:01.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:01.700+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:02.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:02.103+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:02.104532+0000) 2024-04-27T03:36:02.410 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:02.186+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:02.187240+0000) 2024-04-27T03:36:02.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:02.403+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:02.404629+0000) 2024-04-27T03:36:02.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:02.454+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:02.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:02.454+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:02.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:02.454+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:02.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:02.455+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:03.041 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:02.726+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:03.042 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:02.726+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:03.042 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:02.726+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:03.042 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:02.727+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:03.042 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:02.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:03.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:02.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:03.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:02.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:03.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:02.727+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:03.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:03.107+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:03.107958+0000) 2024-04-27T03:36:03.412 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:03.188+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:03.189507+0000) 2024-04-27T03:36:03.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:03.406+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:03.407276+0000) 2024-04-27T03:36:03.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:03.501+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:03.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:03.501+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:03.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:03.501+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:03.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:03.501+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:03.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:03.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:03.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:03.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:03.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:03.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:03.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:03.723+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:03.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:03.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:03.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:03.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:03.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:03.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:03.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:03.712+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:04.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:04.110+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:04.111657+0000) 2024-04-27T03:36:04.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:04.190+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:04.191657+0000) 2024-04-27T03:36:04.522 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:04.408+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:04.409770+0000) 2024-04-27T03:36:04.523 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:04.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:04.523 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:04.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:04.523 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:04.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:04.863 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:04.517+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:04.863 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:04.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:04.864 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:04.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:04.864 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:04.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:04.864 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:04.729+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:04.864 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:04.758+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:04.864 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:04.758+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:04.864 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:04.758+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:04.864 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:04.759+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:05.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:05.113+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:05.114434+0000) 2024-04-27T03:36:05.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:05.193+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:05.194444+0000) 2024-04-27T03:36:05.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:05.411+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:05.412485+0000) 2024-04-27T03:36:05.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:05.505+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:05.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:05.505+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:05.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:05.505+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:05.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:05.505+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:06.042 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:36:05.993+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66889 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:36:06.042 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:05.989+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:05.990744+0000) 2024-04-27T03:36:06.046 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:05.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:06.047 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:05.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:06.047 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:05.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:06.047 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:05.777+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:06.047 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:05.990+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:05.991712+0000) 2024-04-27T03:36:06.047 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:05.775+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:06.047 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:05.775+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:06.047 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:05.775+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:06.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:05.775+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:06.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:05.989+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:05.990386+0000) 2024-04-27T03:36:06.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:06.194+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:06.195066+0000) 2024-04-27T03:36:06.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:06.114+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:06.115599+0000) 2024-04-27T03:36:06.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:06.412+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:06.413368+0000) 2024-04-27T03:36:06.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:06.482+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:06.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:06.482+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:06.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:06.482+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:06.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:06.482+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:07.042 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:06.729+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:07.042 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:06.729+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:07.042 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:06.729+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:07.042 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:06.730+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:07.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:06.732+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:07.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:06.732+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:07.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:06.732+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:07.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:06.733+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:07.421 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:07.117+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:07.118371+0000) 2024-04-27T03:36:07.421 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:07.196+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:07.197276+0000) 2024-04-27T03:36:07.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:07.415+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:07.416275+0000) 2024-04-27T03:36:07.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:07.484+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:07.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:07.484+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:07.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:07.484+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:07.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:07.484+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:07.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:07.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:07.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:07.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:07.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:07.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:07.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:07.713+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:07.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:07.730+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:07.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:07.730+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:07.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:07.730+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:07.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:07.731+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:08.424 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:08.121+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:08.122754+0000) 2024-04-27T03:36:08.424 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:08.199+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:08.200278+0000) 2024-04-27T03:36:08.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:08.418+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:08.419025+0000) 2024-04-27T03:36:08.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:08.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:08.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:08.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:08.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:08.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:08.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:08.494+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:08.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:08.717+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:08.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:08.717+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:08.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:08.717+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:08.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:08.719+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:08.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:08.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:08.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:08.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:08.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:08.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:08.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:08.762+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:09.427 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:09.125+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:09.125958+0000) 2024-04-27T03:36:09.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:09.201+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:09.202465+0000) 2024-04-27T03:36:09.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:09.421+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:09.422245+0000) 2024-04-27T03:36:09.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:09.478+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:09.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:09.478+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:09.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:09.478+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:09.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:09.478+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:10.042 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:09.737+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:10.042 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:09.737+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:10.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:09.737+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:10.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:09.737+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:10.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:09.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:10.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:09.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:10.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:09.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:10.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:09.763+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:10.326 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:10.129+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:10.130276+0000) 2024-04-27T03:36:10.326 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:10.203+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:10.204722+0000) 2024-04-27T03:36:10.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:10.424+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:10.424975+0000) 2024-04-27T03:36:10.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:10.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:10.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:10.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:10.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:10.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:10.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:10.528+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:11.001 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:10.737+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:11.001 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:10.737+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:11.001 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:10.737+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:11.001 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:10.738+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:11.001 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:10.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:11.001 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:10.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:11.002 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:10.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:11.002 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:10.775+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:11.431 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:36:11.003+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66928 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:36:11.431 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:10.997+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:10.998575+0000) 2024-04-27T03:36:11.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:10.998+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:10.999255+0000) 2024-04-27T03:36:11.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:11.130+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:11.130955+0000) 2024-04-27T03:36:11.432 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:10.999+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:11.000249+0000) 2024-04-27T03:36:11.432 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:11.205+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:11.205892+0000) 2024-04-27T03:36:11.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:11.425+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:11.426056+0000) 2024-04-27T03:36:11.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:11.519+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:11.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:11.519+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:11.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:11.519+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:11.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:11.519+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:11.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:11.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:11.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:11.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:11.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:11.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:11.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:11.721+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:11.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:11.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:11.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:11.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:11.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:11.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:11.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:11.803+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:12.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:12.133+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:12.134098+0000) 2024-04-27T03:36:12.434 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:12.207+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:12.208040+0000) 2024-04-27T03:36:12.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:12.428+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:12.429166+0000) 2024-04-27T03:36:12.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:12.550+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:12.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:12.550+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:12.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:12.550+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:12.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:12.550+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:13.042 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:12.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:13.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:12.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:13.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:12.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:13.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:12.732+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:13.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:12.767+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:13.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:12.767+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:13.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:12.767+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:13.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:12.768+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:13.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:13.138+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:13.138922+0000) 2024-04-27T03:36:13.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:13.212+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:13.212825+0000) 2024-04-27T03:36:13.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:13.430+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:13.431601+0000) 2024-04-27T03:36:13.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:13.550+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:13.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:13.550+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:13.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:13.550+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:13.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:13.551+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:14.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:13.741+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:14.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:13.741+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:14.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:13.741+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:14.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:13.742+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:14.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:13.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:14.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:13.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:14.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:13.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:14.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:13.768+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:14.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:14.142+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:14.143155+0000) 2024-04-27T03:36:14.440 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:14.215+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:14.216103+0000) 2024-04-27T03:36:14.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:14.433+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:14.434749+0000) 2024-04-27T03:36:14.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:14.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:14.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:14.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:14.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:14.529+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:14.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:14.529+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:15.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:14.783+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:15.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:14.783+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:15.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:14.783+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:15.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:14.784+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:15.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:14.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:15.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:14.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:15.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:14.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:15.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:14.770+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:15.442 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:15.144+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:15.145303+0000) 2024-04-27T03:36:15.442 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:15.219+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:15.220426+0000) 2024-04-27T03:36:15.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:15.436+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:15.436861+0000) 2024-04-27T03:36:15.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:15.558+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:15.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:15.558+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:15.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:15.558+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:15.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:15.558+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:16.025 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:36:16.011+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 66968 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:36:16.025 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:16.009+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:16.009839+0000) 2024-04-27T03:36:16.025 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:15.765+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:16.025 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:15.765+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:16.026 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:15.765+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:16.026 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:15.766+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:16.026 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:16.009+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:16.009895+0000) 2024-04-27T03:36:16.026 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:15.775+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:16.026 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:15.775+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:16.026 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:15.775+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:16.026 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:15.775+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:16.026 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:16.007+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:16.008643+0000) 2024-04-27T03:36:16.372 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:16.220+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:16.221245+0000) 2024-04-27T03:36:16.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:16.145+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:16.145909+0000) 2024-04-27T03:36:16.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:16.436+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:16.437636+0000) 2024-04-27T03:36:16.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:16.522+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:16.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:16.522+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:16.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:16.522+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:16.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:16.523+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:17.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:16.736+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:17.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:16.736+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:17.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:16.736+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:17.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:16.737+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:17.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:16.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:17.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:16.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:17.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:16.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:17.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:16.787+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:17.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:17.148+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:17.150699+0000) 2024-04-27T03:36:17.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:17.224+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:17.225936+0000) 2024-04-27T03:36:17.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:17.440+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:17.441845+0000) 2024-04-27T03:36:17.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:17.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:17.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:17.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:17.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:17.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:17.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:17.504+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:18.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:17.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:18.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:17.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:18.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:17.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:18.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:17.786+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:18.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:17.767+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:18.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:17.767+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:18.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:17.767+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:18.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:17.768+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:18.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:18.152+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:18.153923+0000) 2024-04-27T03:36:18.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:18.226+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:18.228619+0000) 2024-04-27T03:36:18.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:18.442+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:18.444215+0000) 2024-04-27T03:36:18.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:18.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:18.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:18.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:18.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:18.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:18.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:18.466+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:19.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:18.775+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:19.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:18.775+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:19.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:18.775+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:19.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:18.776+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:19.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:18.831+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:19.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:18.831+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:19.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:18.831+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:19.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:18.832+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:19.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:19.229+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:19.231739+0000) 2024-04-27T03:36:19.332 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:19.154+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:19.156095+0000) 2024-04-27T03:36:19.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:19.444+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:19.446704+0000) 2024-04-27T03:36:19.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:19.510+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:19.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:19.510+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:19.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:19.510+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:19.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:19.510+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:20.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:19.861+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:20.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:19.861+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:20.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:19.861+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:20.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:19.862+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:20.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:19.808+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:20.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:19.808+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:20.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:19.808+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:20.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:19.810+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:20.455 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:20.156+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:20.158296+0000) 2024-04-27T03:36:20.455 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:20.234+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:20.236419+0000) 2024-04-27T03:36:20.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:20.448+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:20.450000+0000) 2024-04-27T03:36:20.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:20.473+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:20.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:20.473+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:20.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:20.473+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:20.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:20.473+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:21.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:21.015+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:21.016892+0000) 2024-04-27T03:36:21.107 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:36:21.018+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67007 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:36:21.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:20.793+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:21.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:20.793+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:21.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:20.793+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:21.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:20.794+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:21.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:21.015+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:21.017083+0000) 2024-04-27T03:36:21.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:20.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:21.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:20.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:21.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:20.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:21.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:20.900+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:21.109 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:21.015+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:21.017702+0000) 2024-04-27T03:36:21.455 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:21.157+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:21.159543+0000) 2024-04-27T03:36:21.456 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:21.235+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:21.237687+0000) 2024-04-27T03:36:21.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:21.448+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:21.450699+0000) 2024-04-27T03:36:21.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:21.479+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:21.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:21.479+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:21.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:21.479+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:21.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:21.479+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:22.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:21.758+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:22.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:21.758+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:22.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:21.758+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:22.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:21.759+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:22.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:21.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:22.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:21.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:22.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:21.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:22.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:21.890+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:22.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:22.160+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:22.162356+0000) 2024-04-27T03:36:22.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:22.238+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:22.240344+0000) 2024-04-27T03:36:22.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:22.452+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:22.454050+0000) 2024-04-27T03:36:22.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:22.524+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:22.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:22.524+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:22.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:22.524+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:22.664 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:22.525+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:22.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:22.738+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:22.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:22.738+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:22.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:22.738+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:22.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:22.740+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:22.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:22.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:22.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:22.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:22.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:22.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:22.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:22.901+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:23.461 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:23.164+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:23.166591+0000) 2024-04-27T03:36:23.462 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:23.242+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:23.244640+0000) 2024-04-27T03:36:23.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:23.454+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:23.456679+0000) 2024-04-27T03:36:23.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:23.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:23.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:23.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:23.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:23.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:23.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:23.550+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:24.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:23.724+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:24.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:23.724+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:24.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:23.724+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:24.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:23.725+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:24.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:23.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:24.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:23.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:24.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:23.874+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:24.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:23.876+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:24.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:24.166+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:24.168774+0000) 2024-04-27T03:36:24.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:24.247+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:24.249422+0000) 2024-04-27T03:36:24.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:24.457+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:24.459294+0000) 2024-04-27T03:36:24.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:24.562+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:24.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:24.562+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:24.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:24.562+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:24.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:24.562+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:24.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:24.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:24.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:24.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:24.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:24.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:24.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:24.712+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:24.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:24.866+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:24.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:24.866+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:24.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:24.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:24.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:24.868+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:25.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:25.169+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:25.170968+0000) 2024-04-27T03:36:25.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:25.252+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:25.254127+0000) 2024-04-27T03:36:25.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:25.460+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:25.462302+0000) 2024-04-27T03:36:25.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:25.553+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:25.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:25.553+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:25.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:25.553+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:25.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:25.554+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:25.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:25.691+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:25.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:25.691+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:25.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:25.691+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:25.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:25.691+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:26.150 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:36:26.026+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67047 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:36:26.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:26.023+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:26.025093+0000) 2024-04-27T03:36:26.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:26.022+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:26.024713+0000) 2024-04-27T03:36:26.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:25.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:26.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:25.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:26.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:25.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:26.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:25.909+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:26.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:26.023+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:26.024849+0000) 2024-04-27T03:36:26.468 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:26.169+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:26.171590+0000) 2024-04-27T03:36:26.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:26.253+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:26.254891+0000) 2024-04-27T03:36:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:26.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:26.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:26.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:26.660+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:26.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:26.461+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:26.463168+0000) 2024-04-27T03:36:26.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:26.588+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:26.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:26.588+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:26.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:26.588+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:26.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:26.589+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:27.178 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:26.896+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:27.179 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:26.896+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:27.179 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:26.896+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:27.179 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:26.896+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:27.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:27.256+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:27.258053+0000) 2024-04-27T03:36:27.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:27.172+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:27.173699+0000) 2024-04-27T03:36:27.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:27.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:27.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:27.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:27.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:27.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:27.619+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:27.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:27.463+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:27.465396+0000) 2024-04-27T03:36:27.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:27.603+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:27.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:27.603+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:27.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:27.603+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:27.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:27.604+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:27.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:27.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:27.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:27.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:27.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:27.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:27.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:27.925+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:28.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:28.175+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:28.176954+0000) 2024-04-27T03:36:28.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:28.259+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:28.261190+0000) 2024-04-27T03:36:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:28.637+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:28.637+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:28.637+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:28.637+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:28.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:28.465+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:28.467773+0000) 2024-04-27T03:36:28.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:28.609+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:28.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:28.609+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:28.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:28.609+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:28.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:28.609+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:29.186 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:28.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:29.186 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:28.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:29.186 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:28.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:29.186 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:28.929+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:29.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:29.180+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:29.181228+0000) 2024-04-27T03:36:29.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:29.263+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:29.263851+0000) 2024-04-27T03:36:29.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:29.469+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:29.470346+0000) 2024-04-27T03:36:29.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:29.636+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:29.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:29.636+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:29.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:29.636+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:29.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:29.636+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:29.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:29.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:29.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:29.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:29.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:29.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:29.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:29.674+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:30.188 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:29.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:30.189 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:29.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:30.189 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:29.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:30.189 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:29.883+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:30.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:30.265+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:30.266133+0000) 2024-04-27T03:36:30.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:30.182+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:30.183528+0000) 2024-04-27T03:36:30.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:30.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:30.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:30.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:30.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:30.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:30.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:30.639+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:30.879 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:30.473+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:30.474563+0000) 2024-04-27T03:36:30.879 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:30.593+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:30.879 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:30.593+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:30.879 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:30.593+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:30.880 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:30.594+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:31.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:31.033+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:31.034238+0000) 2024-04-27T03:36:31.159 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:36:31.036+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67086 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:36:31.160 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:31.033+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:31.034187+0000) 2024-04-27T03:36:31.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:30.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:31.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:30.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:31.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:30.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:31.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:30.874+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:31.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:31.032+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:31.033329+0000) 2024-04-27T03:36:31.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:31.183+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:31.184348+0000) 2024-04-27T03:36:31.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:31.266+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:31.267045+0000) 2024-04-27T03:36:31.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:31.663+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:31.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:31.663+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:31.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:31.663+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:31.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:31.664+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:31.879 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:31.474+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:31.475192+0000) 2024-04-27T03:36:31.880 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:31.593+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:31.880 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:31.593+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:31.880 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:31.593+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:31.880 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:31.594+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:32.190 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:31.872+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:32.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:31.872+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:32.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:31.872+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:32.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:31.873+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:32.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:32.188+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:32.189026+0000) 2024-04-27T03:36:32.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:32.268+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:32.269137+0000) 2024-04-27T03:36:32.773 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:32.477+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:32.478243+0000) 2024-04-27T03:36:32.773 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:32.642+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:32.773 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:32.642+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:32.773 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:32.642+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:32.773 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:32.643+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:32.773 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:32.706+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:32.773 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:32.706+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:32.773 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:32.706+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:32.773 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:32.707+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:33.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:32.922+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:33.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:32.922+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:33.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:32.922+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:33.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:32.922+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:33.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:33.192+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:33.193268+0000) 2024-04-27T03:36:33.307 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:33.272+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:33.273445+0000) 2024-04-27T03:36:33.653 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:33.479+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:33.480416+0000) 2024-04-27T03:36:33.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:33.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:33.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:33.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:33.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:33.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:33.932 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:33.698+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:33.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:33.648+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:33.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:33.648+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:33.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:33.648+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:33.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:33.648+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:34.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:33.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:34.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:33.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:34.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:33.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:34.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:33.927+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:34.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:34.195+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:34.196241+0000) 2024-04-27T03:36:34.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:34.276+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:34.276852+0000) 2024-04-27T03:36:34.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:34.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:34.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:34.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:34.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:34.676+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:34.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:34.677+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:34.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:34.482+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:34.483649+0000) 2024-04-27T03:36:34.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:34.614+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:34.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:34.614+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:34.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:34.614+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:34.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:34.614+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:35.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:34.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:35.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:34.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:35.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:34.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:35.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:34.909+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:35.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:35.197+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:35.198295+0000) 2024-04-27T03:36:35.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:35.278+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:35.279128+0000) 2024-04-27T03:36:35.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:35.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:35.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:35.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:35.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:35.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:35.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:35.656+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:35.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:35.485+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:35.486418+0000) 2024-04-27T03:36:35.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:35.620+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:35.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:35.620+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:35.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:35.620+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:35.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:35.620+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:36.170 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:36:36.044+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67126 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:36:36.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:36.041+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:36.041905+0000) 2024-04-27T03:36:36.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:36.040+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:36.041294+0000) 2024-04-27T03:36:36.171 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:35.924+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:36.171 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:35.924+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:36.171 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:35.924+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:36.172 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:35.924+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:36.172 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:36.040+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:36.041700+0000) 2024-04-27T03:36:36.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:36.198+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:36.198912+0000) 2024-04-27T03:36:36.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:36.279+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:36.279846+0000) 2024-04-27T03:36:36.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:36.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:36.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:36.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:36.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:36.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:36.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:36.635+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:36.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:36.486+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:36.487183+0000) 2024-04-27T03:36:36.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:36.647+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:36.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:36.647+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:36.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:36.647+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:36.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:36.648+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:37.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:36.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:37.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:36.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:37.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:36.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:37.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:36.929+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:37.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:37.200+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:37.200985+0000) 2024-04-27T03:36:37.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:37.283+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:37.284105+0000) 2024-04-27T03:36:37.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:37.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:37.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:37.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:37.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:37.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:37.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:37.607+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:37.818 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:37.489+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:37.490190+0000) 2024-04-27T03:36:37.818 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:37.650+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:37.818 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:37.650+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:37.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:37.650+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:37.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:37.650+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:38.109 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:37.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:38.109 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:37.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:38.109 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:37.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:38.109 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:37.957+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:38.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:38.287+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:38.288734+0000) 2024-04-27T03:36:38.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:38.204+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:38.205261+0000) 2024-04-27T03:36:38.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:38.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:38.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:38.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:38.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:38.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:38.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:38.578+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:38.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:38.492+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:38.493323+0000) 2024-04-27T03:36:38.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:38.618+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:38.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:38.618+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:38.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:38.618+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:38.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:38.618+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:39.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:38.935+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:39.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:38.935+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:39.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:38.935+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:39.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:38.935+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:39.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:39.206+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:39.207358+0000) 2024-04-27T03:36:39.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:39.292+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:39.293032+0000) 2024-04-27T03:36:39.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:39.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:39.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:39.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:39.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:39.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:39.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:39.612+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:39.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:39.494+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:39.495576+0000) 2024-04-27T03:36:39.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:39.612+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:39.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:39.612+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:39.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:39.612+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:39.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:39.613+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:40.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:39.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:40.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:39.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:40.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:39.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:40.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:39.891+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:40.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:40.208+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:40.209413+0000) 2024-04-27T03:36:40.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:40.297+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:40.297892+0000) 2024-04-27T03:36:40.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:40.660+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:40.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:40.660+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:40.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:40.660+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:40.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:40.662+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:40.836 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:40.499+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:40.500356+0000) 2024-04-27T03:36:40.836 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:40.588+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:40.836 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:40.588+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:40.836 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:40.588+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:40.836 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:40.588+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:41.109 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:36:41.053+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67165 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:36:41.110 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:41.049+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:41.050194+0000) 2024-04-27T03:36:41.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:41.049+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:41.050198+0000) 2024-04-27T03:36:41.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:40.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:41.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:40.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:41.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:40.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:41.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:40.941+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:41.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:41.050+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:41.051024+0000) 2024-04-27T03:36:41.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:41.298+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:41.299234+0000) 2024-04-27T03:36:41.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:41.209+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:41.210260+0000) 2024-04-27T03:36:41.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:41.655+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:41.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:41.655+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:41.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:41.655+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:41.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:41.656+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:41.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:41.500+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:41.501747+0000) 2024-04-27T03:36:41.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:41.602+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:41.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:41.602+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:41.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:41.602+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:41.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:41.603+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:42.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:41.958+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:42.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:41.958+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:42.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:41.958+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:42.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:41.959+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:42.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:42.212+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:42.213187+0000) 2024-04-27T03:36:42.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:42.301+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:42.302058+0000) 2024-04-27T03:36:42.861 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:42.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:42.862 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:42.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:42.862 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:42.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:42.862 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:42.699+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:42.862 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:42.503+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:42.504079+0000) 2024-04-27T03:36:42.862 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:42.595+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:42.862 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:42.595+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:42.862 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:42.595+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:42.862 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:42.596+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:43.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:42.945+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:43.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:42.945+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:43.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:42.945+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:43.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:42.946+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:43.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:43.215+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:43.215825+0000) 2024-04-27T03:36:43.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:43.303+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:43.304006+0000) 2024-04-27T03:36:43.874 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:43.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:43.874 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:43.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:43.874 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:43.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:43.874 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:43.683+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:43.874 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:43.507+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:43.507824+0000) 2024-04-27T03:36:43.874 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:43.574+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:43.874 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:43.574+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:43.875 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:43.574+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:43.875 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:43.574+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:44.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:43.986+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:44.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:43.986+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:44.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:43.986+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:44.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:43.987+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:44.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:44.306+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:44.307113+0000) 2024-04-27T03:36:44.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:44.218+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:44.219045+0000) 2024-04-27T03:36:44.904 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:44.511+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:44.512063+0000) 2024-04-27T03:36:44.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:44.593+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:44.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:44.593+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:44.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:44.593+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:44.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:44.594+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:44.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:44.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:44.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:44.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:44.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:44.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:44.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:44.665+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:45.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:45.025+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:45.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:45.025+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:45.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:45.025+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:45.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:45.026+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:45.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:45.221+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:45.221832+0000) 2024-04-27T03:36:45.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:45.308+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:45.309759+0000) 2024-04-27T03:36:45.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:45.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:45.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:45.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:45.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:45.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:45.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:45.646+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:45.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:45.516+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:45.517204+0000) 2024-04-27T03:36:45.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:45.554+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:45.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:45.554+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:45.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:45.554+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:45.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:45.555+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:46.186 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:46.058+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:46.058956+0000) 2024-04-27T03:36:46.186 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:36:46.060+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67205 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:36:46.187 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:46.058+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:46.059148+0000) 2024-04-27T03:36:46.187 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:46.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:46.187 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:46.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:46.187 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:46.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:46.187 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:46.001+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:46.187 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:46.059+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:46.060663+0000) 2024-04-27T03:36:46.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:46.221+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:46.222643+0000) 2024-04-27T03:36:46.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:46.310+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:46.310984+0000) 2024-04-27T03:36:46.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:46.603+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:46.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:46.603+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:46.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:46.603+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:46.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:46.604+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:46.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:46.517+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:46.518375+0000) 2024-04-27T03:36:46.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:46.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:46.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:46.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:46.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:46.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:46.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:46.580+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:47.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:47.020+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:47.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:47.020+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:47.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:47.020+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:47.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:47.021+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:47.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:47.314+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:47.315230+0000) 2024-04-27T03:36:47.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:47.223+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:47.224720+0000) 2024-04-27T03:36:47.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:47.651+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:47.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:47.651+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:47.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:47.651+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:47.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:47.652+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:47.910 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:47.519+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:47.520439+0000) 2024-04-27T03:36:47.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:47.546+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:47.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:47.546+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:47.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:47.546+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:47.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:47.546+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:48.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:47.998+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:48.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:47.998+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:48.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:47.998+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:48.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:47.999+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:48.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:48.319+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:48.319798+0000) 2024-04-27T03:36:48.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:48.226+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:48.226827+0000) 2024-04-27T03:36:48.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:48.623+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:48.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:48.623+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:48.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:48.623+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:48.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:48.625+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:48.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:48.522+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:48.523765+0000) 2024-04-27T03:36:48.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:48.569+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:48.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:48.569+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:48.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:48.569+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:48.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:48.570+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:49.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:49.004+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:49.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:49.004+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:49.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:49.004+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:49.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:49.005+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:49.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:49.321+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:49.322527+0000) 2024-04-27T03:36:49.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:49.228+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:49.228926+0000) 2024-04-27T03:36:49.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T03:36:49.610+0000 7ff0e4ed9700 0 monclient(hunting): authenticate timed out after 300 2024-04-27T03:36:49.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:49.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:49.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:49.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:49.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:49.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:49.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:49.623+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:49.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:49.525+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:49.525864+0000) 2024-04-27T03:36:49.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:49.613+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:49.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:49.613+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:49.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:49.613+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:49.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:49.614+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:50.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:50.032+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:50.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:50.032+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:50.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:50.032+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:50.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:50.033+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:50.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:50.230+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:50.231010+0000) 2024-04-27T03:36:50.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:50.326+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:50.327281+0000) 2024-04-27T03:36:50.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:50.671+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:50.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:50.671+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:50.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:50.671+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:50.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:50.672+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:50.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:50.527+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:50.527991+0000) 2024-04-27T03:36:50.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:50.589+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:50.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:50.589+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:50.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:50.589+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:50.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:50.590+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:51.193 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:36:51.069+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67244 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:36:51.193 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:51.066+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:51.066972+0000) 2024-04-27T03:36:51.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:51.066+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:51.066884+0000) 2024-04-27T03:36:51.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:51.066+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:51.066911+0000) 2024-04-27T03:36:51.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:51.077+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:51.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:51.077+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:51.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:51.077+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:51.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:51.078+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:51.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:51.231+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:51.232435+0000) 2024-04-27T03:36:51.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:51.327+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:51.327947+0000) 2024-04-27T03:36:51.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:51.630+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:51.914 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:51.630+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:51.914 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:51.630+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:51.914 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:51.631+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:51.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:51.528+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:51.528849+0000) 2024-04-27T03:36:51.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:51.621+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:51.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:51.621+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:51.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:51.621+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:51.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:51.622+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:52.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:52.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:52.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:52.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:52.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:52.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:52.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:52.093+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:52.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:52.329+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:52.330043+0000) 2024-04-27T03:36:52.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:52.234+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:52.235303+0000) 2024-04-27T03:36:52.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:52.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:52.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:52.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:52.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:52.654+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:52.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:52.655+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:52.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:52.531+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:52.532094+0000) 2024-04-27T03:36:52.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:52.580+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:52.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:52.580+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:52.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:52.580+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:52.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:52.580+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:53.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:53.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:53.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:53.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:53.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:53.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:53.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:53.097+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:53.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:53.237+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:53.238256+0000) 2024-04-27T03:36:53.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:53.332+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:53.332923+0000) 2024-04-27T03:36:53.874 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:53.688+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:53.874 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:53.688+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:53.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:53.688+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:53.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:53.688+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:53.875 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:53.533+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:53.534470+0000) 2024-04-27T03:36:53.875 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:53.618+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:53.875 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:53.618+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:53.875 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:53.618+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:53.875 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:53.619+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:54.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:54.053+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:54.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:54.053+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:54.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:54.053+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:54.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:54.053+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:54.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:54.240+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:54.240914+0000) 2024-04-27T03:36:54.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:54.335+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:54.336092+0000) 2024-04-27T03:36:54.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:54.696+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:54.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:54.696+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:54.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:54.696+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:54.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:54.697+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:54.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:54.536+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:54.536854+0000) 2024-04-27T03:36:54.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:54.582+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:54.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:54.582+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:54.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:54.582+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:54.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:54.582+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:55.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:55.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:55.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:55.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:55.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:55.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:55.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:55.085+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:55.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:55.243+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:55.244116+0000) 2024-04-27T03:36:55.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:55.337+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:55.338768+0000) 2024-04-27T03:36:55.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:55.714+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:55.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:55.714+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:55.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:55.714+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:55.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:55.715+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:55.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:55.539+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:55.540327+0000) 2024-04-27T03:36:55.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:55.617+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:55.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:55.617+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:55.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:55.617+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:55.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:55.618+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:56.193 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:36:56.077+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67284 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:36:56.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:56.074+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:56.075449+0000) 2024-04-27T03:36:56.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:56.074+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:56.074866+0000) 2024-04-27T03:36:56.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:56.074+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:56.075071+0000) 2024-04-27T03:36:56.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:56.099+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:56.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:56.099+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:56.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:56.099+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:56.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:56.100+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:56.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:56.244+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:56.244998+0000) 2024-04-27T03:36:56.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:56.339+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:56.339993+0000) 2024-04-27T03:36:56.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:56.540+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:56.541041+0000) 2024-04-27T03:36:56.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:56.660+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:56.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:56.660+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:56.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:56.660+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:56.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:56.660+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:56.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:56.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:56.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:56.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:56.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:56.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:56.921 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:56.685+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:57.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:57.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:57.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:57.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:57.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:57.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:57.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:57.060+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:57.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:57.246+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:57.247093+0000) 2024-04-27T03:36:57.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:57.341+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:57.342074+0000) 2024-04-27T03:36:57.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:57.637+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:57.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:57.637+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:57.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:57.637+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:57.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:57.638+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:57.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:57.545+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:57.546045+0000) 2024-04-27T03:36:57.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:57.681+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:57.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:57.681+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:57.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:57.681+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:57.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:57.681+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:58.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:58.034+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:58.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:58.034+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:58.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:58.034+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:58.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:58.035+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:58.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:58.343+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:58.344188+0000) 2024-04-27T03:36:58.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:58.249+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:58.250199+0000) 2024-04-27T03:36:58.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:58.657+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:58.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:58.657+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:58.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:58.657+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:58.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:58.657+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:58.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:58.548+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:58.549495+0000) 2024-04-27T03:36:58.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:58.674+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:58.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:58.674+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:58.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:58.674+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:58.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:58.674+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:59.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:58.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:59.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:58.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:59.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:58.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:36:59.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:58.991+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:59.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:36:59.346+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:59.346846+0000) 2024-04-27T03:36:59.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:59.253+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:59.254452+0000) 2024-04-27T03:36:59.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:59.705+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:59.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:59.705+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:59.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:59.705+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:36:59.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:36:59.706+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:36:59.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:59.550+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:36:59.551769+0000) 2024-04-27T03:36:59.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:59.713+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:59.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:59.713+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:59.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:59.713+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:36:59.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:36:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:36:59.713+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:00.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:00.020+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:00.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:00.020+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:00.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:00.020+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:00.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:00.021+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:00.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:00.255+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:00.256513+0000) 2024-04-27T03:37:00.498 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:00.349+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:00.350102+0000) 2024-04-27T03:37:00.747 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:00.671+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:00.747 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:00.671+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:00.747 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:00.671+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:00.747 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:00.671+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:00.747 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:00.553+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:00.554025+0000) 2024-04-27T03:37:01.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:00.737+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:01.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:00.737+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:01.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:00.737+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:01.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:00.737+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:01.356 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:37:01.085+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67323 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:37:01.356 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:01.081+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:01.082661+0000) 2024-04-27T03:37:01.356 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:01.256+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:01.257342+0000) 2024-04-27T03:37:01.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:01.081+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:01.082129+0000) 2024-04-27T03:37:01.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:01.044+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:01.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:01.044+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:01.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:01.044+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:01.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:01.046+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:01.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:01.082+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:01.083416+0000) 2024-04-27T03:37:01.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:01.350+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:01.351179+0000) 2024-04-27T03:37:01.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:01.553+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:01.554722+0000) 2024-04-27T03:37:01.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:01.763+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:01.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:01.763+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:01.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:01.763+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:01.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:01.764+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:01.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:01.708+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:01.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:01.708+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:01.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:01.708+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:01.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:01.708+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:02.264 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:02.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:02.264 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:02.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:02.264 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:02.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:02.264 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:02.001+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:02.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:02.258+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:02.259265+0000) 2024-04-27T03:37:02.562 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:02.353+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:02.354227+0000) 2024-04-27T03:37:02.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:02.716+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:02.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:02.716+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:02.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:02.716+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:02.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:02.717+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:02.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:02.556+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:02.556946+0000) 2024-04-27T03:37:02.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:02.728+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:02.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:02.728+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:02.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:02.728+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:02.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:02.729+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:03.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:02.970+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:03.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:02.970+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:03.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:02.970+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:03.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:02.970+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:03.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:03.260+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:03.261578+0000) 2024-04-27T03:37:03.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:03.358+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:03.359261+0000) 2024-04-27T03:37:03.874 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:03.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:03.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:03.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:03.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:03.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:03.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:03.713+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:03.875 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:03.559+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:03.559844+0000) 2024-04-27T03:37:03.875 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:03.680+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:03.875 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:03.680+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:03.875 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:03.680+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:03.875 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:03.680+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:04.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:03.936+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:04.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:03.936+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:04.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:03.936+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:04.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:03.938+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:04.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:04.263+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:04.263904+0000) 2024-04-27T03:37:04.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:04.360+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:04.361490+0000) 2024-04-27T03:37:04.743 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:04.561+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:04.562151+0000) 2024-04-27T03:37:04.743 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:04.632+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:04.743 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:04.632+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:04.743 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:04.632+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:04.743 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:04.632+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:05.051 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:04.737+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:05.051 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:04.737+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:05.051 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:04.737+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:05.051 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:04.737+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:05.051 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:04.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:05.051 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:04.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:05.051 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:04.911+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:05.051 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:04.912+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:05.371 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:05.265+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:05.266061+0000) 2024-04-27T03:37:05.666 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:05.564+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:05.564823+0000) 2024-04-27T03:37:05.666 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:05.365+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:05.366271+0000) 2024-04-27T03:37:05.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:05.758+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:05.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:05.758+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:05.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:05.758+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:05.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:05.760+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:05.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:05.660+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:05.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:05.660+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:05.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:05.660+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:05.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:05.660+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:06.194 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:37:06.093+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67363 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:37:06.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:06.089+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:06.090600+0000) 2024-04-27T03:37:06.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:06.090+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:06.090900+0000) 2024-04-27T03:37:06.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:05.922+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:06.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:05.922+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:06.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:05.922+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:06.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:05.922+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:06.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:06.090+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:06.091738+0000) 2024-04-27T03:37:06.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:06.266+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:06.266896+0000) 2024-04-27T03:37:06.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:06.366+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:06.367132+0000) 2024-04-27T03:37:06.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:06.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:06.884 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:06.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:06.884 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:06.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:06.884 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:06.720+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:06.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:06.565+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:06.566068+0000) 2024-04-27T03:37:06.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:06.668+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:06.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:06.668+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:06.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:06.668+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:06.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:06.668+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:07.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:06.877+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:07.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:06.877+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:07.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:06.877+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:07.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:06.878+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:07.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:07.268+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:07.268936+0000) 2024-04-27T03:37:07.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:07.368+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:07.369778+0000) 2024-04-27T03:37:07.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:07.679+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:07.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:07.679+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:07.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:07.679+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:07.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:07.680+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:07.893 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:07.568+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:07.569149+0000) 2024-04-27T03:37:07.893 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:07.697+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:07.893 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:07.697+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:07.893 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:07.697+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:07.893 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:07.698+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:08.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:07.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:08.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:07.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:08.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:07.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:08.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:07.888+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:08.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:08.373+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:08.374080+0000) 2024-04-27T03:37:08.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:08.270+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:08.271016+0000) 2024-04-27T03:37:08.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:08.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:08.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:08.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:08.935 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:08.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:08.935 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:08.633+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:08.935 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:08.572+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:08.573413+0000) 2024-04-27T03:37:08.935 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:08.676+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:08.935 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:08.676+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:08.935 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:08.676+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:08.935 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:08.676+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:09.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:08.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:09.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:08.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:09.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:08.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:09.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:08.929+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:09.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:09.272+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:09.273720+0000) 2024-04-27T03:37:09.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:09.376+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:09.377183+0000) 2024-04-27T03:37:09.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:09.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:09.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:09.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:09.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:09.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:09.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:09.668+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:09.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:09.574+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:09.575628+0000) 2024-04-27T03:37:09.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:09.706+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:09.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:09.706+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:09.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:09.706+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:09.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:09.707+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:10.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:09.918+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:10.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:09.918+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:10.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:09.918+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:10.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:09.920+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:10.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:10.275+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:10.275907+0000) 2024-04-27T03:37:10.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:10.378+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:10.379314+0000) 2024-04-27T03:37:10.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:10.578+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:10.578826+0000) 2024-04-27T03:37:10.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:10.716+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:10.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:10.716+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:10.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:10.716+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:10.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:10.716+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:10.939 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:10.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:10.939 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:10.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:10.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:10.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:10.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:10.621+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:11.197 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:37:11.102+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67402 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:37:11.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:11.097+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:11.098576+0000) 2024-04-27T03:37:11.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:11.098+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:11.099260+0000) 2024-04-27T03:37:11.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:10.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:11.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:10.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:11.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:10.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:11.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:10.957+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:11.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:11.098+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:11.099436+0000) 2024-04-27T03:37:11.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:11.275+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:11.276731+0000) 2024-04-27T03:37:11.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:11.379+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:11.380763+0000) 2024-04-27T03:37:11.752 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:11.578+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:11.579502+0000) 2024-04-27T03:37:11.752 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:11.626+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:11.753 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:11.626+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:11.753 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:11.626+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:11.753 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:11.626+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:12.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:11.945+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:12.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:11.946+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:12.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:11.946+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:12.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:11.946+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:12.049 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:11.746+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:12.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:11.746+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:12.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:11.746+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:12.050 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:11.747+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:12.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:12.280+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:12.281775+0000) 2024-04-27T03:37:12.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:12.581+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:12.582300+0000) 2024-04-27T03:37:12.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:12.382+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:12.383016+0000) 2024-04-27T03:37:12.941 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:12.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:12.941 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:12.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:12.941 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:12.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:12.942 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:12.665+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:12.942 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:12.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:12.942 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:12.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:12.942 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:12.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:12.942 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:12.709+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:13.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:12.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:13.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:12.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:13.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:12.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:13.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:12.997+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:13.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:13.284+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:13.285018+0000) 2024-04-27T03:37:13.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:13.384+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:13.385225+0000) 2024-04-27T03:37:13.764 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:13.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:13.764 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:13.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:13.764 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:13.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:13.764 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:13.628+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:13.765 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:13.585+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:13.586541+0000) 2024-04-27T03:37:13.765 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:13.758+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:13.765 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:13.758+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:13.765 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:13.758+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:14.042 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:13.759+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:14.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:14.036+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:14.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:14.036+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:14.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:14.036+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:14.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:14.037+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:14.596 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:14.286+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:14.287669+0000) 2024-04-27T03:37:14.597 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:14.387+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:14.387890+0000) 2024-04-27T03:37:14.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:14.652+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:14.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:14.652+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:14.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:14.652+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:14.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:14.653+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:14.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:14.590+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:14.591672+0000) 2024-04-27T03:37:14.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:14.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:14.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:14.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:14.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:14.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:14.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:14.755+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:15.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:15.008+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:15.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:15.008+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:15.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:15.008+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:15.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:15.008+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:15.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:15.290+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:15.291759+0000) 2024-04-27T03:37:15.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:15.389+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:15.390162+0000) 2024-04-27T03:37:15.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:15.687+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:15.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:15.687+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:15.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:15.687+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:15.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:15.688+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:15.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:15.595+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:15.596646+0000) 2024-04-27T03:37:15.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:15.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:15.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:15.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:15.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:15.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:15.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:15.712+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:16.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:16.106+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:16.107534+0000) 2024-04-27T03:37:16.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:16.107+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:16.108287+0000) 2024-04-27T03:37:16.196 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:37:16.110+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67442 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:37:16.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:15.997+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:16.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:15.997+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:16.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:15.997+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:16.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:15.998+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:16.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:16.106+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:16.107706+0000) 2024-04-27T03:37:16.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:16.291+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:16.292459+0000) 2024-04-27T03:37:16.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:16.390+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:16.390846+0000) 2024-04-27T03:37:16.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:16.694+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:16.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:16.694+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:16.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:16.694+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:16.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:16.695+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:16.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:16.596+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:16.597554+0000) 2024-04-27T03:37:16.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:16.688+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:16.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:16.688+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:16.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:16.688+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:16.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:16.688+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:17.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:16.965+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:17.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:16.965+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:17.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:16.965+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:17.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:16.966+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:17.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:17.294+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:17.295306+0000) 2024-04-27T03:37:17.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:17.392+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:17.392894+0000) 2024-04-27T03:37:17.935 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:17.699+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:17.935 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:17.699+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:17.935 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:17.699+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:17.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:17.701+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:17.936 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:17.598+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:17.599710+0000) 2024-04-27T03:37:17.936 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:17.718+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:17.936 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:17.718+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:17.936 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:17.718+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:17.936 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:17.719+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:18.189 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:17.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:18.189 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:17.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:18.189 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:17.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:18.189 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:17.930+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:18.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:18.298+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:18.299592+0000) 2024-04-27T03:37:18.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:18.394+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:18.395737+0000) 2024-04-27T03:37:18.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:18.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:18.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:18.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:18.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:18.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:18.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:18.711+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:18.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:18.601+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:18.601936+0000) 2024-04-27T03:37:18.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:18.676+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:18.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:18.676+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:18.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:18.676+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:18.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:18.676+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:19.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:18.897+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:19.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:18.897+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:19.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:18.897+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:19.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:18.899+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:19.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:19.398+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:19.400153+0000) 2024-04-27T03:37:19.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:19.299+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:19.301719+0000) 2024-04-27T03:37:19.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:19.694+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:19.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:19.694+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:19.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:19.694+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:19.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:19.695+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:19.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:19.603+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:19.605162+0000) 2024-04-27T03:37:19.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:19.632+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:19.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:19.632+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:19.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:19.632+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:19.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:19.632+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:20.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:19.945+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:20.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:19.945+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:20.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:19.945+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:20.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:19.945+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:20.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:20.303+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:20.304958+0000) 2024-04-27T03:37:20.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:20.402+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:20.404493+0000) 2024-04-27T03:37:20.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:20.716+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:20.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:20.716+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:20.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:20.716+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:20.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:20.717+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:20.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:20.606+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:20.607821+0000) 2024-04-27T03:37:20.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:20.646+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:20.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:20.646+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:20.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:20.646+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:20.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:20.647+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:21.226 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:37:21.117+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67481 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:37:21.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:21.114+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:21.116437+0000) 2024-04-27T03:37:21.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:21.115+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:21.117121+0000) 2024-04-27T03:37:21.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:20.966+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:21.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:20.966+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:21.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:20.966+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:21.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:20.967+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:21.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:21.114+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:21.116405+0000) 2024-04-27T03:37:21.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:21.303+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:21.305660+0000) 2024-04-27T03:37:21.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:21.403+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:21.405397+0000) 2024-04-27T03:37:21.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:21.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:21.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:21.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:21.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:21.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:21.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:21.721+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:21.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:21.607+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:21.608825+0000) 2024-04-27T03:37:21.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:21.653+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:21.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:21.653+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:21.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:21.653+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:21.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:21.653+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:22.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:22.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:22.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:22.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:22.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:22.000+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:22.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:22.001+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:22.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:22.307+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:22.308821+0000) 2024-04-27T03:37:22.618 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:22.407+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:22.409634+0000) 2024-04-27T03:37:22.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:22.611+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:22.613320+0000) 2024-04-27T03:37:22.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:22.691+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:22.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:22.691+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:22.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:22.691+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:22.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:22.692+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:22.884 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:22.694+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:22.885 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:22.694+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:22.885 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:22.694+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:22.885 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:22.695+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:23.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:23.005+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:23.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:23.005+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:23.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:23.005+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:23.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:23.006+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:23.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:23.309+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:23.311491+0000) 2024-04-27T03:37:23.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:23.410+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:23.412438+0000) 2024-04-27T03:37:23.876 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:23.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:23.876 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:23.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:23.876 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:23.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:23.876 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:23.664+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:23.876 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:23.614+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:23.616228+0000) 2024-04-27T03:37:23.876 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:23.671+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:23.876 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:23.671+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:23.877 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:23.671+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:23.877 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:23.671+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:24.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:24.012+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:24.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:24.012+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:24.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:24.012+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:24.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:24.012+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:24.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:24.313+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:24.315776+0000) 2024-04-27T03:37:24.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:24.413+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:24.415021+0000) 2024-04-27T03:37:24.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:24.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:24.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:24.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:24.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:24.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:24.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:24.682+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:24.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:24.617+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:24.618905+0000) 2024-04-27T03:37:24.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:24.684+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:24.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:24.684+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:24.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:24.684+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:24.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:24.684+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:25.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:25.017+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:25.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:25.017+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:25.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:25.018+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:25.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:25.018+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:25.627 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:25.316+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:25.318512+0000) 2024-04-27T03:37:25.627 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:25.416+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:25.418181+0000) 2024-04-27T03:37:25.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:25.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:25.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:25.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:25.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:25.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:25.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:25.638+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:25.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:25.620+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:25.622226+0000) 2024-04-27T03:37:25.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:25.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:25.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:25.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:25.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:25.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:25.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:25.711+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:26.226 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:37:26.126+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67521 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:37:26.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:26.121+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:26.123517+0000) 2024-04-27T03:37:26.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:26.121+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:26.123549+0000) 2024-04-27T03:37:26.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:25.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:26.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:25.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:26.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:25.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:26.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:25.978+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:26.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:26.121+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:26.123226+0000) 2024-04-27T03:37:26.482 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:26.318+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:26.319963+0000) 2024-04-27T03:37:26.482 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:26.417+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:26.418900+0000) 2024-04-27T03:37:26.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:26.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:26.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:26.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:26.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:26.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:26.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:26.610+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:26.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:26.621+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:26.623187+0000) 2024-04-27T03:37:26.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:26.672+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:26.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:26.672+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:26.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:26.672+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:26.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:26.673+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:27.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:26.959+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:27.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:26.959+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:27.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:26.959+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:27.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:26.961+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:27.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:27.320+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:27.321991+0000) 2024-04-27T03:37:27.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:27.421+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:27.423174+0000) 2024-04-27T03:37:27.937 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:27.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:27.937 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:27.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:27.937 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:27.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:27.937 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:27.566+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:27.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:27.623+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:27.625422+0000) 2024-04-27T03:37:27.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:27.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:27.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:27.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:27.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:27.669+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:27.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:27.669+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:28.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:27.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:28.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:27.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:28.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:27.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:28.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:27.931+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:28.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:28.322+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:28.324666+0000) 2024-04-27T03:37:28.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:28.425+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:28.427503+0000) 2024-04-27T03:37:28.923 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:28.916+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:28.923 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:28.916+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:28.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:28.588+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:28.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:28.588+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:28.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:28.588+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:28.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:28.589+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:28.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:28.625+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:28.627723+0000) 2024-04-27T03:37:28.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:28.668+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:28.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:28.668+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:28.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:28.668+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:28.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:28.668+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:29.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:28.916+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:29.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:28.917+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:29.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:29.326+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:29.327902+0000) 2024-04-27T03:37:29.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:29.429+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:29.431654+0000) 2024-04-27T03:37:29.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:29.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:29.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:29.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:29.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:29.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:29.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:29.571+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:29.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:29.630+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:29.632039+0000) 2024-04-27T03:37:29.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:29.685+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:29.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:29.685+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:29.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:29.685+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:29.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:29.686+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:29.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:29.879+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:29.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:29.879+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:29.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:29.879+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:30.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:29.880+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:30.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:30.331+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:30.332904+0000) 2024-04-27T03:37:30.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:30.435+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:30.436738+0000) 2024-04-27T03:37:30.847 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:30.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:30.847 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:30.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:30.847 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:30.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:30.847 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:30.576+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:30.847 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:30.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:30.848 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:30.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:30.848 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:30.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:30.848 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:30.635+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:30.637014+0000) 2024-04-27T03:37:30.848 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:30.657+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:30.848 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:30.657+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:30.848 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:30.657+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:30.848 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:30.658+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:31.115 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:30.842+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:31.442 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:37:31.135+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67560 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:37:31.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:31.131+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:31.132671+0000) 2024-04-27T03:37:31.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:31.131+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:31.132123+0000) 2024-04-27T03:37:31.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:31.132+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:31.133774+0000) 2024-04-27T03:37:31.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:31.332+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:31.333760+0000) 2024-04-27T03:37:31.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:31.436+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:31.437700+0000) 2024-04-27T03:37:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:31.594+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:31.594+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:31.594+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:31.596+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:31.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:31.637+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:31.637810+0000) 2024-04-27T03:37:31.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:31.643+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:31.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:31.643+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:31.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:31.643+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:31.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:31.643+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:32.115 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:31.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:32.115 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:31.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:32.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:31.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:32.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:31.816+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:32.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:32.335+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:32.336024+0000) 2024-04-27T03:37:32.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:32.439+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:32.440182+0000) 2024-04-27T03:37:32.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:32.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:32.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:32.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:32.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:32.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:32.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:32.569+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:32.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:32.641+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:32.641986+0000) 2024-04-27T03:37:32.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:32.692+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:32.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:32.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:32.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:32.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:32.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:32.693+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:32.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:32.855+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:32.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:32.855+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:32.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:32.855+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:32.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:32.857+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:33.597 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:33.338+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:33.339151+0000) 2024-04-27T03:37:33.598 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:33.441+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:33.442567+0000) 2024-04-27T03:37:33.876 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:33.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:33.876 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:33.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:33.876 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:33.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:33.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:33.592+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:33.877 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:33.643+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:33.644350+0000) 2024-04-27T03:37:33.877 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:33.707+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:33.877 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:33.707+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:33.877 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:33.707+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:33.877 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:33.708+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:33.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:33.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:33.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:33.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:33.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:33.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:33.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:33.813+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:34.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:34.343+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:34.344478+0000) 2024-04-27T03:37:34.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:34.444+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:34.444815+0000) 2024-04-27T03:37:34.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:34.648+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:34.649374+0000) 2024-04-27T03:37:34.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:34.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:34.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:34.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:34.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:34.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:34.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:34.612+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:35.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:34.726+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:35.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:34.726+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:35.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:34.726+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:35.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:34.726+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:35.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:34.843+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:35.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:34.843+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:35.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:34.843+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:35.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:34.844+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:35.606 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:35.348+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:35.349660+0000) 2024-04-27T03:37:35.606 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:35.446+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:35.446961+0000) 2024-04-27T03:37:35.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:35.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:35.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:35.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:35.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:35.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:35.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:35.805+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:35.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:35.600+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:35.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:35.600+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:35.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:35.600+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:35.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:35.601+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:35.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:35.650+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:35.651520+0000) 2024-04-27T03:37:35.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:35.737+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:35.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:35.737+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:35.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:35.737+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:35.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:35.738+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:36.257 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:36.139+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:36.140431+0000) 2024-04-27T03:37:36.258 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:36.139+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:36.140246+0000) 2024-04-27T03:37:36.259 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:37:36.143+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67600 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:37:36.259 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:36.140+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:36.140885+0000) 2024-04-27T03:37:36.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:36.350+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:36.350974+0000) 2024-04-27T03:37:36.582 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:36.446+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:36.447706+0000) 2024-04-27T03:37:36.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:36.576+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:36.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:36.576+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:36.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:36.576+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:36.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:36.576+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:36.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:36.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:36.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:36.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:36.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:36.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:36.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:36.820+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:36.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:36.652+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:36.652848+0000) 2024-04-27T03:37:36.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:36.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:36.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:36.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:36.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:36.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:36.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:36.711+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:37.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:37.354+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:37.355149+0000) 2024-04-27T03:37:37.544 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:37.449+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:37.449879+0000) 2024-04-27T03:37:37.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:37.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:37.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:37.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:37.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:37.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:37.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:37.539+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:37.843 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:37.656+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:37.657596+0000) 2024-04-27T03:37:37.843 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:37.720+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:37.843 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:37.720+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:37.844 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:37.720+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:37.844 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:37.720+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:38.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:37.836+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:38.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:37.836+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:38.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:37.836+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:38.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:37.838+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:38.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:38.356+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:38.357231+0000) 2024-04-27T03:37:38.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:38.453+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:38.454234+0000) 2024-04-27T03:37:38.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:38.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:38.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:38.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:38.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:38.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:38.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:38.495+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:38.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:38.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:38.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:38.842+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:38.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:38.661+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:38.662360+0000) 2024-04-27T03:37:38.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:38.771+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:38.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:38.771+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:38.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:38.771+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:38.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:38.771+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:39.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:39.455+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:39.456618+0000) 2024-04-27T03:37:39.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:39.358+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:39.359333+0000) 2024-04-27T03:37:39.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:39.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:39.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:39.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:39.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:39.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:39.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:39.496+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:39.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:39.664+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:39.664822+0000) 2024-04-27T03:37:39.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:39.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:39.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:39.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:39.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:39.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:39.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:39.821+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:39.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:39.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:39.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:39.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:39.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:39.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:39.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:39.884+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:40.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:40.458+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:40.458849+0000) 2024-04-27T03:37:40.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:40.360+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:40.361532+0000) 2024-04-27T03:37:40.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:40.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:40.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:40.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:40.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:40.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:40.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:40.448+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:40.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:40.859+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:40.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:40.859+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:40.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:40.859+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:40.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:40.859+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:40.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:40.667+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:40.668098+0000) 2024-04-27T03:37:40.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:40.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:40.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:40.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:40.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:40.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:40.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:40.853+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:41.258 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:37:41.151+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67639 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:37:41.258 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:41.148+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:41.149303+0000) 2024-04-27T03:37:41.258 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:41.147+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:41.148504+0000) 2024-04-27T03:37:41.259 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:41.149+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:41.149855+0000) 2024-04-27T03:37:41.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:41.458+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:41.459729+0000) 2024-04-27T03:37:41.674 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:41.362+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:41.362939+0000) 2024-04-27T03:37:41.674 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:41.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:41.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:41.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:41.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:41.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:41.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:41.488+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:41.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:41.852+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:41.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:41.852+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:41.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:41.852+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:41.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:41.853+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:41.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:41.668+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:41.669097+0000) 2024-04-27T03:37:41.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:41.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:41.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:41.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:41.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:41.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:41.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:41.846+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:42.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:42.461+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:42.461903+0000) 2024-04-27T03:37:42.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:42.364+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:42.365176+0000) 2024-04-27T03:37:42.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:42.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:42.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:42.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:42.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:42.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:42.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:42.441+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:42.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:42.670+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:42.671542+0000) 2024-04-27T03:37:42.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:42.840+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:42.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:42.840+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:42.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:42.840+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:42.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:42.840+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:42.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:42.880+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:42.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:42.880+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:42.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:42.880+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:42.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:42.881+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:43.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:43.465+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:43.466233+0000) 2024-04-27T03:37:43.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:43.368+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:43.369383+0000) 2024-04-27T03:37:43.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:43.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:43.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:43.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:43.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:43.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:43.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:43.455+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:43.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:43.917+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:43.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:43.917+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:43.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:43.917+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:43.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:43.918+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:43.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:43.675+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:43.676626+0000) 2024-04-27T03:37:43.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:43.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:43.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:43.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:43.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:43.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:43.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:43.872+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:44.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:44.467+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:44.468574+0000) 2024-04-27T03:37:44.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:44.373+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:44.374026+0000) 2024-04-27T03:37:44.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:44.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:44.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:44.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:44.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:44.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:44.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:44.426+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:44.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:44.912+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:44.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:44.912+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:44.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:44.912+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:44.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:44.913+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:44.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:44.679+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:44.680007+0000) 2024-04-27T03:37:44.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:44.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:44.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:44.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:44.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:44.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:44.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:44.916+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:45.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:45.470+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:45.470843+0000) 2024-04-27T03:37:45.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:45.375+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:45.376730+0000) 2024-04-27T03:37:45.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:45.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:45.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:45.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:45.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:45.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:45.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:45.429+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:45.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:45.868+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:45.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:45.868+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:45.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:45.868+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:45.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:45.869+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:45.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:45.682+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:45.683516+0000) 2024-04-27T03:37:45.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:45.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:45.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:45.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:45.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:45.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:45.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:45.868+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:46.258 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:37:46.160+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67679 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:37:46.259 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:46.157+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:46.157840+0000) 2024-04-27T03:37:46.259 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:46.156+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:46.157785+0000) 2024-04-27T03:37:46.259 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:46.156+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:46.157056+0000) 2024-04-27T03:37:46.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:46.471+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:46.472158+0000) 2024-04-27T03:37:46.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:46.377+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:46.378172+0000) 2024-04-27T03:37:46.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:46.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:46.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:46.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:46.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:46.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:46.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:46.403+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:46.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:46.919+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:46.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:46.919+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:46.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:46.919+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:46.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:46.920+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:46.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:46.684+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:46.685217+0000) 2024-04-27T03:37:46.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:46.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:46.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:46.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:46.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:46.913+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:46.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:46.914+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:47.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:47.361+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:47.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:47.361+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:47.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:47.361+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:47.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:47.362+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:47.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:47.380+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:47.381166+0000) 2024-04-27T03:37:47.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:47.473+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:47.474221+0000) 2024-04-27T03:37:47.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:47.687+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:47.688287+0000) 2024-04-27T03:37:47.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:47.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:47.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:47.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:47.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:47.868+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:47.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:47.869+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:47.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:47.930+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:47.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:47.930+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:47.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:47.930+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:47.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:47.931+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:48.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:48.476+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:48.477229+0000) 2024-04-27T03:37:48.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:48.382+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:48.383583+0000) 2024-04-27T03:37:48.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:48.398+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:48.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:48.398+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:48.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:48.398+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:48.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:48.399+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:48.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:48.690+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:48.690843+0000) 2024-04-27T03:37:48.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:48.837+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:48.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:48.837+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:48.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:48.837+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:48.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:48.838+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:48.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:48.900+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:48.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:48.900+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:48.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:48.900+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:48.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:48.901+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:49.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:49.478+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:49.479543+0000) 2024-04-27T03:37:49.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:49.385+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:49.385843+0000) 2024-04-27T03:37:49.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:49.398+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:49.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:49.398+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:49.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:49.398+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:49.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:49.398+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:49.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:49.694+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:49.695699+0000) 2024-04-27T03:37:49.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:49.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:49.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:49.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:49.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:49.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:49.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:49.870+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:49.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:49.883+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:49.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:49.883+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:49.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:49.883+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:49.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:49.883+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:50.704 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:50.483+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:50.483841+0000) 2024-04-27T03:37:50.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:50.388+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:50.389060+0000) 2024-04-27T03:37:50.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:50.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:50.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:50.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:50.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:50.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:50.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:50.437+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:50.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:50.871+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:50.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:50.871+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:50.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:50.871+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:50.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:50.872+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:50.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:50.697+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:50.698790+0000) 2024-04-27T03:37:50.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:50.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:50.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:50.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:50.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:50.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:50.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:50.910+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:51.259 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:37:51.168+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67718 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:37:51.259 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:51.165+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:51.166226+0000) 2024-04-27T03:37:51.259 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:51.164+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:51.165162+0000) 2024-04-27T03:37:51.260 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:51.164+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:51.165038+0000) 2024-04-27T03:37:51.705 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:51.483+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:51.484634+0000) 2024-04-27T03:37:51.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:51.389+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:51.390257+0000) 2024-04-27T03:37:51.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:51.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:51.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:51.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:51.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:51.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:51.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:51.466+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:51.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:51.699+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:51.700031+0000) 2024-04-27T03:37:51.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:51.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:51.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:51.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:51.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:51.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:51.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:51.868+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:52.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:51.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:52.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:51.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:52.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:51.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:52.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:51.954+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:52.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:52.486+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:52.487404+0000) 2024-04-27T03:37:52.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:52.391+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:52.392361+0000) 2024-04-27T03:37:52.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:52.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:52.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:52.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:52.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:52.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:52.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:52.431+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:52.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:52.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:52.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:52.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:52.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:52.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:52.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:52.852+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:52.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:52.704+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:52.705111+0000) 2024-04-27T03:37:53.259 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:52.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:53.259 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:52.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:53.259 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:52.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:53.259 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:52.994+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:53.543 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:53.488+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:53.489640+0000) 2024-04-27T03:37:53.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:53.393+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:53.394632+0000) 2024-04-27T03:37:53.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:53.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:53.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:53.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:53.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:53.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:53.544 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:53.476+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:53.824 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:53.709+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:53.710120+0000) 2024-04-27T03:37:54.118 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:54.035+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:54.118 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:54.035+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:54.118 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:54.035+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:54.118 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:54.036+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:54.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:53.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:54.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:53.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:54.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:53.818+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:54.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:53.819+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:54.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:54.490+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:54.491719+0000) 2024-04-27T03:37:54.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:54.396+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:54.396885+0000) 2024-04-27T03:37:54.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:54.473+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:54.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:54.473+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:54.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:54.473+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:54.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:54.474+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:54.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:54.711+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:54.712425+0000) 2024-04-27T03:37:54.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:54.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:54.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:54.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:54.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:54.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:54.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:54.770+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:55.259 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:55.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:55.259 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:55.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:55.259 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:55.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:55.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:55.073+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:55.516 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:55.494+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:55.494985+0000) 2024-04-27T03:37:55.516 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:55.398+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:55.399624+0000) 2024-04-27T03:37:55.516 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:55.510+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:55.516 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:55.510+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:55.516 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:55.510+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:55.778 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:55.511+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:55.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:55.713+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:55.714621+0000) 2024-04-27T03:37:56.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:55.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:56.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:55.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:56.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:55.772+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:56.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:55.773+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:56.373 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:37:56.176+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67758 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:37:56.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:56.173+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:56.174153+0000) 2024-04-27T03:37:56.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:56.173+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:56.174226+0000) 2024-04-27T03:37:56.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:56.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:56.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:56.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:56.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:56.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:56.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:56.080+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:56.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:56.174+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:56.175427+0000) 2024-04-27T03:37:56.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:56.494+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:56.495768+0000) 2024-04-27T03:37:56.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:56.399+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:56.400426+0000) 2024-04-27T03:37:56.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:56.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:56.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:56.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:56.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:56.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:56.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:56.502+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:56.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:56.714+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:56.715479+0000) 2024-04-27T03:37:56.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:56.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:56.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:56.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:56.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:56.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:56.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:56.762+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:57.259 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:57.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:57.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:57.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:57.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:57.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:57.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:57.083+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:57.510 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:57.497+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:57.497895+0000) 2024-04-27T03:37:57.511 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:57.401+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:57.402652+0000) 2024-04-27T03:37:57.511 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:57.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:57.511 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:57.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:57.511 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:57.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:57.511 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:57.504+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:57.769 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:57.716+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:57.717765+0000) 2024-04-27T03:37:58.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:57.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:58.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:57.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:58.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:57.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:58.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:57.764+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:58.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:58.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:58.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:58.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:58.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:58.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:58.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:58.054+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:58.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:58.499+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:58.499996+0000) 2024-04-27T03:37:58.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:58.406+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:58.406824+0000) 2024-04-27T03:37:58.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:58.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:58.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:58.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:58.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:58.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:58.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:58.542+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:58.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:58.719+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:58.719843+0000) 2024-04-27T03:37:58.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:58.783+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:58.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:58.783+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:58.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:58.783+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:58.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:58.784+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:59.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:59.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:59.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:59.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:59.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:59.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:37:59.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:59.022+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:59.549 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:59.503+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:59.504303+0000) 2024-04-27T03:37:59.549 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:59.408+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:59.409115+0000) 2024-04-27T03:37:59.549 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:59.543+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:59.549 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:59.543+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:59.549 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:59.543+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:37:59.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:37:59.544+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:37:59.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:59.723+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:37:59.723972+0000) 2024-04-27T03:37:59.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:59.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:59.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:59.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:59.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:59.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:37:59.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:37:59.746+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:00.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:59.974+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:00.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:59.974+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:00.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:59.974+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:00.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:37:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:37:59.974+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:00.512 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:00.410+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:00.411354+0000) 2024-04-27T03:38:00.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:00.727+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:00.728208+0000) 2024-04-27T03:38:00.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:00.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:00.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:00.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:00.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:00.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:00.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:00.569+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:00.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:00.506+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:00.507275+0000) 2024-04-27T03:38:00.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:00.723+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:00.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:00.723+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:00.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:00.723+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:00.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:00.724+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:01.260 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:01.182+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:01.183417+0000) 2024-04-27T03:38:01.260 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:01.181+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:01.182118+0000) 2024-04-27T03:38:01.261 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:38:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:38:01.185+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67797 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:38:01.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:00.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:01.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:00.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:01.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:00.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:01.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:00.982+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:01.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:01.182+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:01.182863+0000) 2024-04-27T03:38:01.513 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:01.411+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:01.412198+0000) 2024-04-27T03:38:01.769 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:01.728+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:01.729199+0000) 2024-04-27T03:38:01.770 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:01.616+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:01.770 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:01.616+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:01.770 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:01.616+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:01.770 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:01.616+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:01.770 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:01.507+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:01.508416+0000) 2024-04-27T03:38:01.770 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:01.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:01.770 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:01.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:01.771 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:01.763+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:02.021 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:01.765+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:02.420 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:02.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:02.420 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:02.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:02.420 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:02.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:02.420 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:02.015+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:02.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:02.509+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:02.510657+0000) 2024-04-27T03:38:02.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:02.414+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:02.414835+0000) 2024-04-27T03:38:02.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:02.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:02.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:02.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:02.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:02.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:02.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:02.618+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:03.007 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:02.730+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:02.731234+0000) 2024-04-27T03:38:03.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:02.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:03.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:02.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:03.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:02.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:03.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:02.777+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:03.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:03.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:03.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:03.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:03.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:03.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:03.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:03.002+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:03.520 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:03.417+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:03.418755+0000) 2024-04-27T03:38:03.780 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:03.732+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:03.733341+0000) 2024-04-27T03:38:03.780 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:03.592+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:03.780 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:03.592+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:03.780 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:03.592+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:03.780 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:03.593+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:03.780 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:03.514+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:03.514975+0000) 2024-04-27T03:38:03.781 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:03.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:03.781 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:03.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:03.781 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:03.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:03.781 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:03.753+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:04.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:03.974+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:04.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:03.974+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:04.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:03.974+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:04.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:03.974+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:04.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:04.421+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:04.422040+0000) 2024-04-27T03:38:04.741 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:04.517+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:04.517999+0000) 2024-04-27T03:38:04.741 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:04.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:04.741 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:04.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:04.742 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:04.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:04.742 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:04.582+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:05.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:04.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:05.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:04.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:05.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:04.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:05.010 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:04.757+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:05.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:04.735+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:04.736296+0000) 2024-04-27T03:38:05.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:04.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:05.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:04.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:05.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:04.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:05.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:04.941+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:05.525 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:05.423+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:05.424142+0000) 2024-04-27T03:38:05.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:05.737+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:05.738607+0000) 2024-04-27T03:38:05.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:05.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:05.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:05.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:05.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:05.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:05.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:05.584+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:05.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:05.519+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:05.520651+0000) 2024-04-27T03:38:05.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:05.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:05.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:05.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:05.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:05.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:05.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:05.742+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:06.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:05.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:06.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:05.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:06.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:05.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:06.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:05.919+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:06.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:06.190+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:06.191596+0000) 2024-04-27T03:38:06.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:06.190+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:06.191336+0000) 2024-04-27T03:38:06.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:38:06.194+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67837 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:38:06.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:06.190+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:06.191262+0000) 2024-04-27T03:38:06.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:06.424+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:06.425326+0000) 2024-04-27T03:38:06.744 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:06.520+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:06.521640+0000) 2024-04-27T03:38:06.745 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:06.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:06.745 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:06.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:06.745 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:06.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:06.745 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:06.613+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:07.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:06.738+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:06.739444+0000) 2024-04-27T03:38:07.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:06.904+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:07.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:06.904+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:07.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:06.904+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:07.010 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:06.904+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:07.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:06.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:07.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:06.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:07.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:06.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:07.011 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:06.771+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:07.529 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:07.426+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:07.427404+0000) 2024-04-27T03:38:07.796 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:07.741+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:07.741851+0000) 2024-04-27T03:38:07.796 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:07.523+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:07.524400+0000) 2024-04-27T03:38:07.797 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:07.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:07.797 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:07.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:07.797 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:07.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:07.797 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:07.647+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:08.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:07.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:08.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:07.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:08.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:07.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:08.055 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:07.866+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:08.055 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:07.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:08.055 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:07.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:08.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:07.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:08.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:07.790+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:08.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:08.528+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:08.529205+0000) 2024-04-27T03:38:08.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:08.428+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:08.429628+0000) 2024-04-27T03:38:08.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:08.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:08.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:08.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:08.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:08.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:08.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:08.635+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:08.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:08.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:08.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:08.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:08.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:08.773+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:08.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:08.773+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:08.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:08.745+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:08.746094+0000) 2024-04-27T03:38:08.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:08.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:08.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:08.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:08.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:08.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:08.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:08.851+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:09.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:09.531+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:09.531922+0000) 2024-04-27T03:38:09.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:09.431+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:09.431893+0000) 2024-04-27T03:38:09.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:09.594+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:09.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:09.594+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:09.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:09.594+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:09.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:09.595+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:10.015 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:09.747+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:09.748533+0000) 2024-04-27T03:38:10.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:09.805+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:10.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:09.805+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:10.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:09.805+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:10.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:09.805+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:10.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:09.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:10.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:09.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:10.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:09.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:10.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:09.824+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:10.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:10.533+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:10.534103+0000) 2024-04-27T03:38:10.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:10.433+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:10.434306+0000) 2024-04-27T03:38:10.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:10.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:10.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:10.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:10.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:10.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:10.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:10.549+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:11.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:10.749+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:10.750770+0000) 2024-04-27T03:38:11.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:10.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:11.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:10.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:11.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:10.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:11.016 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:10.777+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:11.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:10.809+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:11.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:10.809+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:11.017 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:10.809+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:11.017 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:10.809+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:11.327 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:38:11.203+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67876 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:38:11.327 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:11.198+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:11.199404+0000) 2024-04-27T03:38:11.327 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:11.200+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:11.200945+0000) 2024-04-27T03:38:11.328 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:11.198+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:11.199524+0000) 2024-04-27T03:38:11.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:11.434+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:11.435342+0000) 2024-04-27T03:38:11.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:11.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:11.581 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:11.534+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:11.534905+0000) 2024-04-27T03:38:11.841 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:11.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:11.842 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:11.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:11.842 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:11.575+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:11.842 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:11.750+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:11.751616+0000) 2024-04-27T03:38:11.842 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:11.778+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:11.842 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:11.778+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:11.842 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:11.778+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:11.842 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:11.778+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:11.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:11.835+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:11.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:11.835+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:11.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:11.835+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:12.120 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:11.836+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:12.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:12.536+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:12.537034+0000) 2024-04-27T03:38:12.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:12.438+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:12.439587+0000) 2024-04-27T03:38:12.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:12.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:12.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:12.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:12.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:12.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:12.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:12.559+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:13.021 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:12.810+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:13.021 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:12.810+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:13.022 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:12.810+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:13.022 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:12.811+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:13.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:12.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:13.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:12.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:13.023 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:12.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:13.023 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:12.751+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:13.023 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:12.753+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:12.754438+0000) 2024-04-27T03:38:13.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:13.538+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:13.539252+0000) 2024-04-27T03:38:13.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:13.441+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:13.441841+0000) 2024-04-27T03:38:13.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:13.523+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:13.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:13.523+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:13.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:13.523+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:13.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:13.524+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:14.023 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:13.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:14.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:13.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:14.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:13.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:14.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:13.806+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:14.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:13.758+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:13.759174+0000) 2024-04-27T03:38:14.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:13.765+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:14.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:13.765+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:14.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:13.765+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:14.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:13.766+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:14.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:14.540+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:14.541365+0000) 2024-04-27T03:38:14.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:14.443+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:14.444121+0000) 2024-04-27T03:38:14.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:14.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:14.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:14.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:14.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:14.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:14.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:14.529+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:15.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:14.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:15.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:14.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:15.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:14.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:15.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:14.831+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:15.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:14.763+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:14.763982+0000) 2024-04-27T03:38:15.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:14.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:15.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:14.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:15.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:14.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:15.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:14.800+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:15.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:15.542+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:15.543474+0000) 2024-04-27T03:38:15.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:15.445+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:15.446389+0000) 2024-04-27T03:38:15.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:15.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:15.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:15.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:15.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:15.563+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:16.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:15.768+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:15.768862+0000) 2024-04-27T03:38:16.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:15.803+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:16.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:15.803+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:16.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:15.803+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:16.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:15.804+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:16.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:15.814+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:16.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:15.814+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:16.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:15.814+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:16.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:15.815+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:16.338 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:38:16.211+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67916 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:38:16.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:16.207+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:16.208454+0000) 2024-04-27T03:38:16.338 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:16.207+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:16.208498+0000) 2024-04-27T03:38:16.339 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:16.210+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:16.211048+0000) 2024-04-27T03:38:16.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:16.543+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:16.544310+0000) 2024-04-27T03:38:16.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:16.447+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:16.447944+0000) 2024-04-27T03:38:16.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:16.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:16.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:16.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:16.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:16.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:16.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:16.562+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:17.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:16.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:17.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:16.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:17.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:16.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:17.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:16.775+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:17.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:16.768+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:16.769711+0000) 2024-04-27T03:38:17.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:16.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:17.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:16.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:17.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:16.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:17.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:16.830+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:17.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:17.548+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:17.548963+0000) 2024-04-27T03:38:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:17.451+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:17.452570+0000) 2024-04-27T03:38:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:17.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:17.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:17.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:17.545+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:18.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:17.773+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:17.774433+0000) 2024-04-27T03:38:18.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:17.792+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:18.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:17.792+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:18.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:17.792+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:18.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:17.792+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:18.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:17.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:18.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:17.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:18.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:17.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:18.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:17.802+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:18.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:18.552+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:18.553260+0000) 2024-04-27T03:38:18.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:18.456+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:18.457251+0000) 2024-04-27T03:38:18.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:18.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:18.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:18.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:18.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:18.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:18.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:18.565+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:19.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:18.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:19.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:18.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:19.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:18.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:19.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:18.759+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:19.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:18.776+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:18.777371+0000) 2024-04-27T03:38:19.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:18.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:19.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:18.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:19.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:18.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:19.032 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:18.799+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:19.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:19.555+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:19.556224+0000) 2024-04-27T03:38:19.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:19.458+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:19.459530+0000) 2024-04-27T03:38:19.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:19.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:19.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:19.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:19.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:19.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:19.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:19.520+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:20.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:19.723+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:20.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:19.723+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:20.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:19.723+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:20.031 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:19.723+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:20.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:19.761+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:20.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:19.761+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:20.032 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:19.761+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:20.032 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:19.761+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:20.032 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:19.778+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:19.779473+0000) 2024-04-27T03:38:20.565 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:20.558+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:20.559270+0000) 2024-04-27T03:38:20.566 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:20.461+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:20.462223+0000) 2024-04-27T03:38:20.566 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:20.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:20.566 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:20.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:20.566 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:20.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:20.566 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:20.497+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:21.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:20.776+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:21.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:20.776+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:21.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:20.776+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:21.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:20.777+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:21.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:20.779+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:20.781613+0000) 2024-04-27T03:38:21.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:20.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:21.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:20.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:21.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:20.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:21.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:20.735+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:21.345 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:38:21.218+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67955 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:38:21.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:21.215+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:21.216848+0000) 2024-04-27T03:38:21.346 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:21.215+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:21.217457+0000) 2024-04-27T03:38:21.346 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:21.215+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:21.217076+0000) 2024-04-27T03:38:21.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:21.558+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:21.560071+0000) 2024-04-27T03:38:21.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:21.461+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:21.463225+0000) 2024-04-27T03:38:21.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:21.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:21.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:21.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:21.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:21.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:21.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:21.487+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:22.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:21.740+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:22.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:21.740+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:22.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:21.740+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:22.032 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:21.740+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:22.032 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:21.780+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:21.782467+0000) 2024-04-27T03:38:22.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:21.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:22.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:21.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:22.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:21.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:22.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:21.745+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:22.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:22.562+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:22.564380+0000) 2024-04-27T03:38:22.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:22.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:22.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:22.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:22.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:22.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:22.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:22.463+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:22.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:22.463+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:22.465282+0000) 2024-04-27T03:38:23.035 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:22.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:23.035 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:22.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:23.035 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:22.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:23.035 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:22.773+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:23.035 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:22.782+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:22.784599+0000) 2024-04-27T03:38:23.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:22.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:23.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:22.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:23.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:22.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:23.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:22.734+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:23.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:23.564+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:23.566564+0000) 2024-04-27T03:38:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:23.465+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:23.467360+0000) 2024-04-27T03:38:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:23.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:23.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:23.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:23.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:23.506+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:24.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:23.784+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:23.786774+0000) 2024-04-27T03:38:24.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:23.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:24.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:23.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:24.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:23.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:24.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:23.801+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:24.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:23.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:24.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:23.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:24.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:23.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:24.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:23.777+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:24.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:24.567+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:24.568900+0000) 2024-04-27T03:38:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:24.467+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:24.469698+0000) 2024-04-27T03:38:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:24.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:24.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:24.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:24.536+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:25.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:24.758+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:25.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:24.758+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:25.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:24.758+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:25.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:24.758+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:25.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:24.788+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:24.790073+0000) 2024-04-27T03:38:25.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:24.784+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:25.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:24.784+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:25.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:24.784+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:25.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:24.785+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:25.768 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:25.569+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:25.571316+0000) 2024-04-27T03:38:25.768 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:25.470+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:25.472446+0000) 2024-04-27T03:38:25.768 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:25.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:25.768 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:25.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:25.768 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:25.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:25.768 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:25.494+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:26.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:25.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:26.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:25.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:26.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:25.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:26.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:25.800+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:26.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:25.761+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:26.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:25.761+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:26.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:25.761+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:26.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:25.761+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:26.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:25.792+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:25.794378+0000) 2024-04-27T03:38:26.354 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:26.223+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:26.225097+0000) 2024-04-27T03:38:26.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:26.222+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:26.224710+0000) 2024-04-27T03:38:26.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:26.223+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:26.224996+0000) 2024-04-27T03:38:26.355 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:38:26.229+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 67995 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:38:26.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:26.570+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:26.572418+0000) 2024-04-27T03:38:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:26.472+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:26.473984+0000) 2024-04-27T03:38:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:26.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:26.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:26.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:26.487+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:27.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:26.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:27.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:26.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:27.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:26.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:27.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:26.758+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:27.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:26.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:27.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:26.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:27.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:26.756+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:27.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:26.757+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:27.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:26.794+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:26.796004+0000) 2024-04-27T03:38:27.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:27.575+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:27.577438+0000) 2024-04-27T03:38:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:27.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:27.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:27.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:27.447+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:27.474+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:27.476083+0000) 2024-04-27T03:38:28.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:27.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:28.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:27.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:28.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:27.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:28.043 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:27.727+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:28.043 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:27.788+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:28.044 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:27.788+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:28.044 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:27.788+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:28.044 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:27.789+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:28.044 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:27.797+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:27.799136+0000) 2024-04-27T03:38:28.762 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:28.476+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:28.478153+0000) 2024-04-27T03:38:28.762 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:28.484+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:28.762 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:28.484+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:28.762 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:28.484+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:28.762 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:28.484+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:28.763 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:28.577+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:28.579703+0000) 2024-04-27T03:38:28.763 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:28.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:28.763 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:28.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:28.763 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:28.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:29.046 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:28.756+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:29.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:28.799+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:28.801373+0000) 2024-04-27T03:38:29.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:28.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:29.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:28.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:29.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:28.818+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:29.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:28.818+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:29.809 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:29.478+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:29.480296+0000) 2024-04-27T03:38:29.809 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:29.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:29.809 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:29.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:29.809 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:29.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:29.809 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:29.519+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:29.809 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:29.580+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:29.581986+0000) 2024-04-27T03:38:29.809 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:29.723+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:29.810 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:29.723+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:29.810 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:29.723+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:29.810 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:29.724+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:30.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:29.801+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:29.803795+0000) 2024-04-27T03:38:30.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:29.802+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:30.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:29.802+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:30.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:29.802+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:30.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:29.802+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:30.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:30.583+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:30.584820+0000) 2024-04-27T03:38:30.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:30.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:30.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:30.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:30.756 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:30.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:30.756 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:30.481+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:30.482935+0000) 2024-04-27T03:38:30.756 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:30.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:30.756 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:30.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:30.756 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:30.545+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:30.756 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:30.546+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:31.046 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:30.749+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:31.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:30.804+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:31.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:30.804+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:31.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:30.804+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:31.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:30.805+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:30.806899+0000) 2024-04-27T03:38:31.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:30.805+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:31.362 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:38:31.236+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68035 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:38:31.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:31.231+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:31.236696+0000) 2024-04-27T03:38:31.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:31.231+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:31.235808+0000) 2024-04-27T03:38:31.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:31.231+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:31.236210+0000) 2024-04-27T03:38:31.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:31.585+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:31.586272+0000) 2024-04-27T03:38:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:31.482+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:31.483825+0000) 2024-04-27T03:38:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:31.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:31.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:31.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:31.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:31.557+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:32.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:31.770+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:32.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:31.770+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:32.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:31.770+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:32.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:31.771+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:32.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:31.806+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:31.808343+0000) 2024-04-27T03:38:32.047 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:31.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:32.047 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:31.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:32.047 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:31.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:32.047 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:31.790+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:32.808 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:32.587+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:32.588779+0000) 2024-04-27T03:38:32.809 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:32.485+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:32.486217+0000) 2024-04-27T03:38:32.809 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:32.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:32.809 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:32.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:32.809 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:32.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:32.809 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:32.538+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:33.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:32.811+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:32.812595+0000) 2024-04-27T03:38:33.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:32.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:33.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:32.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:33.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:32.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:33.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:32.820+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:33.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:32.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:33.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:32.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:33.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:32.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:33.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:32.802+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:33.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:33.590+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:33.591500+0000) 2024-04-27T03:38:33.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:33.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:33.801 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:33.487+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:33.488160+0000) 2024-04-27T03:38:33.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:33.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:33.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:33.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:33.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:33.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:33.802 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:33.531+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:34.051 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:33.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:34.051 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:33.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:34.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:33.796+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:34.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:33.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:34.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:33.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:34.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:33.799+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:34.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:33.799+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:34.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:33.816+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:33.816857+0000) 2024-04-27T03:38:34.806 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:34.595+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:34.595842+0000) 2024-04-27T03:38:34.806 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:34.489+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:34.490325+0000) 2024-04-27T03:38:34.806 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:34.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:34.807 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:34.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:34.807 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:34.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:34.807 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:34.521+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:34.807 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:34.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:34.807 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:34.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:34.807 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:34.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:34.807 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:34.800+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:35.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:34.818+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:34.819111+0000) 2024-04-27T03:38:35.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:34.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:35.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:34.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:35.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:34.811+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:35.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:34.812+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:35.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:35.480+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:35.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:35.480+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:35.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:35.480+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:35.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:35.481+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:35.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:35.491+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:35.492715+0000) 2024-04-27T03:38:35.805 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:35.597+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:35.598085+0000) 2024-04-27T03:38:35.805 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:35.797+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:35.805 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:35.797+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:35.805 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:35.797+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:36.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:35.799+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:36.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:35.820+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:35.821281+0000) 2024-04-27T03:38:36.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:35.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:36.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:35.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:36.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:35.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:36.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:35.846+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:36.373 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:38:36.246+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68075 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:38:36.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:36.242+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:36.243437+0000) 2024-04-27T03:38:36.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:36.243+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:36.244725+0000) 2024-04-27T03:38:36.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:36.242+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:36.243683+0000) 2024-04-27T03:38:36.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:36.598+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:36.598910+0000) 2024-04-27T03:38:36.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:36.493+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:36.494304+0000) 2024-04-27T03:38:36.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:36.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:36.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:36.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:36.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:36.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:36.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:36.529+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:37.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:36.821+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:36.822157+0000) 2024-04-27T03:38:37.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:36.881+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:37.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:36.881+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:37.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:36.881+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:37.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:36.882+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:37.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:36.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:37.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:36.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:37.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:36.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:37.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:36.815+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:37.829 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:37.600+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:37.601549+0000) 2024-04-27T03:38:37.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:37.497+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:37.498522+0000) 2024-04-27T03:38:37.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:37.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:37.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:37.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:37.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:37.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:37.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:37.559+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:38.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:37.826+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:38.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:37.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:38.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:37.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:38.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:37.827+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:38.124 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:37.823+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:37.824410+0000) 2024-04-27T03:38:38.124 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:37.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:38.124 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:37.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:38.124 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:37.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:38.124 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:37.900+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:38.803 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:38.602+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:38.603651+0000) 2024-04-27T03:38:38.803 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:38.500+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:38.501342+0000) 2024-04-27T03:38:38.803 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:38.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:38.803 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:38.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:38.803 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:38.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:38.803 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:38.585+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:39.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:38.797+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:39.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:38.797+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:39.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:38.797+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:39.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:38.798+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:39.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:38.825+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:38.826664+0000) 2024-04-27T03:38:39.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:38.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:39.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:38.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:39.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:38.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:39.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:38.923+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:39.806 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:39.604+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:39.605780+0000) 2024-04-27T03:38:39.806 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:39.502+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:39.503405+0000) 2024-04-27T03:38:39.807 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:39.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:39.807 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:39.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:39.807 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:39.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:39.807 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:39.574+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:40.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:39.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:40.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:39.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:40.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:39.800+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:40.058 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:39.801+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:40.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:39.828+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:39.829439+0000) 2024-04-27T03:38:40.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:39.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:40.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:39.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:40.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:39.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:40.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:39.874+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:40.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:40.504+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:40.505726+0000) 2024-04-27T03:38:40.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:40.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:40.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:40.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:40.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:40.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:40.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:40.539+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:40.887 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:40.830+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:40.831715+0000) 2024-04-27T03:38:40.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:40.608+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:40.609037+0000) 2024-04-27T03:38:40.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:40.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:40.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:40.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:40.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:40.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:40.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:40.797+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:41.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:40.881+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:41.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:40.881+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:41.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:40.881+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:41.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:40.882+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:41.511 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:38:41.255+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68115 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:38:41.512 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:41.253+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:41.253835+0000) 2024-04-27T03:38:41.512 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:41.252+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:41.253248+0000) 2024-04-27T03:38:41.512 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:41.251+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:41.252119+0000) 2024-04-27T03:38:41.802 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:41.609+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:41.610062+0000) 2024-04-27T03:38:41.803 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:41.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:41.803 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:41.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:41.803 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:41.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:41.803 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:41.505+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:41.506608+0000) 2024-04-27T03:38:41.803 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:41.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:41.803 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:41.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:41.803 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:41.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:41.803 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:41.526+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:42.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:41.797+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:42.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:41.832+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:41.833184+0000) 2024-04-27T03:38:42.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:41.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:42.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:41.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:42.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:41.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:42.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:41.924+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:42.756 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:42.612+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:42.613182+0000) 2024-04-27T03:38:42.757 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:42.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:42.757 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:42.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:42.757 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:42.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:42.757 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:42.490+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:42.757 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:42.508+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:42.509698+0000) 2024-04-27T03:38:43.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:42.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:43.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:42.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:43.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:42.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:43.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:42.752+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:43.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:42.836+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:42.837375+0000) 2024-04-27T03:38:43.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:42.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:43.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:42.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:43.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:42.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:43.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:42.948+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:43.758 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:43.615+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:43.615847+0000) 2024-04-27T03:38:43.758 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:43.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:43.758 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:43.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:43.758 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:43.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:43.758 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:43.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:43.758 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:43.475+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:43.759 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:43.511+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:43.511927+0000) 2024-04-27T03:38:44.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:43.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:44.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:43.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:44.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:43.752+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:44.032 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:43.838+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:43.839509+0000) 2024-04-27T03:38:44.032 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:43.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:44.032 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:43.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:44.033 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:43.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:44.033 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:43.936+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:44.760 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:44.618+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:44.619078+0000) 2024-04-27T03:38:44.760 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:44.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:44.760 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:44.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:44.760 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:44.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:44.760 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:44.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:44.760 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:44.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:44.760 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:44.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:44.760 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:44.470+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:44.761 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:44.513+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:44.514162+0000) 2024-04-27T03:38:45.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:44.754+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:45.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:44.843+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:44.843800+0000) 2024-04-27T03:38:45.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:44.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:45.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:44.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:45.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:44.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:45.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:44.922+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:45.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:45.620+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:45.621766+0000) 2024-04-27T03:38:45.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:45.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:45.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:45.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:45.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:45.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:45.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:45.436+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:45.516+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:45.517202+0000) 2024-04-27T03:38:46.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:45.846+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:45.847124+0000) 2024-04-27T03:38:46.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:45.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:46.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:45.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:46.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:45.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:46.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:45.954+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:46.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:45.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:46.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:45.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:46.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:45.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:46.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:45.799+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:46.372 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:38:46.263+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68155 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:38:46.373 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:46.259+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:46.260708+0000) 2024-04-27T03:38:46.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:46.259+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:46.260719+0000) 2024-04-27T03:38:46.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:46.260+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:46.260907+0000) 2024-04-27T03:38:46.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:46.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:46.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:46.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:46.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:46.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:46.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:46.459+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:46.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:46.518+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:46.518846+0000) 2024-04-27T03:38:46.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:46.847+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:46.848775+0000) 2024-04-27T03:38:46.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:46.621+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:46.622666+0000) 2024-04-27T03:38:46.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:46.845+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:46.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:46.845+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:46.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:46.845+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:46.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:46.846+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:47.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:46.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:47.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:46.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:47.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:46.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:47.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:46.908+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:47.526 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:47.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:47.526 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:47.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:47.526 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:47.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:47.526 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:47.463+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:47.526 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:47.520+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:47.520978+0000) 2024-04-27T03:38:47.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:47.624+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:47.624977+0000) 2024-04-27T03:38:47.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:47.814+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:48.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:47.814+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:48.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:47.814+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:48.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:47.816+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:48.124 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:47.850+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:47.850995+0000) 2024-04-27T03:38:48.124 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:47.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:48.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:47.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:48.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:47.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:48.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:47.924+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:48.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:48.627+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:48.628168+0000) 2024-04-27T03:38:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:48.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:48.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:48.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:48.421+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:48.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:48.522+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:48.523308+0000) 2024-04-27T03:38:49.067 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:48.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:49.067 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:48.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:49.067 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:48.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:49.067 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:48.806+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:49.067 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:48.852+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:48.853690+0000) 2024-04-27T03:38:49.068 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:48.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:49.068 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:48.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:49.068 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:48.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:49.068 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:48.911+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:49.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:49.629+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:49.630353+0000) 2024-04-27T03:38:49.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:49.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:49.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:49.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:49.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:49.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:49.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:49.463+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:49.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:49.526+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:49.527535+0000) 2024-04-27T03:38:50.124 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:49.857+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:49.857961+0000) 2024-04-27T03:38:50.124 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:49.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:50.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:49.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:50.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:49.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:50.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:49.930+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:50.125 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:49.849+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:50.125 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:49.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:50.125 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:49.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:50.125 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:49.851+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:50.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:50.631+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:50.632596+0000) 2024-04-27T03:38:50.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:50.838+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:50.845 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:50.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:50.845 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:50.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:50.845 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:50.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:50.845 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:50.478+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:50.845 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:50.530+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:50.531784+0000) 2024-04-27T03:38:51.124 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:50.859+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:50.860068+0000) 2024-04-27T03:38:51.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:50.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:51.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:50.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:51.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:50.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:51.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:50.947+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:51.125 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:50.838+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:51.125 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:50.838+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:51.125 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:50.839+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:51.398 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:38:51.271+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68194 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:38:51.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:51.267+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:51.268428+0000) 2024-04-27T03:38:51.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:51.268+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:51.268993+0000) 2024-04-27T03:38:51.399 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:51.268+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:51.269766+0000) 2024-04-27T03:38:51.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:51.633+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:51.634295+0000) 2024-04-27T03:38:51.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:51.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:51.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:51.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:51.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:51.474+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:51.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:51.476+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:51.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:51.531+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:51.532685+0000) 2024-04-27T03:38:52.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:51.860+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:51.860970+0000) 2024-04-27T03:38:52.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:51.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:52.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:51.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:52.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:51.908+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:52.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:51.908+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:52.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:51.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:52.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:51.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:52.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:51.795+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:52.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:51.796+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:52.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:52.635+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:52.636543+0000) 2024-04-27T03:38:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:52.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:52.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:52.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:52.429+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:52.534+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:52.535395+0000) 2024-04-27T03:38:53.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:52.864+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:52.865734+0000) 2024-04-27T03:38:53.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:52.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:53.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:52.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:53.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:52.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:53.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:52.922+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:53.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:52.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:53.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:52.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:53.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:52.748+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:53.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:52.749+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:53.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:53.640+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:53.641079+0000) 2024-04-27T03:38:53.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:53.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:53.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:53.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:53.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:53.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:53.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:53.420+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:53.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:53.538+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:53.539571+0000) 2024-04-27T03:38:54.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:53.867+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:53.867846+0000) 2024-04-27T03:38:54.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:53.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:54.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:53.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:54.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:53.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:54.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:53.879+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:54.061 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:53.730+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:54.061 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:53.730+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:54.061 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:53.730+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:54.061 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:53.732+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:54.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:54.642+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:54.643176+0000) 2024-04-27T03:38:54.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:54.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:54.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:54.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:54.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:54.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:54.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:54.444+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:54.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:54.543+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:54.544328+0000) 2024-04-27T03:38:55.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:54.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:55.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:54.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:55.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:54.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:55.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:54.757+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:55.044 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:54.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:55.044 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:54.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:55.044 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:54.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:55.045 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:54.858+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:55.045 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:54.869+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:54.869952+0000) 2024-04-27T03:38:55.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:55.644+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:55.645349+0000) 2024-04-27T03:38:55.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:55.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:55.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:55.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:55.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:55.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:55.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:55.436+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:55.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:55.548+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:55.549078+0000) 2024-04-27T03:38:56.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:55.803+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:56.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:55.803+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:56.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:55.803+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:56.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:55.804+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:56.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:55.871+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:55.872233+0000) 2024-04-27T03:38:56.060 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:55.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:56.061 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:55.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:56.061 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:55.895+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:56.061 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:55.896+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:56.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:56.275+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:56.276797+0000) 2024-04-27T03:38:56.373 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:38:56.280+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68234 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:38:56.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:56.276+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:56.277708+0000) 2024-04-27T03:38:56.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:56.277+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:56.278789+0000) 2024-04-27T03:38:56.652 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:56.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:56.652 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:56.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:56.653 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:56.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:56.653 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:56.460+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:56.653 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:56.549+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:56.549991+0000) 2024-04-27T03:38:56.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:56.872+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:56.873293+0000) 2024-04-27T03:38:56.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:56.926+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:56.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:56.646+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:56.647286+0000) 2024-04-27T03:38:56.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:56.765+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:56.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:56.765+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:56.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:56.765+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:56.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:56.766+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:57.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:56.926+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:57.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:56.926+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:57.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:56.927+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:57.557 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:57.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:57.557 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:57.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:57.557 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:57.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:57.557 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:57.436+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:57.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:57.551+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:57.552281+0000) 2024-04-27T03:38:57.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:57.648+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:57.649558+0000) 2024-04-27T03:38:57.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:57.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:57.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:57.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:57.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:57.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:57.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:57.754+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:58.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:57.877+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:57.878355+0000) 2024-04-27T03:38:58.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:57.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:58.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:57.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:58.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:57.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:58.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:57.879+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:58.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:58.416+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:58.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:58.416+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:58.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:58.416+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:58.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:58.417+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:58.888 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:58.556+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:58.557053+0000) 2024-04-27T03:38:58.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:58.650+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:58.651659+0000) 2024-04-27T03:38:58.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:58.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:58.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:58.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:58.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:58.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:58.889 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:58.759+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:59.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:58.882+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:58.883295+0000) 2024-04-27T03:38:59.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:58.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:59.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:58.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:59.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:58.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:38:59.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:58.924+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:59.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:59.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:59.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:59.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:59.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:59.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:38:59.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:59.437+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:38:59.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:38:59.561+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:59.562264+0000) 2024-04-27T03:38:59.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:59.653+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:59.653800+0000) 2024-04-27T03:38:59.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:59.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:59.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:59.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:59.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:59.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:38:59.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:38:59.717+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:00.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:59.887+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:38:59.888268+0000) 2024-04-27T03:39:00.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:59.896+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:00.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:59.896+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:00.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:59.896+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:00.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:38:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:38:59.896+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:00.572 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:00.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:00.572 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:00.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:00.572 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:00.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:00.573 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:00.426+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:00.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:00.566+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:00.567348+0000) 2024-04-27T03:39:00.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:00.656+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:00.657067+0000) 2024-04-27T03:39:00.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:00.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:00.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:00.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:00.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:00.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:00.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:00.712+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:01.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:00.892+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:00.893363+0000) 2024-04-27T03:39:01.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:00.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:01.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:00.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:01.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:00.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:01.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:00.912+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:01.574 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:39:01.288+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68275 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:39:01.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:01.285+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:01.286641+0000) 2024-04-27T03:39:01.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:01.286+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:01.287326+0000) 2024-04-27T03:39:01.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:01.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:01.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:01.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:01.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:01.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:01.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:01.440+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:01.576 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:01.285+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:01.285914+0000) 2024-04-27T03:39:01.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:01.568+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:01.569544+0000) 2024-04-27T03:39:01.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:01.657+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:01.657991+0000) 2024-04-27T03:39:01.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:01.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:01.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:01.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:01.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:01.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:01.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:01.699+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:02.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:01.893+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:01.894770+0000) 2024-04-27T03:39:02.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:01.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:02.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:01.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:02.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:01.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:02.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:01.941+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:02.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:02.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:02.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:02.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:02.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:02.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:02.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:02.428+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:02.902 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:02.572+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:02.572922+0000) 2024-04-27T03:39:02.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:02.662+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:02.663046+0000) 2024-04-27T03:39:02.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:02.665+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:02.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:02.665+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:02.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:02.665+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:02.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:02.666+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:03.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:02.896+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:02.897214+0000) 2024-04-27T03:39:03.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:02.967+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:03.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:02.967+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:03.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:02.967+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:03.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:02.967+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:03.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:03.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:03.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:03.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:03.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:03.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:03.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:03.411+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:03.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:03.575+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:03.575996+0000) 2024-04-27T03:39:03.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:03.667+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:03.668053+0000) 2024-04-27T03:39:03.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:03.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:03.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:03.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:03.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:03.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:03.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:03.712+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:04.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:03.901+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:03.902382+0000) 2024-04-27T03:39:04.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:03.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:04.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:03.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:04.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:03.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:04.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:03.989+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:04.493 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:04.372+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:04.493 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:04.372+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:04.493 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:04.372+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:04.493 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:04.373+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:04.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:04.578+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:04.578839+0000) 2024-04-27T03:39:04.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:04.672+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:04.673183+0000) 2024-04-27T03:39:04.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:04.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:04.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:04.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:04.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:04.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:04.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:04.689+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:05.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:04.906+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:04.907089+0000) 2024-04-27T03:39:05.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:05.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:05.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:05.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:05.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:05.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:05.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:05.026+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:05.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:05.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:05.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:05.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:05.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:05.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:05.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:05.384+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:05.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:05.582+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:05.583653+0000) 2024-04-27T03:39:05.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:05.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:05.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:05.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:05.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:05.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:05.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:05.673+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:05.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:05.674+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:05.675368+0000) 2024-04-27T03:39:06.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:05.910+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:05.911222+0000) 2024-04-27T03:39:06.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:06.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:06.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:06.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:06.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:06.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:06.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:06.061+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:06.591 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:39:06.296+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68315 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:39:06.591 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:06.293+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:06.294311+0000) 2024-04-27T03:39:06.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:06.294+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:06.295299+0000) 2024-04-27T03:39:06.591 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:06.294+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:06.294825+0000) 2024-04-27T03:39:06.592 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:06.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:06.592 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:06.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:06.592 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:06.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:06.592 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:06.346+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:06.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:06.584+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:06.585727+0000) 2024-04-27T03:39:06.918 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:06.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:06.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:06.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:06.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:06.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:06.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:06.624+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:06.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:06.675+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:06.676339+0000) 2024-04-27T03:39:07.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:06.912+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:06.913333+0000) 2024-04-27T03:39:07.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:07.086+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:07.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:07.086+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:07.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:07.086+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:07.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:07.087+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:07.593 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:07.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:07.593 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:07.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:07.593 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:07.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:07.593 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:07.350+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:07.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:07.587+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:07.588014+0000) 2024-04-27T03:39:07.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:07.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:07.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:07.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:07.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:07.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:07.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:07.664+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:07.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:07.677+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:07.678599+0000) 2024-04-27T03:39:08.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:07.914+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:07.915518+0000) 2024-04-27T03:39:08.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:08.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:08.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:08.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:08.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:08.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:08.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:08.043+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:08.596 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:08.390+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:08.596 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:08.390+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:08.596 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:08.390+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:08.596 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:08.392+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:08.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:08.590+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:08.591128+0000) 2024-04-27T03:39:08.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:08.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:08.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:08.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:08.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:08.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:08.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:08.669+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:08.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:08.680+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:08.681450+0000) 2024-04-27T03:39:09.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:08.917+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:08.918695+0000) 2024-04-27T03:39:09.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:09.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:09.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:09.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:09.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:09.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:09.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:09.042+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:09.598 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:09.372+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:09.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:09.372+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:09.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:09.372+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:09.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:09.373+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:09.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:09.592+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:09.593671+0000) 2024-04-27T03:39:09.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:09.683+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:09.684720+0000) 2024-04-27T03:39:09.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:09.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:09.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:09.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:09.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:09.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:09.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:09.700+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:10.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:09.920+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:09.920989+0000) 2024-04-27T03:39:10.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:09.997+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:10.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:09.997+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:10.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:09.997+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:10.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:09.998+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:10.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:10.348+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:10.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:10.348+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:10.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:10.348+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:10.603 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:10.349+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:10.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:10.597+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:10.597850+0000) 2024-04-27T03:39:10.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:10.686+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:10.687032+0000) 2024-04-27T03:39:10.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:10.727+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:10.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:10.727+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:10.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:10.727+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:10.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:10.728+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:11.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:10.922+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:10.923387+0000) 2024-04-27T03:39:11.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:11.046+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:11.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:11.046+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:11.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:11.046+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:11.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:11.046+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:11.604 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:39:11.305+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68354 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:39:11.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:11.303+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:11.304618+0000) 2024-04-27T03:39:11.605 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:11.301+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:11.301902+0000) 2024-04-27T03:39:11.605 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:11.302+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:11.303416+0000) 2024-04-27T03:39:11.605 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:11.311+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:11.605 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:11.311+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:11.605 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:11.311+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:11.605 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:11.313+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:11.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:11.598+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:11.599192+0000) 2024-04-27T03:39:11.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:11.687+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:11.687952+0000) 2024-04-27T03:39:11.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:11.708+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:11.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:11.708+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:11.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:11.708+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:11.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:11.709+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:12.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:11.923+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:11.924328+0000) 2024-04-27T03:39:12.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:12.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:12.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:12.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:12.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:12.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:12.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:12.066+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:12.606 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:12.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:12.607 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:12.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:12.607 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:12.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:12.607 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:12.333+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:12.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:12.600+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:12.601655+0000) 2024-04-27T03:39:12.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:12.690+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:12.691114+0000) 2024-04-27T03:39:12.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:12.695+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:12.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:12.695+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:12.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:12.695+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:12.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:12.697+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:13.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:12.927+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:12.928479+0000) 2024-04-27T03:39:13.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:13.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:13.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:13.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:13.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:13.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:13.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:13.037+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:13.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:13.342+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:13.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:13.342+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:13.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:13.342+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:13.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:13.344+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:13.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:13.605+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:13.605815+0000) 2024-04-27T03:39:13.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:13.692+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:13.693187+0000) 2024-04-27T03:39:13.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:13.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:13.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:13.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:13.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:13.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:13.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:13.739+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:14.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:13.930+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:13.931072+0000) 2024-04-27T03:39:14.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:14.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:14.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:14.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:14.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:14.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:14.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:14.037+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:14.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:14.376+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:14.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:14.376+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:14.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:14.376+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:14.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:14.376+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:14.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:14.607+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:14.608342+0000) 2024-04-27T03:39:14.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:14.694+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:14.695503+0000) 2024-04-27T03:39:14.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:14.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:14.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:14.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:14.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:14.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:14.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:14.758+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:15.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:14.934+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:14.935308+0000) 2024-04-27T03:39:15.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:14.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:15.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:14.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:15.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:14.994+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:15.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:14.995+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:15.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:15.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:15.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:15.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:15.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:15.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:15.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:15.358+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:15.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:15.611+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:15.612654+0000) 2024-04-27T03:39:15.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:15.696+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:15.697621+0000) 2024-04-27T03:39:15.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:15.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:15.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:15.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:15.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:15.734+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:15.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:15.735+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:16.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:15.937+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:15.937925+0000) 2024-04-27T03:39:16.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:16.011+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:16.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:16.011+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:16.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:16.011+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:16.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:16.011+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:16.619 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:39:16.313+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68394 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:39:16.620 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:16.309+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:16.310616+0000) 2024-04-27T03:39:16.620 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:16.309+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:16.310290+0000) 2024-04-27T03:39:16.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:16.310+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:16.311278+0000) 2024-04-27T03:39:16.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:16.370+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:16.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:16.370+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:16.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:16.370+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:16.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:16.371+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:16.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:16.613+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:16.614402+0000) 2024-04-27T03:39:16.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:16.697+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:16.698580+0000) 2024-04-27T03:39:16.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:16.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:16.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:16.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:16.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:16.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:16.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:16.756+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:17.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:16.939+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:16.940096+0000) 2024-04-27T03:39:17.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:16.970+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:17.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:16.970+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:17.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:16.970+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:17.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:16.970+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:17.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:17.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:17.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:17.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:17.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:17.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:17.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:17.365+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:17.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:17.616+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:17.617114+0000) 2024-04-27T03:39:17.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:17.702+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:17.703623+0000) 2024-04-27T03:39:17.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:17.789+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:17.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:17.789+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:17.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:17.789+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:17.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:17.790+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:18.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:17.942+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:17.943072+0000) 2024-04-27T03:39:18.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:17.997+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:18.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:17.997+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:18.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:17.997+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:18.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:17.998+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:18.626 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:18.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:18.626 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:18.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:18.627 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:18.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:18.627 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:18.333+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:18.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:18.620+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:18.621346+0000) 2024-04-27T03:39:18.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:18.704+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:18.705782+0000) 2024-04-27T03:39:18.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:18.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:18.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:18.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:18.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:18.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:18.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:18.804+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:19.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:18.944+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:18.945508+0000) 2024-04-27T03:39:19.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:18.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:19.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:18.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:19.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:18.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:19.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:18.994+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:19.629 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:19.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:19.630 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:19.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:19.630 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:19.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:19.630 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:19.333+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:19.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:19.623+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:19.624348+0000) 2024-04-27T03:39:19.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:19.709+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:19.710068+0000) 2024-04-27T03:39:19.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:19.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:19.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:19.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:19.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:19.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:19.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:19.854+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:20.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:19.946+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:19.947579+0000) 2024-04-27T03:39:20.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:19.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:20.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:19.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:20.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:19.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:20.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:19.948+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:20.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:20.335+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:20.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:20.335+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:20.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:20.335+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:20.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:20.336+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:20.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:20.626+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:20.627534+0000) 2024-04-27T03:39:20.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:20.712+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:20.713179+0000) 2024-04-27T03:39:21.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:20.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:21.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:20.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:21.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:20.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:21.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:20.890+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:21.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:20.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:21.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:20.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:21.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:20.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:21.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:20.910+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:21.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:20.949+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:20.950522+0000) 2024-04-27T03:39:21.633 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:39:21.321+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68433 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:39:21.634 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:21.318+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:21.319315+0000) 2024-04-27T03:39:21.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:21.318+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:21.318838+0000) 2024-04-27T03:39:21.634 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:21.317+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:21.318798+0000) 2024-04-27T03:39:21.634 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:21.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:21.634 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:21.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:21.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:21.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:21.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:21.374+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:21.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:21.627+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:21.628491+0000) 2024-04-27T03:39:21.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:21.713+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:21.714174+0000) 2024-04-27T03:39:21.914 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:21.907+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:22.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:21.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:22.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:21.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:22.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:21.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:22.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:21.915+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:22.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:21.950+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:21.951609+0000) 2024-04-27T03:39:22.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:21.907+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:22.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:21.907+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:22.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:21.908+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:22.637 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:22.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:22.637 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:22.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:22.637 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:22.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:22.637 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:22.421+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:22.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:22.631+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:22.631846+0000) 2024-04-27T03:39:22.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:22.715+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:22.716230+0000) 2024-04-27T03:39:22.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:22.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:22.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:22.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:22.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:22.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:22.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:22.870+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:23.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:22.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:23.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:22.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:23.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:22.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:23.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:22.919+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:23.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:22.953+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:22.954683+0000) 2024-04-27T03:39:23.640 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:23.404+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:23.640 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:23.404+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:23.640 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:23.404+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:23.640 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:23.404+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:23.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:23.633+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:23.634838+0000) 2024-04-27T03:39:23.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:23.716+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:23.718471+0000) 2024-04-27T03:39:23.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:23.826+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:23.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:23.826+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:23.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:23.826+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:23.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:23.828+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:24.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:23.955+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:23.957396+0000) 2024-04-27T03:39:24.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:23.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:24.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:23.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:24.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:23.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:24.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:23.958+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:24.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:24.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:24.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:24.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:24.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:24.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:24.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:24.454+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:24.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:24.637+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:24.639057+0000) 2024-04-27T03:39:24.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:24.721+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:24.723258+0000) 2024-04-27T03:39:24.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:24.833+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:24.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:24.833+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:24.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:24.833+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:24.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:24.834+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:25.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:24.959+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:24.960972+0000) 2024-04-27T03:39:25.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:24.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:25.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:24.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:25.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:24.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:25.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:24.992+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:25.646 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:25.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:25.646 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:25.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:25.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:25.439+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:25.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:25.440+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:25.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:25.639+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:25.641406+0000) 2024-04-27T03:39:25.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:25.723+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:25.725397+0000) 2024-04-27T03:39:25.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:25.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:25.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:25.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:25.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:25.862+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:25.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:25.864+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:26.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:25.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:26.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:25.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:26.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:25.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:26.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:25.944+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:26.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:25.963+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:25.965259+0000) 2024-04-27T03:39:26.495 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:39:26.328+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68473 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:39:26.496 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:26.325+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:26.327614+0000) 2024-04-27T03:39:26.496 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:26.324+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:26.326592+0000) 2024-04-27T03:39:26.496 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:26.325+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:26.327659+0000) 2024-04-27T03:39:26.496 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:26.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:26.497 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:26.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:26.497 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:26.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:26.497 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:26.430+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:26.787 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:26.725+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:26.727466+0000) 2024-04-27T03:39:26.788 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:26.640+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:26.642326+0000) 2024-04-27T03:39:27.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:26.885+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:27.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:26.885+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:27.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:26.885+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:27.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:26.887+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:27.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:26.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:27.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:26.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:27.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:26.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:27.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:26.949+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:27.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:26.964+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:26.966587+0000) 2024-04-27T03:39:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:27.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:27.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:27.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:27.396+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:27.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:27.644+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:27.646529+0000) 2024-04-27T03:39:28.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:27.942+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:28.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:27.942+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:28.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:27.942+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:28.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:27.942+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:28.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:27.967+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:27.968838+0000) 2024-04-27T03:39:28.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:27.727+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:27.729713+0000) 2024-04-27T03:39:28.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:27.905+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:28.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:27.905+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:28.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:27.905+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:28.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:27.906+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:28.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:28.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:28.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:28.445+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:28.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:28.646+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:28.648779+0000) 2024-04-27T03:39:29.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:28.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:29.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:28.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:29.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:28.940+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:29.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:28.940+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:29.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:28.971+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:28.973550+0000) 2024-04-27T03:39:29.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:28.730+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:28.731820+0000) 2024-04-27T03:39:29.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:28.903+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:29.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:28.903+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:29.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:28.903+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:29.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:28.904+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:29.741 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:29.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:29.741 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:29.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:29.741 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:29.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:29.741 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:29.478+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:29.742 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:29.649+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:29.651479+0000) 2024-04-27T03:39:30.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:29.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:30.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:29.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:30.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:29.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:30.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:29.918+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:30.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:29.974+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:29.975860+0000) 2024-04-27T03:39:30.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:29.734+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:29.736146+0000) 2024-04-27T03:39:30.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:29.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:30.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:29.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:30.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:29.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:30.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:29.868+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:30.660 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:30.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:30.660 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:30.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:30.660 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:30.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:30.660 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:30.448+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:30.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:30.652+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:30.654538+0000) 2024-04-27T03:39:30.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:30.736+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:30.738774+0000) 2024-04-27T03:39:30.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:30.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:30.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:30.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:31.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:30.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:31.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:30.909+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:31.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:30.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:31.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:30.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:31.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:30.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:31.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:30.915+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:31.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:30.976+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:30.977962+0000) 2024-04-27T03:39:31.661 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:39:31.337+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68512 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:39:31.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:31.332+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:31.334183+0000) 2024-04-27T03:39:31.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:31.332+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:31.334117+0000) 2024-04-27T03:39:31.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:31.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:31.662 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:31.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:31.662 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:31.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:31.662 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:31.472+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:31.662 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:31.333+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:31.335281+0000) 2024-04-27T03:39:31.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:31.738+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:31.740632+0000) 2024-04-27T03:39:31.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:31.654+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:31.655830+0000) 2024-04-27T03:39:31.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:31.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:31.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:31.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:31.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:31.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:31.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:31.867+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:32.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:31.977+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:31.979064+0000) 2024-04-27T03:39:32.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:31.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:32.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:31.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:32.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:31.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:32.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:31.908+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:32.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:32.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:32.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:32.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:32.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:32.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:32.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:32.507+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:32.937 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:32.657+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:32.658991+0000) 2024-04-27T03:39:32.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:32.741+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:32.742830+0000) 2024-04-27T03:39:32.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:32.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:32.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:32.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:32.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:32.822+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:32.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:32.823+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:33.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:32.979+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:32.981177+0000) 2024-04-27T03:39:33.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:32.930+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:33.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:32.930+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:33.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:32.930+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:33.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:32.930+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:33.854 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:33.743+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:33.744996+0000) 2024-04-27T03:39:33.854 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:33.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:33.855 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:33.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:33.855 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:33.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:33.855 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:33.513+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:33.855 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:33.661+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:33.663209+0000) 2024-04-27T03:39:34.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:33.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:34.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:33.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:34.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:33.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:34.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:33.847+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:34.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:33.982+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:33.983849+0000) 2024-04-27T03:39:34.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:33.946+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:34.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:33.946+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:34.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:33.946+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:34.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:33.946+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:34.753 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:34.492+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:34.753 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:34.492+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:34.753 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:34.492+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:34.753 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:34.493+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:34.753 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:34.663+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:34.665357+0000) 2024-04-27T03:39:35.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:34.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:35.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:34.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:35.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:34.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:35.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:34.867+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:35.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:34.985+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:34.987117+0000) 2024-04-27T03:39:35.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:34.745+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:34.747667+0000) 2024-04-27T03:39:35.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:34.915+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:35.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:34.915+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:35.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:34.915+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:35.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:34.916+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:35.757 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:35.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:35.757 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:35.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:35.757 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:35.471+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:35.757 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:35.472+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:35.757 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:35.666+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:35.667578+0000) 2024-04-27T03:39:36.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:35.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:36.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:35.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:36.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:35.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:36.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:35.917+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:36.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:35.988+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:35.989592+0000) 2024-04-27T03:39:36.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:35.751+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:35.751881+0000) 2024-04-27T03:39:36.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:35.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:36.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:35.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:36.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:35.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:36.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:35.890+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:36.673 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:39:36.347+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68552 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:39:36.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:36.343+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:36.344094+0000) 2024-04-27T03:39:36.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:36.344+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:36.344946+0000) 2024-04-27T03:39:36.674 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:36.344+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:36.344889+0000) 2024-04-27T03:39:36.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:36.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:36.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:36.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:36.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:36.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:36.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:36.456+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:36.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:36.667+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:36.668567+0000) 2024-04-27T03:39:36.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:36.920+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:36.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:36.920+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:36.927 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:36.920+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:36.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:36.752+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:36.752869+0000) 2024-04-27T03:39:36.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:36.909+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:36.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:36.909+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:36.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:36.909+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:36.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:36.910+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:37.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:36.921+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:37.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:36.989+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:36.990723+0000) 2024-04-27T03:39:37.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:37.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:37.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:37.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:37.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:37.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:37.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:37.459+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:37.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:37.670+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:37.671385+0000) 2024-04-27T03:39:37.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:37.754+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:37.755537+0000) 2024-04-27T03:39:37.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:37.915+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:37.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:37.915+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:37.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:37.915+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:37.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:37.917+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:38.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:37.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:38.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:37.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:38.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:37.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:38.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:37.954+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:38.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:37.992+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:37.992931+0000) 2024-04-27T03:39:38.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:38.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:38.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:38.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:38.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:38.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:38.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:38.497+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:38.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:38.673+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:38.673819+0000) 2024-04-27T03:39:38.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:38.756+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:38.757795+0000) 2024-04-27T03:39:38.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:38.947+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:38.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:38.947+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:38.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:38.947+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:39.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:38.948+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:39.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:38.994+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:38.995058+0000) 2024-04-27T03:39:39.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:39.002+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:39.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:39.002+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:39.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:39.002+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:39.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:39.003+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:39.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:39.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:39.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:39.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:39.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:39.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:39.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:39.486+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:39.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:39.676+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:39.676926+0000) 2024-04-27T03:39:39.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:39.759+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:39.759926+0000) 2024-04-27T03:39:40.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:39.954+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:40.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:39.954+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:40.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:39.954+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:40.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:39.955+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:40.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:39.996+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:39.997391+0000) 2024-04-27T03:39:40.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:40.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:40.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:40.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:40.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:40.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:40.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:40.036+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:40.961 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:40.761+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:40.762107+0000) 2024-04-27T03:39:40.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:40.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:40.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:40.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:40.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:40.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:40.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:40.528+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:40.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:40.678+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:40.679236+0000) 2024-04-27T03:39:41.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:40.995+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:41.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:40.995+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:41.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:40.995+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:41.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:40.996+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:41.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:41.000+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:41.001643+0000) 2024-04-27T03:39:41.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:41.047+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:41.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:41.047+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:41.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:41.047+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:41.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:41.047+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:41.559 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:39:41.355+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68591 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:39:41.559 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:41.351+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:41.352010+0000) 2024-04-27T03:39:41.562 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:41.351+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:41.352760+0000) 2024-04-27T03:39:41.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:41.351+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:41.352742+0000) 2024-04-27T03:39:41.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:41.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:41.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:41.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:41.562 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:41.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:41.961 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:41.762+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:41.763601+0000) 2024-04-27T03:39:41.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:41.554+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:41.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:41.679+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:41.680557+0000) 2024-04-27T03:39:42.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:42.001+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:42.002721+0000) 2024-04-27T03:39:42.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:42.018+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:42.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:42.018+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:42.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:42.018+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:42.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:42.018+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:42.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:41.999+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:42.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:41.999+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:42.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:41.999+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:42.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:42.000+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:42.961 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:42.766+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:42.767678+0000) 2024-04-27T03:39:42.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:42.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:42.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:42.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:42.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:42.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:42.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:42.528+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:42.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:42.681+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:42.682601+0000) 2024-04-27T03:39:43.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:42.979+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:43.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:42.979+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:43.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:42.979+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:43.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:42.979+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:43.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:43.004+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:43.004811+0000) 2024-04-27T03:39:43.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:42.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:43.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:42.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:43.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:42.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:43.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:42.975+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:43.885 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:43.769+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:43.770389+0000) 2024-04-27T03:39:43.885 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:43.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:43.885 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:43.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:43.885 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:43.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:43.885 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:43.526+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:43.885 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:43.684+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:43.685766+0000) 2024-04-27T03:39:44.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:44.006+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:44.007041+0000) 2024-04-27T03:39:44.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:44.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:44.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:44.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:44.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:44.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:44.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:44.006+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:44.153 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:43.981+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:44.153 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:43.981+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:44.153 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:43.981+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:44.153 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:43.981+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:44.780 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:44.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:44.780 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:44.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:44.780 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:44.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:44.780 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:44.515+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:44.780 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:44.687+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:44.687824+0000) 2024-04-27T03:39:45.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:45.009+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:45.010314+0000) 2024-04-27T03:39:45.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:44.773+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:44.774702+0000) 2024-04-27T03:39:45.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:44.954+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:45.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:44.954+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:45.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:44.954+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:45.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:44.955+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:45.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:45.046+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:45.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:45.046+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:45.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:45.046+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:45.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:45.046+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:45.783 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:45.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:45.783 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:45.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:45.783 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:45.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:45.783 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:45.532+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:45.783 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:45.689+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:45.690363+0000) 2024-04-27T03:39:46.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:46.012+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:46.013044+0000) 2024-04-27T03:39:46.046 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:45.776+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:45.777685+0000) 2024-04-27T03:39:46.046 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:45.998+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:46.046 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:45.998+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:46.046 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:45.998+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:46.046 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:45.999+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:46.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:46.040+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:46.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:46.040+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:46.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:46.040+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:46.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:46.040+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:46.698 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:39:46.364+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68631 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:39:46.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:46.360+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:46.360947+0000) 2024-04-27T03:39:46.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:46.361+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:46.362049+0000) 2024-04-27T03:39:46.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:46.361+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:46.362643+0000) 2024-04-27T03:39:46.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:46.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:46.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:46.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:46.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:46.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:46.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:46.570+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:46.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:46.691+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:46.692240+0000) 2024-04-27T03:39:46.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:46.778+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:46.778920+0000) 2024-04-27T03:39:47.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:46.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:47.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:46.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:47.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:46.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:47.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:46.968+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:47.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:47.013+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:47.014074+0000) 2024-04-27T03:39:47.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:47.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:47.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:47.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:47.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:47.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:47.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:47.014+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:47.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:47.780+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:47.780991+0000) 2024-04-27T03:39:47.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:47.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:47.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:47.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:47.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:47.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:47.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:47.603+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:47.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:47.694+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:47.695611+0000) 2024-04-27T03:39:48.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:47.995+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:48.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:47.995+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:48.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:47.995+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:48.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:47.996+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:48.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:48.015+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:48.016569+0000) 2024-04-27T03:39:48.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:48.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:48.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:48.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:48.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:48.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:48.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:48.038+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:48.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:48.783+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:48.784075+0000) 2024-04-27T03:39:48.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:48.644+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:48.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:48.644+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:48.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:48.644+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:48.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:48.646+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:48.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:48.698+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:48.699783+0000) 2024-04-27T03:39:49.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:49.020+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:49.020826+0000) 2024-04-27T03:39:49.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:49.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:49.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:49.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:49.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:49.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:49.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:49.058+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:49.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:49.008+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:49.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:49.008+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:49.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:49.008+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:49.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:49.008+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:49.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:49.785+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:49.786186+0000) 2024-04-27T03:39:49.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:49.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:49.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:49.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:49.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:49.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:49.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:49.682+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:49.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:49.701+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:49.702008+0000) 2024-04-27T03:39:50.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:50.043+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:50.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:50.043+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:50.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:50.043+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:50.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:50.044+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:50.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:50.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:50.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:50.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:50.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:50.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:50.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:50.016+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:50.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:50.022+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:50.023057+0000) 2024-04-27T03:39:50.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:50.789+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:50.790526+0000) 2024-04-27T03:39:50.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:50.671+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:50.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:50.672+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:50.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:50.672+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:50.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:50.673+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:50.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:50.703+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:50.704171+0000) 2024-04-27T03:39:51.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:51.051+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:51.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:51.051+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:51.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:51.051+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:51.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:51.051+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:51.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:51.024+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:51.025416+0000) 2024-04-27T03:39:51.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:51.047+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:51.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:51.047+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:51.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:51.047+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:51.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:51.047+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:51.655 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:39:51.372+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68671 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:39:51.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:51.369+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:51.370165+0000) 2024-04-27T03:39:51.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:51.370+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:51.371415+0000) 2024-04-27T03:39:51.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:51.649+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:51.656 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:51.368+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:51.369464+0000) 2024-04-27T03:39:51.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:51.790+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:51.791657+0000) 2024-04-27T03:39:51.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:51.649+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:51.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:51.649+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:51.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:51.650+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:51.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:51.704+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:51.705182+0000) 2024-04-27T03:39:52.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:52.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:52.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:52.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:52.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:52.017+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:52.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:52.017+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:52.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:52.026+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:52.027322+0000) 2024-04-27T03:39:52.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:52.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:52.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:52.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:52.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:52.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:52.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:52.092+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:52.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:52.792+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:52.793777+0000) 2024-04-27T03:39:52.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:52.608+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:52.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:52.608+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:52.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:52.608+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:52.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:52.609+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:52.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:52.708+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:52.709357+0000) 2024-04-27T03:39:53.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:53.069+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:53.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:53.069+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:53.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:53.069+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:53.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:53.069+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:53.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:53.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:53.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:53.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:53.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:53.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:53.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:53.024+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:53.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:53.028+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:53.029563+0000) 2024-04-27T03:39:53.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:53.795+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:53.796449+0000) 2024-04-27T03:39:53.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:53.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:53.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:53.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:53.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:53.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:53.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:53.633+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:53.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:53.710+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:53.711481+0000) 2024-04-27T03:39:54.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:54.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:54.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:54.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:54.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:54.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:54.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:54.013+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:54.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:54.033+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:54.034300+0000) 2024-04-27T03:39:54.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:54.020+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:54.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:54.020+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:54.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:54.020+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:54.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:54.022+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:54.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:54.797+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:54.798547+0000) 2024-04-27T03:39:54.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:54.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:54.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:54.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:54.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:54.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:54.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:54.635+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:54.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:54.712+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:54.713777+0000) 2024-04-27T03:39:55.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:55.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:55.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:55.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:55.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:55.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:55.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:55.023+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:55.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:55.007+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:55.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:55.007+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:55.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:55.007+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:55.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:55.008+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:55.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:55.036+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:55.037303+0000) 2024-04-27T03:39:55.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:55.800+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:55.801593+0000) 2024-04-27T03:39:55.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:55.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:55.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:55.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:55.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:55.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:55.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:55.667+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:55.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:55.715+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:55.715925+0000) 2024-04-27T03:39:56.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:56.038+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:56.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:56.038+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:56.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:56.038+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:56.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:56.039+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:56.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:56.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:56.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:56.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:56.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:56.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:56.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:56.016+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:56.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:56.038+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:56.039427+0000) 2024-04-27T03:39:56.499 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:39:56.383+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68710 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:39:56.499 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:56.377+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:56.378328+0000) 2024-04-27T03:39:56.499 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:56.377+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:56.378515+0000) 2024-04-27T03:39:56.500 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:56.376+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:56.377258+0000) 2024-04-27T03:39:56.808 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:56.621+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:56.808 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:56.621+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:56.808 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:56.621+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:56.808 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:56.621+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:56.808 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:56.717+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:56.718203+0000) 2024-04-27T03:39:57.062 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:57.040+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:57.040823+0000) 2024-04-27T03:39:57.062 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:56.801+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:56.802641+0000) 2024-04-27T03:39:57.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:57.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:57.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:57.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:57.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:57.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:57.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:57.059+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:57.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:57.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:57.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:57.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:57.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:57.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:57.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:57.056+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:57.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:57.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:57.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:57.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:57.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:57.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:57.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:57.628+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:57.719+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:57.720345+0000) 2024-04-27T03:39:57.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:57.805+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:57.805819+0000) 2024-04-27T03:39:58.273 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:58.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:58.273 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:58.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:58.273 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:58.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:58.273 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:58.094+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:58.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:58.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:58.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:58.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:58.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:58.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:58.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:58.036+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:58.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:58.042+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:58.043144+0000) 2024-04-27T03:39:58.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:58.807+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:58.808509+0000) 2024-04-27T03:39:58.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:58.674+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:58.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:58.674+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:58.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:58.674+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:58.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:58.675+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:58.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:58.722+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:58.723331+0000) 2024-04-27T03:39:59.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:59.044+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:59.045419+0000) 2024-04-27T03:39:59.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:59.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:59.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:59.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:59.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:59.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:39:59.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:39:59.077+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:59.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:59.142+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:59.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:59.142+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:59.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:59.142+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:39:59.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:59.142+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:59.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:39:59.809+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:59.810716+0000) 2024-04-27T03:39:59.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:59.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:59.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:59.660+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:59.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:59.660+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:39:59.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:59.660+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:39:59.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:39:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:39:59.724+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:39:59.725577+0000) 2024-04-27T03:40:00.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:00.046+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:00.047703+0000) 2024-04-27T03:40:00.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:00.088+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:00.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:00.088+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:00.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:00.088+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:00.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:00.088+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:00.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:00.145+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:00.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:00.145+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:00.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:00.145+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:00.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:00.146+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:00.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:00.812+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:00.812846+0000) 2024-04-27T03:40:00.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:00.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:00.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:00.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:00.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:00.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:00.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:00.612+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:00.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:00.727+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:00.727859+0000) 2024-04-27T03:40:01.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:01.115+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:01.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:01.115+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:01.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:01.115+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:01.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:01.116+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:01.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:01.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:01.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:01.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:01.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:01.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:01.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:01.043+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:01.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:01.049+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:01.049818+0000) 2024-04-27T03:40:01.597 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:40:01.391+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68750 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:40:01.597 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:01.388+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:01.389083+0000) 2024-04-27T03:40:01.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:01.389+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:01.389950+0000) 2024-04-27T03:40:01.598 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:01.387+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:01.388747+0000) 2024-04-27T03:40:01.598 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:01.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:01.598 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:01.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:01.598 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:01.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:01.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:01.813+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:01.813900+0000) 2024-04-27T03:40:01.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:01.592+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:01.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:01.728+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:01.728900+0000) 2024-04-27T03:40:02.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:02.082+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:02.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:02.082+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:02.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:02.082+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:02.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:02.083+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:02.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:02.050+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:02.050990+0000) 2024-04-27T03:40:02.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:02.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:02.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:02.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:02.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:02.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:02.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:02.055+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:02.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:02.815+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:02.815978+0000) 2024-04-27T03:40:02.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:02.592+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:02.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:02.592+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:02.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:02.592+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:02.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:02.593+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:02.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:02.730+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:02.730965+0000) 2024-04-27T03:40:03.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:03.079+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:03.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:03.079+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:03.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:03.079+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:03.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:03.080+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:03.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:03.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:03.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:03.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:03.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:03.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:03.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:03.037+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:03.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:03.052+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:03.053190+0000) 2024-04-27T03:40:03.886 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:03.817+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:03.818590+0000) 2024-04-27T03:40:03.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:03.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:03.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:03.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:03.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:03.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:03.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:03.596+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:03.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:03.733+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:03.734095+0000) 2024-04-27T03:40:04.179 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:04.032+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:04.179 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:04.032+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:04.179 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:04.032+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:04.179 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:04.032+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:04.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:04.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:04.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:04.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:04.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:04.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:04.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:04.049+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:04.180 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:04.055+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:04.056187+0000) 2024-04-27T03:40:04.826 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:04.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:04.826 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:04.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:04.826 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:04.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:04.826 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:04.581+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:04.826 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:04.735+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:04.736317+0000) 2024-04-27T03:40:05.103 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:05.057+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:05.058298+0000) 2024-04-27T03:40:05.103 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:04.819+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:04.820780+0000) 2024-04-27T03:40:05.103 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:05.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:05.103 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:05.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:05.104 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:05.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:05.104 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:05.043+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:05.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:05.097+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:05.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:05.097+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:05.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:05.097+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:05.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:05.097+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:05.829 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:05.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:05.829 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:05.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:05.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:05.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:05.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:05.573+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:05.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:05.737+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:05.738499+0000) 2024-04-27T03:40:06.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:06.059+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:06.060491+0000) 2024-04-27T03:40:06.133 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:05.823+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:05.824349+0000) 2024-04-27T03:40:06.133 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:06.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:06.133 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:06.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:06.134 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:06.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:06.134 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:06.025+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:06.399 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:06.139+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:06.399 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:06.139+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:06.399 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:06.139+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:06.399 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:06.140+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:06.659 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:40:06.399+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68789 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:40:06.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:06.396+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:06.397761+0000) 2024-04-27T03:40:06.660 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:06.396+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:06.397421+0000) 2024-04-27T03:40:06.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:06.395+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:06.396547+0000) 2024-04-27T03:40:06.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:06.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:06.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:06.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:06.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:06.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:06.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:06.535+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:06.965 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:06.738+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:06.739562+0000) 2024-04-27T03:40:06.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:06.824+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:06.825690+0000) 2024-04-27T03:40:07.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:06.985+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:07.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:06.985+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:07.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:06.985+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:07.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:06.986+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:07.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:07.061+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:07.062498+0000) 2024-04-27T03:40:07.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:07.164+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:07.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:07.164+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:07.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:07.164+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:07.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:07.164+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:07.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:07.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:07.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:07.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:07.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:07.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:07.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:07.512+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:08.007 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:07.741+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:07.741820+0000) 2024-04-27T03:40:08.007 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:07.829+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:07.829931+0000) 2024-04-27T03:40:08.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:08.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:08.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:08.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:08.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:08.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:08.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:08.002+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:08.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:08.063+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:08.064616+0000) 2024-04-27T03:40:08.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:08.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:08.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:08.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:08.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:08.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:08.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:08.194+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:08.838 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:08.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:08.839 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:08.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:08.839 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:08.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:08.839 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:08.555+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:08.839 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:08.743+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:08.744060+0000) 2024-04-27T03:40:09.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:09.067+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:09.067927+0000) 2024-04-27T03:40:09.134 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:08.832+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:08.833255+0000) 2024-04-27T03:40:09.134 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:09.048+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:09.134 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:09.048+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:09.134 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:09.048+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:09.134 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:09.049+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:09.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:09.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:09.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:09.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:09.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:09.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:09.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:09.210+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:09.752 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:09.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:09.752 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:09.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:09.752 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:09.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:09.752 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:09.531+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:10.008 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:09.745+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:09.746699+0000) 2024-04-27T03:40:10.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:09.834+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:09.835442+0000) 2024-04-27T03:40:10.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:10.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:10.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:10.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:10.008 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:10.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:10.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:10.002+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:10.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:10.069+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:10.069998+0000) 2024-04-27T03:40:10.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:10.247+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:10.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:10.247+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:10.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:10.247+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:10.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:10.247+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:10.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:10.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:10.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:10.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:10.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:10.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:10.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:10.514+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:11.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:10.837+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:10.838298+0000) 2024-04-27T03:40:11.045 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:10.748+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:10.749028+0000) 2024-04-27T03:40:11.303 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:11.039+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:11.303 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:11.039+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:11.303 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:11.039+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:11.303 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:11.040+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:11.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:11.071+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:11.072682+0000) 2024-04-27T03:40:11.304 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:11.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:11.304 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:11.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:11.304 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:11.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:11.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:11.405+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:11.406605+0000) 2024-04-27T03:40:11.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:40:11.407+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68830 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:40:11.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:11.297+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:11.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:11.404+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:11.405438+0000) 2024-04-27T03:40:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:11.404+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:11.405524+0000) 2024-04-27T03:40:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:11.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:11.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:11.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:11.495+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:12.020 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:11.749+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:11.750228+0000) 2024-04-27T03:40:12.021 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:11.839+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:11.839916+0000) 2024-04-27T03:40:12.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:12.073+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:12.073828+0000) 2024-04-27T03:40:12.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:12.014+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:12.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:12.014+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:12.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:12.014+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:12.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:12.015+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:12.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:12.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:12.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:12.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:12.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:12.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:12.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:12.497+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:12.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:12.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:12.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:12.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:12.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:12.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:12.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:12.344+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:13.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:12.751+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:12.752270+0000) 2024-04-27T03:40:13.068 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:12.841+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:12.842112+0000) 2024-04-27T03:40:13.068 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:12.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:13.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:12.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:13.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:12.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:13.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:12.975+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:13.394 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:13.075+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:13.075845+0000) 2024-04-27T03:40:13.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:13.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:13.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:13.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:13.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:13.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:13.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:13.497+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:13.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:13.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:13.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:13.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:13.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:13.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:13.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:13.389+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:14.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:13.755+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:13.756509+0000) 2024-04-27T03:40:14.068 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:13.845+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:13.846418+0000) 2024-04-27T03:40:14.068 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:13.980+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:14.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:13.980+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:14.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:13.980+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:14.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:13.980+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:14.360 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:14.079+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:14.080073+0000) 2024-04-27T03:40:14.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:14.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:14.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:14.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:14.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:14.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:14.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:14.461+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:14.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:14.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:14.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:14.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:14.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:14.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:14.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:14.354+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:15.028 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:14.758+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:14.759330+0000) 2024-04-27T03:40:15.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:14.849+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:14.850770+0000) 2024-04-27T03:40:15.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:15.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:15.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:15.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:15.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:15.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:15.336 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:15.023+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:15.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:15.082+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:15.082844+0000) 2024-04-27T03:40:15.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:15.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:15.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:15.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:15.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:15.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:15.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:15.423+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:15.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:15.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:15.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:15.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:15.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:15.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:15.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:15.330+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:16.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:15.760+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:15.761452+0000) 2024-04-27T03:40:16.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:15.854+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:15.855555+0000) 2024-04-27T03:40:16.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:16.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:16.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:16.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:16.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:16.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:16.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:16.002+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:16.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:16.085+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:16.086035+0000) 2024-04-27T03:40:16.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:40:16.415+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68864 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:40:16.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:16.411+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:16.412783+0000) 2024-04-27T03:40:16.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:16.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:16.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:16.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:16.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:16.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:16.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:16.387+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:16.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:16.411+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:16.412659+0000) 2024-04-27T03:40:16.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:16.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:16.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:16.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:16.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:16.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:16.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:16.374+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:16.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:16.412+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:16.413516+0000) 2024-04-27T03:40:17.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:16.762+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:16.763427+0000) 2024-04-27T03:40:17.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:16.855+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:16.856579+0000) 2024-04-27T03:40:17.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:17.087+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:17.088056+0000) 2024-04-27T03:40:17.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:17.032+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:17.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:17.032+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:17.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:17.032+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:17.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:17.033+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:17.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:17.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:17.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:17.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:17.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:17.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:17.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:17.413+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:17.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:17.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:17.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:17.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:17.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:17.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:17.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:17.390+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:18.025 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:17.765+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:17.766352+0000) 2024-04-27T03:40:18.026 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:17.858+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:17.859610+0000) 2024-04-27T03:40:18.026 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:18.018+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:18.026 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:18.018+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:18.026 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:18.018+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:18.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:18.019+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:18.384 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:18.089+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:18.090205+0000) 2024-04-27T03:40:18.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:18.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:18.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:18.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:18.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:18.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:18.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:18.437+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:18.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:18.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:18.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:18.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:18.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:18.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:18.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:18.378+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:19.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:18.768+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:18.769025+0000) 2024-04-27T03:40:19.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:18.860+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:18.861791+0000) 2024-04-27T03:40:19.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:19.092+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:19.093659+0000) 2024-04-27T03:40:19.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:19.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:19.346 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:19.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:19.346 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:19.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:19.346 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:19.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:19.346 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:19.067+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:19.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:19.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:19.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:19.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:19.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:19.339+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:19.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:19.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:19.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:19.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:19.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:19.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:19.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:19.456+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:20.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:19.771+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:19.772152+0000) 2024-04-27T03:40:20.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:19.863+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:19.864551+0000) 2024-04-27T03:40:20.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:20.019+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:20.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:20.019+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:20.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:20.019+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:20.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:20.020+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:20.326 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:20.095+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:20.096486+0000) 2024-04-27T03:40:20.327 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:20.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:20.327 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:20.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:20.327 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:20.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:20.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:20.321+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:20.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:20.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:20.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:20.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:20.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:20.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:20.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:20.447+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:21.053 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:20.773+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:20.774795+0000) 2024-04-27T03:40:21.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:20.866+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:20.866904+0000) 2024-04-27T03:40:21.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:21.046+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:21.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:21.046+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:21.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:21.098+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:21.099722+0000) 2024-04-27T03:40:21.365 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:21.046+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:21.365 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:21.047+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:21.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:40:21.424+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68905 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:40:21.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:21.420+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:21.421239+0000) 2024-04-27T03:40:21.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:21.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:21.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:21.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:21.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:21.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:21.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:21.488+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:21.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:21.420+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:21.421114+0000) 2024-04-27T03:40:21.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:21.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:21.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:21.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:21.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:21.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:21.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:21.359+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:21.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:21.420+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:21.421485+0000) 2024-04-27T03:40:22.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:21.775+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:21.776428+0000) 2024-04-27T03:40:22.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:21.868+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:21.868966+0000) 2024-04-27T03:40:22.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:22.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:22.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:22.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:22.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:22.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:22.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:22.014+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:22.329 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:22.099+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:22.100790+0000) 2024-04-27T03:40:22.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:22.323+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:22.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:22.323+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:22.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:22.323+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:22.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:22.324+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:22.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:22.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:22.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:22.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:22.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:22.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:22.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:22.537+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:23.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:22.777+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:22.778669+0000) 2024-04-27T03:40:23.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:22.872+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:22.873298+0000) 2024-04-27T03:40:23.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:22.997+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:23.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:22.997+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:23.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:22.997+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:23.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:22.997+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:23.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:23.102+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:23.103071+0000) 2024-04-27T03:40:23.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:23.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:23.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:23.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:23.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:23.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:23.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:23.312+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:23.787 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:23.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:23.787 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:23.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:23.787 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:23.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:23.787 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:23.514+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:24.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:23.781+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:23.781788+0000) 2024-04-27T03:40:24.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:23.874+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:23.875541+0000) 2024-04-27T03:40:24.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:24.006+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:24.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:24.006+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:24.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:24.006+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:24.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:24.007+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:24.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:24.106+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:24.107172+0000) 2024-04-27T03:40:24.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:24.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:24.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:24.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:24.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:24.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:24.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:24.265+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:24.789 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:24.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:24.789 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:24.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:24.790 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:24.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:24.790 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:24.520+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:25.060 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:24.783+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:24.784290+0000) 2024-04-27T03:40:25.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:24.876+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:24.877639+0000) 2024-04-27T03:40:25.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:24.980+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:25.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:24.980+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:25.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:24.980+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:25.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:24.981+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:25.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:25.110+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:25.111463+0000) 2024-04-27T03:40:25.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:25.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:25.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:25.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:25.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:25.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:25.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:25.215+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:25.791 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:25.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:25.792 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:25.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:25.792 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:25.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:25.792 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:25.510+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:26.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:25.784+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:25.786496+0000) 2024-04-27T03:40:26.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:25.878+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:25.879823+0000) 2024-04-27T03:40:26.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:25.991+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:26.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:25.991+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:26.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:25.991+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:26.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:25.991+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:26.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:26.113+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:26.115758+0000) 2024-04-27T03:40:26.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:26.192+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:26.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:26.192+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:26.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:26.192+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:26.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:26.193+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:26.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:26.427+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:26.429455+0000) 2024-04-27T03:40:26.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:26.430+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:26.432237+0000) 2024-04-27T03:40:26.727 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:40:26.431+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68943 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:40:26.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:26.427+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:26.429067+0000) 2024-04-27T03:40:26.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:26.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:26.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:26.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:26.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:26.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:26.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:26.539+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:27.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:26.786+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:26.788525+0000) 2024-04-27T03:40:27.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:26.879+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:26.880849+0000) 2024-04-27T03:40:27.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:27.036+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:27.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:27.036+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:27.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:27.036+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:27.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:27.037+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:27.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:27.115+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:27.117448+0000) 2024-04-27T03:40:27.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:27.196+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:27.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:27.196+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:27.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:27.196+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:27.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:27.196+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:27.796 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:27.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:27.796 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:27.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:27.796 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:27.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:27.797 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:27.565+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:28.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:27.789+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:27.791356+0000) 2024-04-27T03:40:28.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:27.881+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:27.882914+0000) 2024-04-27T03:40:28.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:28.119+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:28.121354+0000) 2024-04-27T03:40:28.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:28.211+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:28.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:28.211+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:28.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:28.211+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:28.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:28.212+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:28.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:28.077+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:28.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:28.077+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:28.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:28.077+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:28.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:28.078+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:28.800 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:28.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:28.801 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:28.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:28.801 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:28.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:28.801 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:28.548+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:29.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:28.793+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:28.795524+0000) 2024-04-27T03:40:29.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:28.884+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:28.886123+0000) 2024-04-27T03:40:29.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:29.072+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:29.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:29.072+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:29.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:29.072+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:29.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:29.073+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:29.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:29.121+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:29.123524+0000) 2024-04-27T03:40:29.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:29.195+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:29.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:29.195+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:29.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:29.195+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:29.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:29.196+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:29.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:29.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:29.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:29.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:29.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:29.513+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:29.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:29.514+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:30.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:29.796+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:29.798509+0000) 2024-04-27T03:40:30.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:29.886+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:29.888266+0000) 2024-04-27T03:40:30.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:30.105+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:30.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:30.105+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:30.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:30.105+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:30.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:30.105+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:30.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:30.126+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:30.127791+0000) 2024-04-27T03:40:30.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:30.229+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:30.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:30.229+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:30.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:30.229+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:30.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:30.230+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:30.806 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:30.562+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:30.807 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:30.562+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:30.807 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:30.562+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:30.807 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:30.563+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:31.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:30.799+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:30.801562+0000) 2024-04-27T03:40:31.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:30.888+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:30.890424+0000) 2024-04-27T03:40:31.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:31.128+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:31.130513+0000) 2024-04-27T03:40:31.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:31.195+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:31.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:31.195+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:31.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:31.195+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:31.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:31.195+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:31.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:31.116+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:31.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:31.116+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:31.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:31.116+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:31.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:31.116+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:31.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:40:31.439+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 68984 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:40:31.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:31.436+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:31.437833+0000) 2024-04-27T03:40:31.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:31.436+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:31.438039+0000) 2024-04-27T03:40:31.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:31.437+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:31.438868+0000) 2024-04-27T03:40:31.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:31.573+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:31.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:31.573+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:31.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:31.573+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:31.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:31.574+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:32.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:31.801+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:31.802849+0000) 2024-04-27T03:40:32.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:31.890+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:31.892045+0000) 2024-04-27T03:40:32.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:32.155+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:32.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:32.155+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:32.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:32.155+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:32.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:32.156+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:32.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:32.129+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:32.131791+0000) 2024-04-27T03:40:32.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:32.176+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:32.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:32.176+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:32.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:32.176+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:32.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:32.176+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:32.810 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:32.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:32.810 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:32.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:32.810 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:32.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:32.811 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:32.531+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:32.811 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:32.803+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:32.805005+0000) 2024-04-27T03:40:33.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:32.893+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:32.895144+0000) 2024-04-27T03:40:33.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:33.114+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:33.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:33.114+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:33.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:33.114+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:33.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:33.115+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:33.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:33.134+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:33.136545+0000) 2024-04-27T03:40:33.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:33.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:33.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:33.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:33.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:33.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:33.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:33.184+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:33.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:33.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:33.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:33.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:33.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:33.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:33.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:33.550+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:34.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:33.807+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:33.809252+0000) 2024-04-27T03:40:34.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:33.898+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:33.899927+0000) 2024-04-27T03:40:34.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:34.136+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:34.138692+0000) 2024-04-27T03:40:34.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:34.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:34.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:34.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:34.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:34.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:34.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:34.179+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:34.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:34.133+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:34.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:34.133+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:34.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:34.133+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:34.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:34.134+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:34.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:34.560+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:34.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:34.560+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:34.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:34.560+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:34.819 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:34.561+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:34.811+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:34.813427+0000) 2024-04-27T03:40:35.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:34.901+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:34.903110+0000) 2024-04-27T03:40:35.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:35.141+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:35.142966+0000) 2024-04-27T03:40:35.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:35.183+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:35.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:35.183+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:35.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:35.183+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:35.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:35.183+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:35.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:35.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:35.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:35.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:35.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:35.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:35.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:35.099+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:35.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:35.523+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:35.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:35.523+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:35.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:35.523+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:35.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:35.525+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:36.119 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:35.813+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:35.815683+0000) 2024-04-27T03:40:36.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:35.906+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:35.907878+0000) 2024-04-27T03:40:36.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:36.112+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:36.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:36.112+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:36.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:36.112+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:36.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:36.113+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:36.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:36.143+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:36.145645+0000) 2024-04-27T03:40:36.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:36.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:36.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:36.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:36.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:36.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:36.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:36.204+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:36.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:36.441+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:36.447026+0000) 2024-04-27T03:40:36.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:40:36.446+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69022 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:40:36.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:36.441+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:36.446447+0000) 2024-04-27T03:40:36.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:36.441+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:36.447211+0000) 2024-04-27T03:40:36.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:36.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:36.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:36.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:36.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:36.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:36.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:36.563+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:37.111 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:36.815+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:36.817035+0000) 2024-04-27T03:40:37.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:36.908+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:36.909535+0000) 2024-04-27T03:40:37.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:37.104+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:37.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:37.104+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:37.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:37.104+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:37.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:37.105+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:37.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:37.145+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:37.146820+0000) 2024-04-27T03:40:37.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:37.231+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:37.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:37.231+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:37.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:37.231+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:37.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:37.232+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:37.824 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:37.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:37.824 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:37.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:37.824 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:37.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:37.824 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:37.518+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:38.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:37.817+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:37.819105+0000) 2024-04-27T03:40:38.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:37.911+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:37.912050+0000) 2024-04-27T03:40:38.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:38.110+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:38.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:38.110+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:38.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:38.110+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:38.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:38.111+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:38.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:38.148+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:38.148913+0000) 2024-04-27T03:40:38.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:38.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:38.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:38.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:38.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:38.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:38.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:38.209+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:38.827 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:38.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:38.827 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:38.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:38.828 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:38.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:38.828 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:38.546+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:39.137 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:38.821+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:38.821896+0000) 2024-04-27T03:40:39.137 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:38.913+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:38.914254+0000) 2024-04-27T03:40:39.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:39.155+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:39.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:39.155+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:39.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:39.155+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:39.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:39.156+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:39.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:39.150+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:39.151063+0000) 2024-04-27T03:40:39.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:39.192+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:39.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:39.192+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:39.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:39.192+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:39.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:39.193+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:39.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:39.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:39.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:39.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:39.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:39.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:39.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:39.502+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:40.118 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:39.823+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:39.824498+0000) 2024-04-27T03:40:40.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:39.916+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:39.917211+0000) 2024-04-27T03:40:40.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:40.112+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:40.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:40.112+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:40.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:40.112+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:40.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:40.113+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:40.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:40.153+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:40.153851+0000) 2024-04-27T03:40:40.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:40.212+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:40.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:40.212+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:40.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:40.212+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:40.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:40.212+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:40.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:40.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:40.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:40.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:40.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:40.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:40.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:40.455+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:41.137 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:40.826+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:40.827482+0000) 2024-04-27T03:40:41.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:40.918+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:40.919392+0000) 2024-04-27T03:40:41.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:41.069+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:41.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:41.069+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:41.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:41.069+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:41.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:41.070+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:41.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:41.156+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:41.157144+0000) 2024-04-27T03:40:41.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:41.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:41.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:41.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:41.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:41.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:41.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:41.254+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:41.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:41.453+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:41.454397+0000) 2024-04-27T03:40:41.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:40:41.457+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69063 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:40:41.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:41.453+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:41.454110+0000) 2024-04-27T03:40:41.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:41.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:41.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:41.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:41.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:41.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:41.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:41.436+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:41.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:41.453+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:41.454677+0000) 2024-04-27T03:40:42.118 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:41.828+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:41.829601+0000) 2024-04-27T03:40:42.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:41.919+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:41.920570+0000) 2024-04-27T03:40:42.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:42.112+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:42.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:42.112+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:42.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:42.112+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:42.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:42.113+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:42.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:42.157+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:42.158292+0000) 2024-04-27T03:40:42.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:42.263+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:42.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:42.263+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:42.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:42.263+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:42.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:42.264+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:42.837 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:42.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:42.838 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:42.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:42.838 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:42.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:42.838 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:42.473+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:43.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:42.831+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:42.832496+0000) 2024-04-27T03:40:43.104 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:42.924+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:42.924869+0000) 2024-04-27T03:40:43.104 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:43.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:43.104 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:43.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:43.104 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:43.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:43.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:43.098+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:43.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:43.161+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:43.162558+0000) 2024-04-27T03:40:43.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:43.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:43.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:43.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:43.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:43.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:43.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:43.232+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:43.840 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:43.522+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:43.840 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:43.522+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:43.840 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:43.522+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:43.840 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:43.523+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:44.100 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:43.926+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:43.926991+0000) 2024-04-27T03:40:44.100 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:43.833+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:43.834735+0000) 2024-04-27T03:40:44.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:44.163+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:44.164708+0000) 2024-04-27T03:40:44.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:44.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:44.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:44.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:44.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:44.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:44.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:44.205+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:44.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:44.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:44.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:44.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:44.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:44.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:44.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:44.135+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:44.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:44.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:44.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:44.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:44.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:44.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:44.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:44.555+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:44.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:44.837+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:44.837826+0000) 2024-04-27T03:40:45.137 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:44.930+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:44.931332+0000) 2024-04-27T03:40:45.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:45.176+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:45.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:45.176+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:45.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:45.176+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:45.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:45.177+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:45.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:45.167+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:45.167944+0000) 2024-04-27T03:40:45.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:45.222+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:45.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:45.222+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:45.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:45.222+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:45.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:45.223+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:45.846 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:45.577+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:45.846 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:45.577+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:45.846 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:45.577+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:45.846 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:45.578+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:46.138 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:45.839+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:45.840585+0000) 2024-04-27T03:40:46.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:45.932+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:45.933596+0000) 2024-04-27T03:40:46.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:46.180+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:46.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:46.180+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:46.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:46.180+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:46.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:46.181+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:46.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:46.169+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:46.170724+0000) 2024-04-27T03:40:46.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:46.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:46.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:46.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:46.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:46.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:46.466 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:46.254+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:46.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:40:46.465+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69101 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:40:46.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:46.461+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:46.462345+0000) 2024-04-27T03:40:46.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:46.462+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:46.463061+0000) 2024-04-27T03:40:46.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:46.462+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:46.463238+0000) 2024-04-27T03:40:46.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:46.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:46.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:46.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:46.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:46.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:46.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:46.596+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:47.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:46.934+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:46.935782+0000) 2024-04-27T03:40:47.138 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:46.841+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:46.842289+0000) 2024-04-27T03:40:47.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:47.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:47.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:47.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:47.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:47.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:47.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:47.166+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:47.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:47.171+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:47.172438+0000) 2024-04-27T03:40:47.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:47.229+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:47.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:47.229+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:47.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:47.229+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:47.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:47.230+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:47.851 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:47.623+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:47.852 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:47.623+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:47.852 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:47.623+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:47.852 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:47.624+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:48.138 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:47.845+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:47.846473+0000) 2024-04-27T03:40:48.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:47.938+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:47.939020+0000) 2024-04-27T03:40:48.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:48.183+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:48.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:48.183+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:48.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:48.183+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:48.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:48.184+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:48.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:48.176+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:48.177139+0000) 2024-04-27T03:40:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:48.277+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:48.277+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:48.277+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:48.278+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:48.854 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:48.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:48.854 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:48.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:48.854 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:48.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:48.854 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:48.597+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:49.138 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:48.848+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:48.848904+0000) 2024-04-27T03:40:49.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:48.943+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:48.943815+0000) 2024-04-27T03:40:49.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:49.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:49.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:49.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:49.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:49.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:49.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:49.211+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:49.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:49.181+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:49.181923+0000) 2024-04-27T03:40:49.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:49.299+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:49.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:49.299+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:49.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:49.299+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:49.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:49.300+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:49.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:49.576+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:49.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:49.576+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:49.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:49.576+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:49.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:49.577+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:50.138 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:49.852+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:49.852995+0000) 2024-04-27T03:40:50.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:49.945+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:49.945930+0000) 2024-04-27T03:40:50.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:50.199+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:50.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:50.199+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:50.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:50.199+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:50.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:50.199+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:50.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:50.184+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:50.184836+0000) 2024-04-27T03:40:50.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:50.288+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:50.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:50.288+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:50.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:50.288+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:50.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:50.288+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:50.861 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:50.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:50.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:50.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:50.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:50.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:50.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:50.612+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:51.138 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:50.855+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:50.856107+0000) 2024-04-27T03:40:51.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:50.947+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:50.948083+0000) 2024-04-27T03:40:51.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:51.187+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:51.188459+0000) 2024-04-27T03:40:51.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:51.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:51.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:51.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:51.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:51.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:51.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:51.335+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:51.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:51.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:51.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:51.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:51.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:51.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:51.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:51.171+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:51.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:40:51.474+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69143 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:40:51.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:51.470+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:51.471204+0000) 2024-04-27T03:40:51.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:51.471+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:51.472348+0000) 2024-04-27T03:40:51.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:51.472+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:51.473025+0000) 2024-04-27T03:40:51.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:51.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:51.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:51.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:51.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:51.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:51.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:51.568+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:52.130 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:51.857+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:51.857941+0000) 2024-04-27T03:40:52.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:51.948+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:51.949168+0000) 2024-04-27T03:40:52.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:52.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:52.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:52.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:52.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:52.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:52.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:52.126+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:52.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:52.188+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:52.189656+0000) 2024-04-27T03:40:52.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:52.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:52.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:52.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:52.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:52.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:52.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:52.373+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:52.865 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:52.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:52.865 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:52.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:52.865 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:52.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:52.865 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:52.607+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:53.138 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:52.859+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:52.860079+0000) 2024-04-27T03:40:53.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:52.951+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:52.951833+0000) 2024-04-27T03:40:53.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:53.105+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:53.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:53.105+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:53.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:53.105+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:53.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:53.105+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:53.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:53.192+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:53.192839+0000) 2024-04-27T03:40:53.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:53.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:53.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:53.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:53.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:53.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:53.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:53.326+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:53.869 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:53.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:53.869 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:53.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:53.869 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:53.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:53.869 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:53.570+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:54.139 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:53.863+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:53.864008+0000) 2024-04-27T03:40:54.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:53.953+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:53.954665+0000) 2024-04-27T03:40:54.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:54.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:54.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:54.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:54.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:54.134+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:54.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:54.134+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:54.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:54.196+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:54.197557+0000) 2024-04-27T03:40:54.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:54.296+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:54.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:54.296+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:54.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:54.296+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:54.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:54.297+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:54.873 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:54.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:54.873 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:54.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:54.874 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:54.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:54.874 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:54.602+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:55.139 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:54.867+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:54.868234+0000) 2024-04-27T03:40:55.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:54.958+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:54.959419+0000) 2024-04-27T03:40:55.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:55.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:55.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:55.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:55.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:55.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:55.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:55.151+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:55.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:55.201+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:55.202366+0000) 2024-04-27T03:40:55.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:55.261+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:55.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:55.261+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:55.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:55.261+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:55.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:55.261+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:55.811 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:55.650+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:55.811 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:55.650+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:55.811 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:55.650+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:55.811 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:55.650+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:56.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:55.871+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:55.872574+0000) 2024-04-27T03:40:56.073 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:55.961+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:55.962415+0000) 2024-04-27T03:40:56.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:56.204+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:56.205329+0000) 2024-04-27T03:40:56.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:56.275+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:56.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:56.275+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:56.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:56.275+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:56.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:56.276+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:56.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:56.198+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:56.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:56.198+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:56.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:56.198+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:56.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:56.198+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:56.647 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:40:56.483+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69181 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:40:56.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:56.479+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:56.480624+0000) 2024-04-27T03:40:56.648 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:56.479+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:56.480263+0000) 2024-04-27T03:40:56.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:56.480+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:56.480927+0000) 2024-04-27T03:40:56.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:56.641+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:56.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:56.641+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:56.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:56.641+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:56.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:56.642+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:56.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:56.873+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:56.873822+0000) 2024-04-27T03:40:57.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:57.205+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:57.206602+0000) 2024-04-27T03:40:57.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:56.963+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:56.964667+0000) 2024-04-27T03:40:57.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:57.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:57.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:57.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:57.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:57.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:57.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:57.190+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:57.506 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:57.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:57.506 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:57.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:57.506 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:57.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:57.506 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:57.276+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:57.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:57.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:57.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:57.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:57.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:57.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:57.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:57.625+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:58.139 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:57.876+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:57.876983+0000) 2024-04-27T03:40:58.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:57.969+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:57.970224+0000) 2024-04-27T03:40:58.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:58.157+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:58.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:58.157+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:58.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:58.157+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:58.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:58.158+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:58.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:58.210+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:58.211584+0000) 2024-04-27T03:40:58.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:58.322+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:58.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:58.322+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:58.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:58.322+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:58.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:58.322+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:58.884 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:58.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:58.884 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:58.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:58.884 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:58.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:58.885 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:58.662+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:59.139 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:58.878+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:58.879255+0000) 2024-04-27T03:40:59.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:58.971+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:58.972349+0000) 2024-04-27T03:40:59.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:59.213+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:59.213885+0000) 2024-04-27T03:40:59.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:59.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:59.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:59.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:59.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:59.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:40:59.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:40:59.348+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:59.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:59.168+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:59.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:59.168+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:59.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:59.168+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:40:59.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:59.170+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:40:59.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:59.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:59.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:59.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:59.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:59.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:40:59.888 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:59.708+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:00.139 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:40:59.881+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:59.882243+0000) 2024-04-27T03:41:00.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:40:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:40:59.973+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:40:59.974529+0000) 2024-04-27T03:41:00.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:00.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:00.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:00.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:00.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:00.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:00.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:00.124+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:00.420 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:00.217+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:00.218111+0000) 2024-04-27T03:41:00.420 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:00.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:00.420 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:00.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:00.420 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:00.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:00.420 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:00.335+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:01.073 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:00.975+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:00.976714+0000) 2024-04-27T03:41:01.073 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:00.729+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:01.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:00.729+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:01.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:00.729+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:01.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:00.730+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:01.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:00.885+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:00.886259+0000) 2024-04-27T03:41:01.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:01.219+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:01.220325+0000) 2024-04-27T03:41:01.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:01.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:01.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:01.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:01.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:01.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:01.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:01.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:01.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:01.132+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:01.689 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:41:01.492+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69222 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:41:01.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:01.489+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:01.489965+0000) 2024-04-27T03:41:01.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:01.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:01.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:01.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:01.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:01.382+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:01.690 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:01.488+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:01.488981+0000) 2024-04-27T03:41:01.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:01.488+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:01.489306+0000) 2024-04-27T03:41:01.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:01.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:01.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:01.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:01.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:01.683+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:01.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:01.684+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:01.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:01.886+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:01.887407+0000) 2024-04-27T03:41:02.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:01.977+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:01.978046+0000) 2024-04-27T03:41:02.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:02.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:02.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:02.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:02.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:02.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:02.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:02.166+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:02.506 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:02.220+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:02.221619+0000) 2024-04-27T03:41:02.506 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:02.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:02.506 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:02.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:02.506 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:02.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:02.506 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:02.367+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:02.894 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:02.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:02.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:02.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:02.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:02.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:02.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:02.648+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:03.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:02.888+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:02.889432+0000) 2024-04-27T03:41:03.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:02.981+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:02.982331+0000) 2024-04-27T03:41:03.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:03.125+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:03.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:03.125+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:03.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:03.125+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:03.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:03.126+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:03.506 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:03.223+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:03.224778+0000) 2024-04-27T03:41:03.506 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:03.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:03.506 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:03.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:03.506 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:03.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:03.507 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:03.402+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:03.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:03.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:03.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:03.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:03.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:03.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:03.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:03.667+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:04.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:03.891+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:03.892364+0000) 2024-04-27T03:41:04.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:03.986+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:03.987059+0000) 2024-04-27T03:41:04.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:04.090+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:04.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:04.090+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:04.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:04.090+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:04.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:04.090+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:04.506 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:04.226+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:04.227549+0000) 2024-04-27T03:41:04.506 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:04.448+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:04.506 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:04.448+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:04.507 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:04.448+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:04.507 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:04.449+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:04.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:04.671+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:04.902 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:04.671+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:04.902 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:04.671+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:04.902 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:04.672+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:05.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:04.895+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:04.896422+0000) 2024-04-27T03:41:05.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:04.988+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:04.989631+0000) 2024-04-27T03:41:05.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:05.083+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:05.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:05.083+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:05.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:05.083+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:05.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:05.084+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:05.506 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:05.231+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:05.232340+0000) 2024-04-27T03:41:05.507 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:05.431+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:05.507 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:05.431+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:05.507 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:05.431+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:05.507 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:05.431+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:05.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:05.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:05.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:05.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:05.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:05.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:05.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:05.658+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:06.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:05.899+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:05.900551+0000) 2024-04-27T03:41:06.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:05.993+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:05.993926+0000) 2024-04-27T03:41:06.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:06.056+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:06.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:06.056+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:06.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:06.056+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:06.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:06.057+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:06.499 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:06.236+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:06.237121+0000) 2024-04-27T03:41:06.500 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:06.457+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:06.500 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:06.457+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:06.500 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:06.457+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:06.500 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:06.457+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:06.907 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:41:06.500+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69260 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:41:06.907 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:06.497+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:06.498634+0000) 2024-04-27T03:41:06.908 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:06.497+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:06.497940+0000) 2024-04-27T03:41:06.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:06.496+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:06.497123+0000) 2024-04-27T03:41:06.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:06.670+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:06.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:06.670+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:06.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:06.670+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:06.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:06.672+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:06.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:06.901+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:06.901848+0000) 2024-04-27T03:41:07.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:06.994+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:06.995211+0000) 2024-04-27T03:41:07.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:07.015+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:07.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:07.015+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:07.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:07.015+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:07.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:07.016+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:07.506 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:07.238+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:07.239744+0000) 2024-04-27T03:41:07.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:07.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:07.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:07.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:07.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:07.500+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:07.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:07.500+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:07.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:07.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:07.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:07.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:07.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:07.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:07.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:07.712+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:08.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:07.904+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:07.905765+0000) 2024-04-27T03:41:08.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:07.997+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:07.997847+0000) 2024-04-27T03:41:08.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:08.046+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:08.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:08.046+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:08.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:08.046+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:08.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:08.046+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:08.507 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:08.242+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:08.243137+0000) 2024-04-27T03:41:08.507 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:08.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:08.507 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:08.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:08.507 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:08.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:08.507 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:08.470+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:08.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:08.734+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:08.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:08.734+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:08.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:08.734+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:08.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:08.735+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:09.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:08.907+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:08.908083+0000) 2024-04-27T03:41:09.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:09.000+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:09.001073+0000) 2024-04-27T03:41:09.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:09.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:09.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:09.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:09.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:09.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:09.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:09.060+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:09.507 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:09.247+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:09.248269+0000) 2024-04-27T03:41:09.507 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:09.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:09.507 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:09.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:09.507 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:09.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:09.507 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:09.457+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:10.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:09.782+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:10.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:09.782+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:10.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:09.782+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:10.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:09.783+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:10.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:09.910+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:09.911214+0000) 2024-04-27T03:41:10.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:10.003+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:10.003898+0000) 2024-04-27T03:41:10.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:10.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:10.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:10.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:10.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:10.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:10.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:10.016+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:10.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:10.252+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:10.253312+0000) 2024-04-27T03:41:10.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:10.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:10.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:10.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:10.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:10.455+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:10.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:10.456+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:11.074 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:11.005+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:11.005961+0000) 2024-04-27T03:41:11.074 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:11.031+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:11.074 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:11.031+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:11.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:11.031+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:11.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:11.032+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:11.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:10.807+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:11.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:10.807+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:11.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:10.807+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:11.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:10.809+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:11.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:10.913+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:10.913900+0000) 2024-04-27T03:41:11.456 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:11.257+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:11.258336+0000) 2024-04-27T03:41:11.456 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:11.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:11.456 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:11.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:11.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:41:11.508+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69301 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:41:11.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:11.505+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:11.505998+0000) 2024-04-27T03:41:11.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:11.505+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:11.506168+0000) 2024-04-27T03:41:11.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:11.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:11.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:11.450+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:11.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:11.504+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:11.505437+0000) 2024-04-27T03:41:12.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:11.858+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:12.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:11.858+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:12.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:11.858+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:12.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:11.859+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:12.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:11.914+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:11.915075+0000) 2024-04-27T03:41:12.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:11.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:12.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:11.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:12.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:11.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:12.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:11.997+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:12.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:12.006+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:12.007654+0000) 2024-04-27T03:41:12.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:12.260+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:12.261084+0000) 2024-04-27T03:41:12.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:12.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:12.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:12.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:12.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:12.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:12.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:12.417+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:13.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:12.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:13.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:12.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:13.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:12.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:13.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:12.852+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:13.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:12.918+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:12.919302+0000) 2024-04-27T03:41:13.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:13.009+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:13.009993+0000) 2024-04-27T03:41:13.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:13.032+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:13.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:13.032+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:13.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:13.032+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:13.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:13.033+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:13.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:13.263+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:13.263912+0000) 2024-04-27T03:41:13.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:13.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:13.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:13.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:13.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:13.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:13.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:13.414+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:14.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:13.859+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:14.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:13.859+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:14.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:13.859+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:14.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:13.860+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:14.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:13.921+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:13.921946+0000) 2024-04-27T03:41:14.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:14.011+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:14.012247+0000) 2024-04-27T03:41:14.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:14.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:14.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:14.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:14.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:14.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:14.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:14.066+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:14.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:14.266+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:14.267166+0000) 2024-04-27T03:41:14.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:14.385+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:14.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:14.385+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:14.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:14.385+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:14.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:14.386+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:15.103 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:15.014+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:15.015205+0000) 2024-04-27T03:41:15.103 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:15.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:15.103 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:15.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:15.103 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:15.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:15.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:14.829+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:15.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:14.829+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:15.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:14.829+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:15.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:14.830+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:15.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:14.924+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:14.925077+0000) 2024-04-27T03:41:15.369 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:15.098+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:15.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:15.268+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:15.269472+0000) 2024-04-27T03:41:15.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:15.362+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:15.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:15.362+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:15.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:15.362+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:15.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:15.363+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:16.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:15.814+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:16.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:15.814+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:16.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:15.815+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:16.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:15.816+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:16.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:15.926+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:15.927736+0000) 2024-04-27T03:41:16.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:16.017+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:16.017860+0000) 2024-04-27T03:41:16.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:16.049+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:16.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:16.049+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:16.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:16.049+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:16.077 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:16.050+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:16.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:16.271+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:16.272375+0000) 2024-04-27T03:41:16.642 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:41:16.517+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69339 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:41:16.643 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:16.514+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:16.515024+0000) 2024-04-27T03:41:16.643 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:16.513+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:16.514741+0000) 2024-04-27T03:41:16.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:16.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:16.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:16.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:16.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:16.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:16.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:16.392+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:16.644 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:16.514+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:16.515025+0000) 2024-04-27T03:41:16.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:16.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:16.935 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:16.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:16.935 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:16.830+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:16.935 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:16.831+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:17.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:16.928+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:16.929476+0000) 2024-04-27T03:41:17.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:17.018+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:17.019576+0000) 2024-04-27T03:41:17.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:17.072+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:17.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:17.072+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:17.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:17.072+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:17.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:17.073+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:17.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:17.272+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:17.273781+0000) 2024-04-27T03:41:17.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:17.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:17.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:17.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:17.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:17.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:17.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:17.432+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:18.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:18.020+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:18.021668+0000) 2024-04-27T03:41:18.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:18.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:18.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:18.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:18.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:18.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:18.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:18.043+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:18.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:17.866+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:18.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:17.866+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:18.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:17.866+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:18.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:17.867+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:18.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:17.932+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:17.933664+0000) 2024-04-27T03:41:18.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:18.275+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:18.276617+0000) 2024-04-27T03:41:18.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:18.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:18.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:18.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:18.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:18.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:18.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:18.410+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:19.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:18.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:19.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:18.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:19.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:18.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:19.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:18.887+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:19.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:18.934+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:18.935771+0000) 2024-04-27T03:41:19.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:19.015+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:19.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:19.015+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:19.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:19.015+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:19.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:19.016+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:19.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:19.025+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:19.026418+0000) 2024-04-27T03:41:19.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:19.277+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:19.278787+0000) 2024-04-27T03:41:19.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:19.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:19.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:19.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:19.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:19.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:19.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:19.412+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:20.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:19.896+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:20.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:19.896+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:20.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:19.896+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:20.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:19.897+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:20.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:19.937+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:19.937906+0000) 2024-04-27T03:41:20.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:20.021+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:20.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:20.021+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:20.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:20.021+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:20.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:20.021+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:20.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:20.028+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:20.029288+0000) 2024-04-27T03:41:20.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:20.280+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:20.281012+0000) 2024-04-27T03:41:20.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:20.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:20.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:20.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:20.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:20.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:20.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:20.428+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:20.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:20.864+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:20.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:20.864+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:20.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:20.864+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:20.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:20.865+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:21.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:20.939+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:20.939973+0000) 2024-04-27T03:41:21.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:21.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:21.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:21.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:21.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:21.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:21.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:21.011+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:21.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:21.030+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:21.031419+0000) 2024-04-27T03:41:21.525 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:21.284+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:21.285299+0000) 2024-04-27T03:41:21.525 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:21.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:21.525 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:21.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:21.525 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:21.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:21.525 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:21.470+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:21.871 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:41:21.525+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69380 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:41:21.871 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:21.522+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:21.523043+0000) 2024-04-27T03:41:21.871 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:21.864+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:21.871 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:21.864+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:21.871 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:21.864+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:21.872 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:21.522+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:21.523476+0000) 2024-04-27T03:41:21.872 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:21.522+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:21.522917+0000) 2024-04-27T03:41:22.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:21.865+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:22.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:21.940+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:21.941313+0000) 2024-04-27T03:41:22.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:21.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:22.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:21.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:22.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:21.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:22.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:21.997+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:22.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:22.031+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:22.032620+0000) 2024-04-27T03:41:22.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:22.286+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:22.287614+0000) 2024-04-27T03:41:22.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:22.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:22.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:22.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:22.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:22.432+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:22.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:22.433+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:23.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:22.891+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:23.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:22.891+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:23.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:22.891+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:23.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:22.892+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:23.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:22.945+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:22.946312+0000) 2024-04-27T03:41:23.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:23.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:23.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:23.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:23.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:23.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:23.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:23.011+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:23.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:23.036+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:23.037334+0000) 2024-04-27T03:41:23.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:23.289+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:23.289846+0000) 2024-04-27T03:41:23.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:23.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:23.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:23.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:23.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:23.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:23.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:23.443+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:24.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:23.915+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:24.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:23.915+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:24.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:23.915+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:24.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:23.916+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:24.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:23.950+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:23.951345+0000) 2024-04-27T03:41:24.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:23.963+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:24.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:23.963+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:24.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:23.963+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:24.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:23.964+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:24.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:24.039+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:24.040301+0000) 2024-04-27T03:41:24.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:24.293+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:24.294039+0000) 2024-04-27T03:41:24.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:24.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:24.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:24.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:24.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:24.470+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:24.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:24.471+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:25.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:24.909+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:25.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:24.909+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:25.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:24.909+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:25.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:24.911+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:25.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:24.952+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:24.953489+0000) 2024-04-27T03:41:25.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:24.984+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:25.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:24.984+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:25.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:24.984+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:25.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:24.986+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:25.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:25.041+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:25.042545+0000) 2024-04-27T03:41:25.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:25.295+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:25.296280+0000) 2024-04-27T03:41:25.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:25.440+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:25.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:25.440+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:25.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:25.440+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:25.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:25.441+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:25.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:25.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:25.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:25.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:25.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:25.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:25.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:25.922+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:26.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:25.956+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:25.957698+0000) 2024-04-27T03:41:26.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:26.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:26.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:26.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:26.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:26.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:26.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:26.011+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:26.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:26.044+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:26.045378+0000) 2024-04-27T03:41:26.531 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:26.298+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:26.299057+0000) 2024-04-27T03:41:26.531 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:26.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:26.531 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:26.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:26.531 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:26.465+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:26.531 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:26.466+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:26.888 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:41:26.535+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69418 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:41:26.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:26.530+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:26.531116+0000) 2024-04-27T03:41:26.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:26.530+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:26.531703+0000) 2024-04-27T03:41:26.889 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:26.529+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:26.530688+0000) 2024-04-27T03:41:26.889 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:26.882+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:26.889 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:26.882+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:26.889 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:26.882+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:27.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:26.882+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:27.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:26.959+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:26.960281+0000) 2024-04-27T03:41:27.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:27.047+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:27.047957+0000) 2024-04-27T03:41:27.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:27.049+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:27.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:27.049+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:27.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:27.049+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:27.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:27.050+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:27.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:27.300+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:27.301762+0000) 2024-04-27T03:41:27.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:27.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:27.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:27.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:27.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:27.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:27.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:27.468+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:28.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:27.878+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:28.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:27.878+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:28.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:27.878+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:28.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:27.879+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:28.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:27.961+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:27.963186+0000) 2024-04-27T03:41:28.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:28.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:28.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:28.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:28.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:28.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:28.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:28.014+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:28.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:28.049+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:28.051214+0000) 2024-04-27T03:41:28.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:28.302+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:28.303856+0000) 2024-04-27T03:41:28.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:28.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:28.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:28.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:28.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:28.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:28.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:28.425+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:29.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:28.862+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:29.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:28.862+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:29.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:28.862+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:29.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:28.863+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:29.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:28.964+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:28.965815+0000) 2024-04-27T03:41:29.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:28.972+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:29.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:28.972+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:29.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:28.972+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.327665+0000 (oldest deadline 2024-04-27T03:31:20.327665+0000) 2024-04-27T03:41:29.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:28.973+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:29.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:29.051+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:29.053502+0000) 2024-04-27T03:41:29.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:29.304+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:29.306167+0000) 2024-04-27T03:41:29.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:29.450+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:29.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:29.450+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:29.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:29.450+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:29.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:29.451+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:30.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:29.924+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:30.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:30.054+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:30.056502+0000) 2024-04-27T03:41:30.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:29.819+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:30.144 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:29.819+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:30.144 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:29.819+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:30.144 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:29.821+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:30.144 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:29.966+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:29.968146+0000) 2024-04-27T03:41:30.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:30.307+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:30.309235+0000) 2024-04-27T03:41:30.810 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:30.495+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:30.810 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:30.495+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:30.810 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:30.495+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:30.810 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:30.495+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:31.066 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:30.891+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:31.066 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:30.802+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:31.066 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:30.802+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:31.066 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:30.802+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:31.066 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:30.803+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:31.067 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:30.969+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:30.971179+0000) 2024-04-27T03:41:31.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:31.059+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:31.060843+0000) 2024-04-27T03:41:31.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:31.539+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:31.540888+0000) 2024-04-27T03:41:31.669 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:41:31.543+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69459 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:41:31.669 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:31.543+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:31.544968+0000) 2024-04-27T03:41:31.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:31.311+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:31.313503+0000) 2024-04-27T03:41:31.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:31.482+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:31.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:31.482+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:31.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:31.482+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:31.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:31.483+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:31.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:31.540+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:31.542300+0000) 2024-04-27T03:41:31.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:31.904+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:31.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:31.776+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:31.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:31.776+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:31.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:31.776+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:31.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:31.777+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:32.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:31.971+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:31.972885+0000) 2024-04-27T03:41:32.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:32.060+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:32.062456+0000) 2024-04-27T03:41:32.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:32.313+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:32.314984+0000) 2024-04-27T03:41:32.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:32.473+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:32.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:32.473+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:32.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:32.473+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:32.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:32.474+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:33.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:32.915+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:33.072 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:32.795+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:33.073 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:32.795+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:33.073 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:32.795+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:33.073 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:32.796+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:33.073 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:32.973+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:32.974946+0000) 2024-04-27T03:41:33.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:33.315+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:33.317061+0000) 2024-04-27T03:41:33.432 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:33.065+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:33.067169+0000) 2024-04-27T03:41:33.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:33.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:33.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:33.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:33.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:33.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:00.942704+0000 (oldest deadline 2024-04-27T03:31:20.942704+0000) 2024-04-27T03:41:33.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:33.425+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:34.075 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:33.903+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:34.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:33.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:34.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:33.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:34.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:33.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:34.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:33.802+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:34.075 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:33.975+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:33.977058+0000) 2024-04-27T03:41:34.463 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:34.067+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:34.069731+0000) 2024-04-27T03:41:34.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:34.317+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:34.319269+0000) 2024-04-27T03:41:34.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:34.456+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:35.077 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:34.932+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:35.077 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:34.789+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:35.077 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:34.789+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:35.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:34.789+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:35.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:34.790+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:35.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:34.977+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:34.979305+0000) 2024-04-27T03:41:35.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:35.070+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:35.072381+0000) 2024-04-27T03:41:35.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:35.320+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:35.322325+0000) 2024-04-27T03:41:35.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:35.415+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:36.077 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:35.950+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:36.077 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:35.742+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:36.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:35.742+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:36.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:35.742+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:36.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:35.743+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:36.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:35.979+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:35.981389+0000) 2024-04-27T03:41:36.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:36.072+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:36.074721+0000) 2024-04-27T03:41:36.677 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:41:36.551+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69497 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:41:36.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:36.549+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:36.550868+0000) 2024-04-27T03:41:36.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:36.547+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:36.549740+0000) 2024-04-27T03:41:36.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:36.322+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:36.324621+0000) 2024-04-27T03:41:36.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:36.379+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:36.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:36.547+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:36.549347+0000) 2024-04-27T03:41:36.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:36.775+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:36.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:36.775+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:36.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:36.775+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:36.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:36.775+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:37.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:36.955+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:37.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:37.074+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:37.075913+0000) 2024-04-27T03:41:37.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:36.980+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:36.982640+0000) 2024-04-27T03:41:37.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:37.324+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:37.326258+0000) 2024-04-27T03:41:37.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:37.336+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:38.077 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:37.953+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:38.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:37.810+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:38.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:37.810+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:38.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:37.810+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:31:01.221609+0000 (oldest deadline 2024-04-27T03:31:21.221609+0000) 2024-04-27T03:41:38.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:37.811+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:38.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:37.983+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:37.985306+0000) 2024-04-27T03:41:38.333 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:38.296+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:38.334 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:38.076+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:38.078033+0000) 2024-04-27T03:41:38.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:38.326+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:38.328353+0000) 2024-04-27T03:41:39.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:38.927+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:39.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:38.800+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:39.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:38.986+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:38.987917+0000) 2024-04-27T03:41:39.336 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:39.080+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:39.082384+0000) 2024-04-27T03:41:39.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:39.302+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:39.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:39.329+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:39.330756+0000) 2024-04-27T03:41:40.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:39.773+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:40.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:39.989+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:39.991085+0000) 2024-04-27T03:41:40.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:39.889+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:40.338 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:40.083+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:40.084678+0000) 2024-04-27T03:41:40.339 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:40.259+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:40.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:40.332+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:40.333455+0000) 2024-04-27T03:41:41.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:40.859+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:41.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:40.797+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:41.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:40.992+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:40.993449+0000) 2024-04-27T03:41:41.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:41.088+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:41.089031+0000) 2024-04-27T03:41:41.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:41.299+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:41.685 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:41:41.560+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69538 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:41:41.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:41.557+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:41.558067+0000) 2024-04-27T03:41:41.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:41.557+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:41.558484+0000) 2024-04-27T03:41:41.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:41.337+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:41.337941+0000) 2024-04-27T03:41:41.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:41.556+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:41.557465+0000) 2024-04-27T03:41:41.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:41.756+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:41.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:41.894+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:42.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:41.994+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:41.994840+0000) 2024-04-27T03:41:42.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:42.089+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:42.090431+0000) 2024-04-27T03:41:42.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:42.258+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:42.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:42.338+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:42.339603+0000) 2024-04-27T03:41:42.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:42.715+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:42.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:42.853+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:43.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:43.094+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:43.094820+0000) 2024-04-27T03:41:43.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:42.998+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:42.999031+0000) 2024-04-27T03:41:43.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:43.284+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:43.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:43.340+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:43.341793+0000) 2024-04-27T03:41:43.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:43.903+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:43.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:43.704+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:44.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:44.002+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:44.003301+0000) 2024-04-27T03:41:44.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:44.098+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:44.099192+0000) 2024-04-27T03:41:44.682 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:44.270+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:44.682 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:44.343+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:44.343900+0000) 2024-04-27T03:41:44.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:44.676+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:45.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:45.005+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:45.006134+0000) 2024-04-27T03:41:45.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:44.951+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:45.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:45.100+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:45.101375+0000) 2024-04-27T03:41:45.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:45.304+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:45.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:45.345+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:45.346021+0000) 2024-04-27T03:41:45.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:45.692+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:46.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:46.008+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:46.009191+0000) 2024-04-27T03:41:46.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:45.952+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:46.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:46.102+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:46.103508+0000) 2024-04-27T03:41:46.568 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:46.257+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:46.568 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:46.347+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:46.348099+0000) 2024-04-27T03:41:46.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:41:46.568+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69576 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:41:46.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:46.566+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:46.567395+0000) 2024-04-27T03:41:46.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:46.565+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:46.565900+0000) 2024-04-27T03:41:46.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:46.664+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:46.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:46.565+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:46.566329+0000) 2024-04-27T03:41:46.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:46.912+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:47.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:47.009+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:47.010427+0000) 2024-04-27T03:41:47.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:47.103+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:47.104735+0000) 2024-04-27T03:41:47.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:47.301+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:47.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:47.348+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:47.349352+0000) 2024-04-27T03:41:47.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:47.694+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:47.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:47.868+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:48.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:48.012+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:48.013229+0000) 2024-04-27T03:41:48.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:48.106+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:48.107405+0000) 2024-04-27T03:41:48.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:48.257+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:48.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:48.351+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:48.352260+0000) 2024-04-27T03:41:48.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:48.679+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:48.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:48.891+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:49.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:49.014+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:49.015355+0000) 2024-04-27T03:41:49.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:49.108+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:49.109641+0000) 2024-04-27T03:41:49.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T03:41:49.611+0000 7ff0e4ed9700 0 monclient(hunting): authenticate timed out after 300 2024-04-27T03:41:49.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:49.255+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:49.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:49.353+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:49.354459+0000) 2024-04-27T03:41:49.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:49.642+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:50.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:50.019+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:50.020346+0000) 2024-04-27T03:41:50.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:49.889+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:50.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:50.110+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:50.111780+0000) 2024-04-27T03:41:50.609 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:50.227+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:50.609 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:50.357+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:50.358723+0000) 2024-04-27T03:41:50.885 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:50.603+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:51.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:51.024+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:51.025455+0000) 2024-04-27T03:41:51.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:50.878+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:51.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:50.878+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:51.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:50.878+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:51.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:50.879+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:51.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:51.113+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:51.114531+0000) 2024-04-27T03:41:51.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:51.264+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:51.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:51.360+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:51.360851+0000) 2024-04-27T03:41:51.848 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:41:51.577+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69617 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:41:51.849 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:51.573+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:51.574147+0000) 2024-04-27T03:41:51.849 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:51.574+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:51.574879+0000) 2024-04-27T03:41:51.849 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:51.625+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:51.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:51.574+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:51.575235+0000) 2024-04-27T03:41:51.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:51.842+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:52.121 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:52.027+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:52.028366+0000) 2024-04-27T03:41:52.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:51.842+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:52.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:51.842+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:52.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:51.843+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:52.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:52.115+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:52.116329+0000) 2024-04-27T03:41:52.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:52.301+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:52.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:52.361+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:52.362641+0000) 2024-04-27T03:41:52.870 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:52.606+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:53.124 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:53.030+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:53.031156+0000) 2024-04-27T03:41:53.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:52.864+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:53.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:52.864+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:53.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:52.864+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:53.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:52.865+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:53.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:53.117+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:53.118581+0000) 2024-04-27T03:41:53.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:53.343+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:53.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:53.366+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:53.367579+0000) 2024-04-27T03:41:53.833 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:53.568+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:54.126 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:54.032+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:54.033696+0000) 2024-04-27T03:41:54.126 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:53.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:54.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:53.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:54.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:53.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:54.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:53.828+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:54.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:54.120+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:54.121032+0000) 2024-04-27T03:41:54.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:54.325+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:54.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:54.369+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:54.369817+0000) 2024-04-27T03:41:54.846 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:54.582+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:55.129 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:55.035+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:55.036444+0000) 2024-04-27T03:41:55.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:54.840+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:55.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:54.840+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:55.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:54.840+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:55.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:54.841+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:55.380 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:55.122+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:55.123791+0000) 2024-04-27T03:41:55.380 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:55.325+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:55.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:55.373+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:55.374687+0000) 2024-04-27T03:41:55.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:55.598+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:56.079 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:56.040+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:56.041544+0000) 2024-04-27T03:41:56.080 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:55.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:56.080 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:55.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:56.080 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:55.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:56.080 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:55.822+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:56.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:56.127+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:56.128185+0000) 2024-04-27T03:41:56.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:56.312+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:56.713 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:41:56.585+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69655 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:41:56.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:56.582+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:56.582946+0000) 2024-04-27T03:41:56.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:56.588+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:56.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:56.378+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:56.379556+0000) 2024-04-27T03:41:56.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:56.582+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:56.583126+0000) 2024-04-27T03:41:56.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:56.581+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:56.582656+0000) 2024-04-27T03:41:56.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:56.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:56.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:56.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:56.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:56.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:56.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:56.807+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:57.262 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:57.128+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:57.129417+0000) 2024-04-27T03:41:57.262 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:57.043+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:57.044493+0000) 2024-04-27T03:41:57.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:57.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:41:57.557 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:57.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:41:57.557 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:57.347+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:41:57.557 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:57.347+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:57.557 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:57.381+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:57.382542+0000) 2024-04-27T03:41:57.837 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:57.550+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:58.139 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:58.045+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:58.046601+0000) 2024-04-27T03:41:58.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:57.831+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:58.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:57.831+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:58.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:57.831+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:58.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:57.832+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:58.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:58.132+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:58.133712+0000) 2024-04-27T03:41:58.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:58.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:41:58.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:58.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:41:58.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:58.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:41:58.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:58.343+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:58.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:58.385+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:58.386770+0000) 2024-04-27T03:41:58.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:58.518+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:59.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:59.048+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:59.049750+0000) 2024-04-27T03:41:59.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:58.849+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:59.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:58.849+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:59.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:58.849+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:41:59.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:58.850+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:59.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:59.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:41:59.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:59.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:41:59.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:59.372+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:41:59.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:59.373+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:59.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:59.137+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:59.137865+0000) 2024-04-27T03:41:59.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:41:59.521+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:41:59.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:41:59.389+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:41:59.390795+0000) 2024-04-27T03:42:00.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:00.051+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:00.051896+0000) 2024-04-27T03:42:00.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:59.835+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:00.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:59.835+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:00.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:59.835+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:00.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:41:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:41:59.836+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:00.399 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:00.141+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:00.142199+0000) 2024-04-27T03:42:00.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:00.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:00.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:00.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:00.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:00.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:00.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:00.370+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:00.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:00.513+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:00.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:00.393+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:00.394479+0000) 2024-04-27T03:42:01.080 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:01.053+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:01.054073+0000) 2024-04-27T03:42:01.080 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:00.814+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:01.080 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:00.814+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:01.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:00.814+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:01.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:00.815+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:01.372 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:01.144+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:01.144916+0000) 2024-04-27T03:42:01.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:42:01.593+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69698 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:42:01.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:01.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:01.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:01.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:01.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:01.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:01.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:01.497+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:01.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:01.590+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:01.591734+0000) 2024-04-27T03:42:01.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:01.589+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:01.590287+0000) 2024-04-27T03:42:01.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:01.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:01.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:01.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:01.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:01.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:01.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:01.366+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:01.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:01.396+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:01.396950+0000) 2024-04-27T03:42:01.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:01.590+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:01.591644+0000) 2024-04-27T03:42:01.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:01.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:01.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:01.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:01.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:01.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:01.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:01.857+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:02.262 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:02.145+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:02.146664+0000) 2024-04-27T03:42:02.263 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:02.055+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:02.055922+0000) 2024-04-27T03:42:02.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:02.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:02.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:02.498+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:02.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:02.498+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:02.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:02.499+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:02.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:02.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:02.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:02.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:02.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:02.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:02.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:02.377+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:02.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:02.398+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:02.399404+0000) 2024-04-27T03:42:03.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:03.057+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:03.058224+0000) 2024-04-27T03:42:03.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:02.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:03.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:02.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:03.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:02.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:03.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:02.868+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:03.409 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:03.148+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:03.148923+0000) 2024-04-27T03:42:03.410 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:03.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:03.410 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:03.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:03.410 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:03.390+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:03.410 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:03.391+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:03.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:03.403+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:03.404223+0000) 2024-04-27T03:42:03.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:03.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:03.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:03.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:03.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:03.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:03.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:03.498+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:04.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:04.060+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:04.061291+0000) 2024-04-27T03:42:04.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:03.894+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:04.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:03.894+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:04.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:03.894+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:04.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:03.894+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:04.412 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:04.150+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:04.151101+0000) 2024-04-27T03:42:04.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:04.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:04.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:04.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:04.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:04.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:04.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:04.371+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:04.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:04.406+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:04.407177+0000) 2024-04-27T03:42:04.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:04.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:04.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:04.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:04.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:04.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:04.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:04.452+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:05.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:05.062+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:05.063412+0000) 2024-04-27T03:42:05.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:04.941+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:05.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:04.941+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:05.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:04.941+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:05.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:04.941+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:05.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:05.155+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:05.156203+0000) 2024-04-27T03:42:05.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:05.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:05.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:05.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:05.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:05.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:05.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:05.487+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:05.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:05.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:05.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:05.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:05.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:05.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:05.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:05.387+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:05.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:05.409+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:05.410356+0000) 2024-04-27T03:42:06.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:06.066+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:06.067344+0000) 2024-04-27T03:42:06.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:05.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:06.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:05.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:06.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:05.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:06.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:05.952+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:06.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:06.157+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:06.158326+0000) 2024-04-27T03:42:06.524 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:06.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:06.524 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:06.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:06.524 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:06.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:06.524 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:06.389+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:06.524 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:06.411+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:06.412530+0000) 2024-04-27T03:42:06.953 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:42:06.601+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69736 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:42:06.954 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:06.597+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:06.598345+0000) 2024-04-27T03:42:06.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:06.598+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:06.599276+0000) 2024-04-27T03:42:06.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:06.947+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:06.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:06.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:06.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:06.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:06.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:06.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:06.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:06.519+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:06.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:06.598+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:06.598957+0000) 2024-04-27T03:42:07.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:07.067+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:07.068768+0000) 2024-04-27T03:42:07.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:06.947+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:07.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:06.947+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:07.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:06.948+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:07.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:07.158+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:07.159627+0000) 2024-04-27T03:42:07.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:07.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:07.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:07.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:07.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:07.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:07.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:07.493+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:07.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:07.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:07.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:07.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:07.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:07.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:07.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:07.399+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:07.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:07.414+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:07.415484+0000) 2024-04-27T03:42:08.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:08.071+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:08.072087+0000) 2024-04-27T03:42:08.167 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:07.905+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:08.167 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:07.905+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:08.167 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:07.905+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:08.167 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:07.906+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:08.423 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:08.160+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:08.161711+0000) 2024-04-27T03:42:08.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:08.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:08.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:08.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:08.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:08.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:08.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:08.521+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:08.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:08.417+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:08.418352+0000) 2024-04-27T03:42:08.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:08.448+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:08.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:08.448+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:08.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:08.448+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:08.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:08.448+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:09.169 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:09.074+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:09.074831+0000) 2024-04-27T03:42:09.170 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:08.909+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:09.170 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:08.909+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:09.170 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:08.909+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:09.170 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:08.909+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:09.170 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:09.163+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:09.164388+0000) 2024-04-27T03:42:09.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:09.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:09.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:09.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:09.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:09.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:09.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:09.407+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:09.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:09.420+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:09.421228+0000) 2024-04-27T03:42:09.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:09.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:09.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:09.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:09.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:09.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:09.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:09.518+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:10.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:10.076+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:10.077264+0000) 2024-04-27T03:42:10.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:09.875+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:10.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:09.875+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:10.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:09.875+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:10.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:09.876+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:10.426 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:10.166+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:10.167342+0000) 2024-04-27T03:42:10.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:10.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:10.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:10.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:10.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:10.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:10.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:10.478+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:10.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:10.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:10.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:10.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:10.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:10.420+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:10.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:10.420+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:10.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:10.423+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:10.424231+0000) 2024-04-27T03:42:11.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:11.080+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:11.081410+0000) 2024-04-27T03:42:11.109 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:10.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:11.109 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:10.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:11.109 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:10.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:11.109 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:10.841+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:11.389 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:11.170+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:11.171761+0000) 2024-04-27T03:42:11.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:42:11.609+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69777 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:42:11.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:11.605+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:11.606709+0000) 2024-04-27T03:42:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:11.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:11.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:11.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:11.471+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:11.605+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:11.606515+0000) 2024-04-27T03:42:11.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:11.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:11.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:11.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:11.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:11.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:11.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:11.383+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:11.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:11.425+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:11.426405+0000) 2024-04-27T03:42:11.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:11.606+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:11.607352+0000) 2024-04-27T03:42:12.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:11.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:12.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:11.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:12.082 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:11.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:12.082 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:11.823+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:12.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:12.082+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:12.083239+0000) 2024-04-27T03:42:12.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:12.172+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:12.173039+0000) 2024-04-27T03:42:12.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:12.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:12.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:12.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:12.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:12.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:12.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:12.353+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:12.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:12.427+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:12.427798+0000) 2024-04-27T03:42:12.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:12.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:12.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:12.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:12.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:12.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:12.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:12.478+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:13.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:12.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:13.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:12.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:13.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:12.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:13.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:12.803+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:13.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:13.085+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:13.085823+0000) 2024-04-27T03:42:13.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:13.176+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:13.177347+0000) 2024-04-27T03:42:13.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:13.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:13.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:13.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:13.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:13.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:13.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:13.310+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:13.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:13.429+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:13.429843+0000) 2024-04-27T03:42:13.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:13.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:13.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:13.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:13.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:13.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:13.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:13.525+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:14.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:13.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:14.082 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:13.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:14.082 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:13.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:14.082 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:13.808+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:14.348 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:14.087+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:14.088440+0000) 2024-04-27T03:42:14.348 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:14.179+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:14.180237+0000) 2024-04-27T03:42:14.348 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:14.321+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:14.348 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:14.321+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:14.348 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:14.321+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:14.349 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:14.321+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:14.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:14.431+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:14.432638+0000) 2024-04-27T03:42:14.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:14.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:14.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:14.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:14.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:14.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:14.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:14.546+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:15.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:14.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:15.082 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:14.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:15.082 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:14.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:15.082 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:14.823+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:15.348 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:15.090+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:15.091403+0000) 2024-04-27T03:42:15.348 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:15.183+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:15.184556+0000) 2024-04-27T03:42:15.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:15.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:15.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:15.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:15.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:15.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:15.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:15.548+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:15.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:15.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:15.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:15.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:15.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:15.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:15.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:15.358+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:15.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:15.435+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:15.435834+0000) 2024-04-27T03:42:15.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:15.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:15.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:15.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:15.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:15.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:15.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:15.823+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:16.264 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:16.093+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:16.094335+0000) 2024-04-27T03:42:16.264 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:16.185+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:16.186705+0000) 2024-04-27T03:42:16.560 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:16.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:16.560 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:16.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:16.561 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:16.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:16.561 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:16.336+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:16.561 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:16.437+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:16.438651+0000) 2024-04-27T03:42:16.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:42:16.618+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69815 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:42:16.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:16.614+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:16.615518+0000) 2024-04-27T03:42:16.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:16.615+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:16.616230+0000) 2024-04-27T03:42:16.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:16.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:16.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:16.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:16.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:16.786+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:16.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:16.786+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:16.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:16.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:16.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:16.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:16.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:16.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:16.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:16.555+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:16.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:16.615+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:16.616644+0000) 2024-04-27T03:42:17.264 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:17.096+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:17.096822+0000) 2024-04-27T03:42:17.264 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:17.187+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:17.188585+0000) 2024-04-27T03:42:17.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:17.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:17.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:17.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:17.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:17.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:17.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:17.382+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:17.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:17.439+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:17.440597+0000) 2024-04-27T03:42:17.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:17.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:17.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:17.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:17.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:17.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:17.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:17.602+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:17.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:17.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:17.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:17.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:17.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:17.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:17.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:17.738+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:18.264 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:18.098+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:18.099467+0000) 2024-04-27T03:42:18.264 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:18.192+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:18.193472+0000) 2024-04-27T03:42:18.622 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:18.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:18.622 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:18.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:18.622 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:18.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:18.622 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:18.344+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:18.622 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:18.441+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:18.442694+0000) 2024-04-27T03:42:18.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:18.615+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:18.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:18.615+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:18.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:18.615+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:18.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:18.616+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:18.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:18.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:18.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:18.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:18.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:18.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:18.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:18.696+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:19.264 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:19.101+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:19.102575+0000) 2024-04-27T03:42:19.265 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:19.197+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:19.198387+0000) 2024-04-27T03:42:19.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:19.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:19.664 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:19.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:19.664 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:19.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:19.664 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:19.388+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:19.664 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:19.444+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:19.445428+0000) 2024-04-27T03:42:19.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:19.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:19.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:19.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:19.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:19.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:19.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:19.660+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:19.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:19.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:19.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:19.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:19.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:19.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:19.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:19.659+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:20.264 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:20.104+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:20.105420+0000) 2024-04-27T03:42:20.265 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:20.199+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:20.200541+0000) 2024-04-27T03:42:20.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:20.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:20.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:20.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:20.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:20.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:20.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:20.416+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:20.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:20.448+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:20.449686+0000) 2024-04-27T03:42:20.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:20.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:20.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:20.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:20.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:20.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:20.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:20.699+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:20.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:20.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:20.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:20.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:20.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:20.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:20.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:20.629+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:21.264 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:21.107+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:21.108298+0000) 2024-04-27T03:42:21.265 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:21.204+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:21.204931+0000) 2024-04-27T03:42:21.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:21.427+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:21.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:21.427+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:21.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:21.427+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:21.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:21.428+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:21.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:21.453+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:21.453949+0000) 2024-04-27T03:42:21.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:42:21.626+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69856 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:42:21.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:21.623+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:21.624532+0000) 2024-04-27T03:42:21.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:21.624+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:21.625781+0000) 2024-04-27T03:42:21.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:21.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:21.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:21.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:21.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:21.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:21.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:21.658+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:21.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:21.622+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:21.623223+0000) 2024-04-27T03:42:21.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:21.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:21.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:21.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:21.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:21.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:21.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:21.700+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:22.264 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:22.109+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:22.109822+0000) 2024-04-27T03:42:22.265 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:22.206+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:22.207513+0000) 2024-04-27T03:42:22.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:22.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:22.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:22.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:22.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:22.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:22.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:22.420+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:22.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:22.454+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:22.455333+0000) 2024-04-27T03:42:22.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:22.650+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:22.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:22.650+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:22.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:22.650+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:22.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:22.651+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:22.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:22.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:22.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:22.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:22.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:22.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:22.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:22.723+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:23.264 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:23.113+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:23.114339+0000) 2024-04-27T03:42:23.265 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:23.209+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:23.210712+0000) 2024-04-27T03:42:23.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:23.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:23.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:23.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:23.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:23.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:23.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:23.397+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:23.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:23.456+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:23.457431+0000) 2024-04-27T03:42:23.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:23.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:23.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:23.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:23.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:23.737+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:23.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:23.738+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:23.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:23.671+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:23.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:23.671+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:23.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:23.671+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:23.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:23.673+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:24.265 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:24.212+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:24.212843+0000) 2024-04-27T03:42:24.265 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:24.117+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:24.118652+0000) 2024-04-27T03:42:24.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:24.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:24.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:24.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:24.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:24.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:24.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:24.425+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:24.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:24.461+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:24.462343+0000) 2024-04-27T03:42:24.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:24.703+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:24.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:24.703+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:24.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:24.703+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:24.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:24.704+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:24.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:24.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:24.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:24.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:24.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:24.698+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:24.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:24.699+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:25.265 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:25.120+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:25.120978+0000) 2024-04-27T03:42:25.265 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:25.214+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:25.215501+0000) 2024-04-27T03:42:25.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:25.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:25.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:25.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:25.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:25.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:25.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:25.412+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:25.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:25.463+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:25.464497+0000) 2024-04-27T03:42:25.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:25.689+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:25.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:25.689+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:25.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:25.689+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:25.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:25.690+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:25.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:25.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:25.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:25.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:25.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:25.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:25.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:25.674+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:26.254 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:26.125+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:26.125983+0000) 2024-04-27T03:42:26.255 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:26.216+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:26.217663+0000) 2024-04-27T03:42:26.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:26.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:26.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:26.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:26.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:26.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:26.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:26.422+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:26.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:26.466+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:26.467396+0000) 2024-04-27T03:42:26.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:42:26.635+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69894 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:42:26.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:26.631+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:26.632269+0000) 2024-04-27T03:42:26.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:26.633+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:26.634265+0000) 2024-04-27T03:42:26.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:26.672+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:26.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:26.672+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:26.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:26.672+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:26.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:26.674+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:26.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:26.631+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:26.632530+0000) 2024-04-27T03:42:26.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:26.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:26.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:26.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:26.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:26.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:26.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:26.683+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:27.265 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:27.128+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:27.128948+0000) 2024-04-27T03:42:27.265 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:27.218+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:27.219115+0000) 2024-04-27T03:42:27.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:27.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:27.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:27.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:27.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:27.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:27.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:27.428+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:27.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:27.468+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:27.469283+0000) 2024-04-27T03:42:27.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:27.710+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:27.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:27.710+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:27.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:27.710+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:27.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:27.712+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:27.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:27.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:27.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:27.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:27.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:27.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:27.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:27.694+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:28.265 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:28.133+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:28.133986+0000) 2024-04-27T03:42:28.265 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:28.220+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:28.221733+0000) 2024-04-27T03:42:28.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:28.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:28.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:28.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:28.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:28.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:28.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:28.418+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:28.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:28.473+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:28.474174+0000) 2024-04-27T03:42:28.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:28.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:28.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:28.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:28.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:28.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:28.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:28.648+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:28.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:28.724+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:28.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:28.724+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:28.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:28.724+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:28.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:28.726+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:29.231 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:29.138+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:29.139077+0000) 2024-04-27T03:42:29.481 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:29.429+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:29.482 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:29.429+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:29.482 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:29.429+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:29.482 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:29.429+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:29.483 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:29.225+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:29.226098+0000) 2024-04-27T03:42:29.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:29.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:29.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:29.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:29.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:29.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:29.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:29.723+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:29.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:29.475+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:29.476318+0000) 2024-04-27T03:42:29.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:29.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:29.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:29.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:29.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:29.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:29.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:29.677+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:30.234 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:30.142+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:30.144050+0000) 2024-04-27T03:42:30.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:30.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:30.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:30.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:30.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:30.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:30.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:30.402+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:30.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:30.477+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:30.478940+0000) 2024-04-27T03:42:30.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:30.227+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:30.229199+0000) 2024-04-27T03:42:30.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:30.714+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:30.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:30.714+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:30.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:30.714+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:30.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:30.716+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:30.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:30.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:30.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:30.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:30.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:30.721+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:30.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:30.721+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:31.236 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:31.147+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:31.149168+0000) 2024-04-27T03:42:31.489 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:31.229+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:31.231443+0000) 2024-04-27T03:42:31.489 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:31.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:31.489 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:31.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:31.489 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:31.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:31.489 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:31.369+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:31.758 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:42:31.643+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69935 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:42:31.759 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:31.639+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:31.640815+0000) 2024-04-27T03:42:31.759 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:31.639+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:31.640833+0000) 2024-04-27T03:42:31.759 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:31.481+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:31.483206+0000) 2024-04-27T03:42:31.759 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:31.639+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:31.641343+0000) 2024-04-27T03:42:32.083 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:31.755+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:32.083 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:31.755+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:32.083 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:31.755+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:32.084 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:31.757+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:32.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:31.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:32.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:31.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:32.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:31.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:32.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:31.752+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:32.369 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:32.231+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:32.233528+0000) 2024-04-27T03:42:32.369 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:32.150+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:32.152276+0000) 2024-04-27T03:42:32.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:32.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:32.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:32.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:32.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:32.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:32.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:32.384+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:32.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:32.483+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:32.485795+0000) 2024-04-27T03:42:32.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:32.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:32.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:32.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:32.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:32.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:32.981 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:32.767+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:32.981 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:32.720+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:32.981 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:32.720+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:32.981 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:32.720+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:32.981 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:32.721+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:33.244 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:33.152+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:33.154549+0000) 2024-04-27T03:42:33.495 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:33.236+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:33.238477+0000) 2024-04-27T03:42:33.495 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:33.340+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:33.495 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:33.340+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:33.495 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:33.340+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:33.495 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:33.341+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:33.778 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:33.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:33.778 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:33.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:33.778 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:33.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:33.779 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:33.697+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:33.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:33.488+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:33.489966+0000) 2024-04-27T03:42:34.083 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:33.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:34.083 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:33.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:34.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:33.770+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:34.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:33.771+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:34.369 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:34.239+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:34.241021+0000) 2024-04-27T03:42:34.369 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:34.157+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:34.159557+0000) 2024-04-27T03:42:34.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:34.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:34.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:34.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:34.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:34.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:34.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:34.362+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:34.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:34.492+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:34.493812+0000) 2024-04-27T03:42:34.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:34.649+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:34.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:34.649+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:34.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:34.649+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:34.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:34.649+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:34.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:34.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:34.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:34.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:34.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:34.799+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:34.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:34.799+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:35.250 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:35.161+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:35.163790+0000) 2024-04-27T03:42:35.504 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:35.243+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:35.245354+0000) 2024-04-27T03:42:35.504 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:35.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:35.504 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:35.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:35.504 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:35.395+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:35.504 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:35.396+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:35.814 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:35.497+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:35.498826+0000) 2024-04-27T03:42:35.816 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:35.694+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:35.816 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:35.694+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:35.816 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:35.694+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:35.816 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:35.695+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:36.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:35.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:36.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:35.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:36.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:35.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:36.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:35.806+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:36.365 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:36.164+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:36.166177+0000) 2024-04-27T03:42:36.365 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:36.245+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:36.247632+0000) 2024-04-27T03:42:36.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:36.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:36.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:36.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:36.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:36.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:36.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:36.436+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:36.652 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:36.500+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:36.501934+0000) 2024-04-27T03:42:36.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:42:36.652+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 69973 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:42:36.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:36.648+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:36.650650+0000) 2024-04-27T03:42:36.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:36.649+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:36.651024+0000) 2024-04-27T03:42:36.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:36.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:36.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:36.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:36.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:36.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:36.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:36.713+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:36.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:36.648+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:36.650290+0000) 2024-04-27T03:42:36.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:36.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:36.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:36.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:36.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:36.821+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:36.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:36.821+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:37.255 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:37.167+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:37.169249+0000) 2024-04-27T03:42:37.509 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:37.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:37.509 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:37.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:37.509 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:37.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:37.509 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:37.422+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:37.510 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:37.247+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:37.249593+0000) 2024-04-27T03:42:37.808 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:37.501+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:37.503782+0000) 2024-04-27T03:42:37.808 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:37.669+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:37.808 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:37.669+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:37.808 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:37.669+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:37.808 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:37.671+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:38.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:37.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:38.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:37.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:38.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:37.801+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:38.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:37.801+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:38.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:38.169+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:38.171399+0000) 2024-04-27T03:42:38.381 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:38.252+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:38.254682+0000) 2024-04-27T03:42:38.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:38.454+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:38.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:38.454+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:38.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:38.454+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:38.636 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:38.455+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:38.636 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:38.505+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:38.506889+0000) 2024-04-27T03:42:38.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:38.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:38.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:38.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:38.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:38.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:38.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:38.629+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:38.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:38.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:38.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:38.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:38.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:38.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:38.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:38.767+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:39.265 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:39.172+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:39.174316+0000) 2024-04-27T03:42:39.624 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:39.257+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:39.259757+0000) 2024-04-27T03:42:39.625 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:39.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:39.625 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:39.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:39.625 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:39.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:39.625 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:39.415+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:39.625 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:39.507+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:39.509203+0000) 2024-04-27T03:42:39.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:39.797+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:39.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:39.797+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:39.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:39.797+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:39.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:39.798+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:39.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:39.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:39.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:39.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:39.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:39.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:39.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:39.618+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:40.266 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:40.176+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:40.178554+0000) 2024-04-27T03:42:40.516 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:40.261+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:40.262987+0000) 2024-04-27T03:42:40.517 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:40.403+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:40.517 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:40.403+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:40.517 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:40.403+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:40.517 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:40.403+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:40.788 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:40.509+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:40.511540+0000) 2024-04-27T03:42:40.788 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:40.616+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:40.788 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:40.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:40.789 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:40.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:40.789 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:40.617+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:41.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:40.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:41.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:40.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:41.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:40.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:41.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:40.783+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:41.384 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:41.179+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:41.181355+0000) 2024-04-27T03:42:41.384 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:41.263+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:41.265645+0000) 2024-04-27T03:42:41.660 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:41.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:41.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:41.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:41.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:41.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:41.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:41.569+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:41.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:41.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:41.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:41.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:41.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:41.386+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:41.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:41.387+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:41.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:41.512+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:41.513852+0000) 2024-04-27T03:42:41.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:41.656+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:41.658254+0000) 2024-04-27T03:42:41.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:42:41.661+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70014 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:42:41.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:41.656+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:41.658748+0000) 2024-04-27T03:42:41.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:41.657+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:41.658906+0000) 2024-04-27T03:42:41.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:41.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:41.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:41.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:41.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:41.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:41.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:41.760+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:42.267 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:42.182+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:42.183271+0000) 2024-04-27T03:42:42.522 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:42.265+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:42.267026+0000) 2024-04-27T03:42:42.522 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:42.349+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:42.523 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:42.349+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:42.523 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:42.349+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:42.523 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:42.349+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:42.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:42.516+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:42.516954+0000) 2024-04-27T03:42:42.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:42.573+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:42.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:42.573+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:42.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:42.573+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:42.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:42.574+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:42.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:42.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:42.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:42.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:42.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:42.743+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:42.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:42.744+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:43.267 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:43.184+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:43.185500+0000) 2024-04-27T03:42:43.525 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:43.268+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:43.269227+0000) 2024-04-27T03:42:43.533 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:43.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:43.533 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:43.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:43.533 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:43.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:43.533 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:43.309+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:43.778 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:43.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:43.778 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:43.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:43.778 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:43.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:43.779 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:43.596+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:43.779 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:43.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:43.779 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:43.519+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:43.519831+0000) 2024-04-27T03:42:44.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:43.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:44.085 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:43.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:44.085 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:43.770+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:44.391 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:44.186+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:44.187715+0000) 2024-04-27T03:42:44.391 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:44.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:44.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:44.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:44.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:44.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:44.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:44.286+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:44.392 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:44.272+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:44.273650+0000) 2024-04-27T03:42:44.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:44.522+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:44.522946+0000) 2024-04-27T03:42:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:44.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:44.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:44.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:44.598+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:45.085 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:44.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:45.085 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:44.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:45.085 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:44.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:45.085 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:44.748+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:45.391 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:45.276+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:45.277002+0000) 2024-04-27T03:42:45.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:45.189+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:45.190094+0000) 2024-04-27T03:42:45.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:45.243+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:45.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:45.243+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:45.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:45.243+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:45.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:45.244+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:45.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:45.524+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:45.525689+0000) 2024-04-27T03:42:45.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:45.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:45.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:45.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:45.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:45.613+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:46.085 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:45.730+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:46.085 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:45.730+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:46.085 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:45.730+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:46.085 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:45.731+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:46.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:46.191+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:46.192432+0000) 2024-04-27T03:42:46.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:46.280+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:46.281381+0000) 2024-04-27T03:42:46.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:46.250+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:46.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:46.250+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:46.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:46.250+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:46.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:46.250+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:46.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:46.528+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:46.529016+0000) 2024-04-27T03:42:46.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:42:46.671+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70052 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:42:46.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:46.667+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:46.667948+0000) 2024-04-27T03:42:46.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:46.669+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:46.670140+0000) 2024-04-27T03:42:46.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:46.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:46.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:46.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:46.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:46.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:46.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:46.732+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:46.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:46.652+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:46.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:46.652+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:46.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:46.652+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:46.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:46.653+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:46.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:46.667+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:46.668271+0000) 2024-04-27T03:42:47.238 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:47.193+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:47.193955+0000) 2024-04-27T03:42:47.537 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:47.281+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:47.282757+0000) 2024-04-27T03:42:47.537 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:47.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:47.537 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:47.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:47.537 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:47.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:47.538 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:47.232+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:47.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:47.530+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:47.531660+0000) 2024-04-27T03:42:47.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:47.702+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:47.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:47.702+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:47.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:47.702+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:47.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:47.703+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:47.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:47.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:47.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:47.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:47.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:47.731+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:47.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:47.732+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:48.267 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:48.198+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:48.198822+0000) 2024-04-27T03:42:48.539 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:48.286+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:48.287443+0000) 2024-04-27T03:42:48.540 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:48.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:48.540 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:48.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:48.540 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:48.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:48.540 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:48.261+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:48.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:48.533+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:48.534140+0000) 2024-04-27T03:42:48.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:48.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:48.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:48.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:48.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:48.744+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:48.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:48.745+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:48.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:48.706+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:48.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:48.706+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:48.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:48.706+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:48.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:48.707+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:49.255 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:49.200+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:49.200841+0000) 2024-04-27T03:42:49.543 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:49.291+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:49.292501+0000) 2024-04-27T03:42:49.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:49.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:49.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:49.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:49.543 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:49.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:49.544 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:49.249+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:49.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:49.537+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:49.537889+0000) 2024-04-27T03:42:49.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:49.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:49.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:49.746+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:49.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:49.747+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:49.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:49.747+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:49.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:49.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:49.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:49.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:49.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:49.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:49.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:49.745+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:50.268 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:50.202+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:50.202971+0000) 2024-04-27T03:42:50.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:50.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:50.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:50.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:50.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:50.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:50.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:50.216+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:50.545 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:50.296+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:50.297520+0000) 2024-04-27T03:42:50.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:50.539+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:50.540286+0000) 2024-04-27T03:42:50.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:50.703+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:50.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:50.703+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:50.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:50.703+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:50.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:50.704+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:50.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:50.708+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:50.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:50.708+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:50.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:50.708+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:50.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:50.709+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:51.268 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:51.204+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:51.205665+0000) 2024-04-27T03:42:51.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:51.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:51.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:51.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:51.269 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:51.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:51.269 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:51.175+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:51.518 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:51.301+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:51.302623+0000) 2024-04-27T03:42:51.806 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:42:51.679+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70093 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:42:51.806 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:51.543+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:51.543847+0000) 2024-04-27T03:42:51.806 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:51.675+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:51.676603+0000) 2024-04-27T03:42:51.806 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:51.676+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:51.677157+0000) 2024-04-27T03:42:51.807 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:51.692+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:51.807 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:51.692+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:51.807 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:51.692+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:51.807 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:51.693+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:51.807 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:51.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:51.807 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:51.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:51.807 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:51.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:51.807 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:51.669+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:51.807 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:51.676+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:51.676882+0000) 2024-04-27T03:42:52.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:52.206+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:52.207155+0000) 2024-04-27T03:42:52.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:52.305+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:52.305801+0000) 2024-04-27T03:42:52.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:52.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:52.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:52.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:52.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:52.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:52.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:52.167+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:52.653 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:52.544+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:52.545287+0000) 2024-04-27T03:42:52.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:52.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:52.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:52.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:52.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:52.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:52.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:52.693+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:52.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:52.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:52.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:52.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:52.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:52.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:52.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:52.647+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:53.268 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:53.209+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:53.210221+0000) 2024-04-27T03:42:53.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:53.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:53.269 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:53.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:53.269 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:53.208+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:53.269 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:53.208+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:53.552 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:53.307+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:53.308084+0000) 2024-04-27T03:42:53.895 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:53.546+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:53.547361+0000) 2024-04-27T03:42:53.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:53.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:53.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:53.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:53.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:53.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:53.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:53.693+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:53.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:53.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:53.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:53.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:53.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:53.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:53.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:53.640+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:54.406 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:54.214+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:54.214932+0000) 2024-04-27T03:42:54.406 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:54.309+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:54.310422+0000) 2024-04-27T03:42:54.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:54.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:54.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:54.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:54.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:54.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:54.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:54.168+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:54.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:54.548+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:54.549785+0000) 2024-04-27T03:42:54.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:54.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:54.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:54.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:54.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:54.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:54.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:54.648+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:55.086 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:54.736+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:55.086 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:54.736+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:55.086 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:54.736+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:55.086 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:54.737+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:55.406 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:55.218+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:55.219751+0000) 2024-04-27T03:42:55.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:55.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:55.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:55.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:55.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:55.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:55.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:55.169+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:55.407 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:55.311+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:55.312657+0000) 2024-04-27T03:42:55.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:55.551+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:55.552074+0000) 2024-04-27T03:42:55.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:55.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:55.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:55.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:55.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:55.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:55.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:55.761+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:55.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:55.696+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:55.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:55.696+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:55.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:55.696+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:55.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:55.697+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:56.268 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:56.223+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:56.224497+0000) 2024-04-27T03:42:56.269 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:56.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:56.269 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:56.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:56.269 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:56.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:56.269 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:56.151+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:56.559 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:56.314+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:56.314892+0000) 2024-04-27T03:42:56.812 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:42:56.688+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70131 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:42:56.813 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:56.553+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:56.554406+0000) 2024-04-27T03:42:56.813 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:56.684+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:56.685068+0000) 2024-04-27T03:42:56.813 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:56.685+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:56.686602+0000) 2024-04-27T03:42:56.813 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:56.702+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:56.813 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:56.702+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:56.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:56.702+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:56.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:56.703+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:56.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:56.685+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:56.685838+0000) 2024-04-27T03:42:56.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:56.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:56.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:56.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:56.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:56.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:56.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:56.770+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:57.407 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:57.226+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:57.227512+0000) 2024-04-27T03:42:57.407 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:57.317+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:57.318115+0000) 2024-04-27T03:42:57.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:57.175+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:57.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:57.175+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:57.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:57.175+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:57.408 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:57.176+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:57.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:57.555+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:57.556378+0000) 2024-04-27T03:42:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:57.710+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:57.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:57.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:57.712+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:57.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:57.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:57.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:57.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:57.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:57.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:57.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:57.786+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:58.236 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:58.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:58.236 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:58.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:58.236 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:58.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:58.236 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:58.183+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:58.564 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:58.229+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:58.230286+0000) 2024-04-27T03:42:58.565 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:58.319+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:58.320220+0000) 2024-04-27T03:42:58.825 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:58.558+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:58.559259+0000) 2024-04-27T03:42:58.825 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:58.663+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:58.826 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:58.663+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:58.826 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:58.663+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:58.826 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:58.664+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:59.086 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:58.819+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:59.086 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:58.819+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:59.086 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:58.819+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:42:59.086 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:58.820+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:59.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:59.233+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:59.234439+0000) 2024-04-27T03:42:59.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:59.164+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:59.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:59.164+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:59.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:59.164+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:42:59.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:59.164+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:42:59.412 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:59.321+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:59.322539+0000) 2024-04-27T03:42:59.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:42:59.561+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:42:59.562020+0000) 2024-04-27T03:42:59.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:59.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:59.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:59.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:59.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:59.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:42:59.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:42:59.634+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:00.086 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:59.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:00.086 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:59.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:00.086 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:59.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:00.086 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:42:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:42:59.826+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:00.412 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:00.324+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:00.325358+0000) 2024-04-27T03:43:00.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:00.238+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:00.239362+0000) 2024-04-27T03:43:00.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:00.142+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:00.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:00.142+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:00.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:00.142+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:00.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:00.142+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:00.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:00.564+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:00.565272+0000) 2024-04-27T03:43:00.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:00.631+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:00.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:00.631+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:00.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:00.631+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:00.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:00.632+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:01.154 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:00.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:01.154 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:00.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:01.154 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:00.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:01.154 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:00.850+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:01.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:01.241+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:01.242101+0000) 2024-04-27T03:43:01.412 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:01.326+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:01.327534+0000) 2024-04-27T03:43:01.412 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:01.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:01.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:01.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:01.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:01.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:01.413 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:01.180+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:01.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:01.567+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:01.568153+0000) 2024-04-27T03:43:01.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:01.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:01.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:01.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:01.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:01.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:01.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:01.621+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:01.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:01.693+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:01.694120+0000) 2024-04-27T03:43:01.977 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:43:01.696+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70172 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:43:01.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:01.694+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:01.695018+0000) 2024-04-27T03:43:01.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:01.694+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:01.694843+0000) 2024-04-27T03:43:01.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:01.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:01.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:01.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:01.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:01.867+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:01.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:01.868+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:02.249 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:02.217+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:02.249 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:02.217+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:02.249 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:02.217+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:02.249 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:02.218+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:02.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:02.328+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:02.328972+0000) 2024-04-27T03:43:02.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:02.243+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:02.243841+0000) 2024-04-27T03:43:02.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:02.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:02.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:02.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:02.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:02.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:02.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:02.576+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:02.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:02.569+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:02.569888+0000) 2024-04-27T03:43:03.154 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:02.887+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:03.154 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:02.887+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:03.154 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:02.887+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:03.154 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:02.888+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:03.414 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:03.247+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:03.247825+0000) 2024-04-27T03:43:03.415 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:03.330+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:03.331085+0000) 2024-04-27T03:43:03.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:03.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:03.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:03.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:03.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:03.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:03.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:03.188+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:03.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:03.571+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:03.572684+0000) 2024-04-27T03:43:03.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:03.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:03.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:03.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:03.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:03.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:03.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:03.617+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:04.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:04.139+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:04.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:04.139+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:04.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:04.139+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:04.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:04.140+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:04.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:03.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:04.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:03.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:04.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:03.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:04.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:03.928+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:04.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:04.250+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:04.250984+0000) 2024-04-27T03:43:04.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:04.332+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:04.333382+0000) 2024-04-27T03:43:04.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:04.574+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:04.574964+0000) 2024-04-27T03:43:04.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:04.661+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:04.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:04.661+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:04.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:04.661+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:04.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:04.662+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:05.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:05.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:05.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:05.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:05.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:05.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:05.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:05.134+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:05.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:04.937+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:05.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:04.937+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:05.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:04.937+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:05.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:04.938+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:05.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:05.252+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:05.253658+0000) 2024-04-27T03:43:05.582 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:05.335+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:05.336283+0000) 2024-04-27T03:43:05.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:05.576+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:05.577057+0000) 2024-04-27T03:43:05.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:05.689+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:05.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:05.689+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:05.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:05.689+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:05.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:05.690+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:06.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:05.927+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:06.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:05.927+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:06.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:05.927+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:06.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:05.928+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:06.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:06.103+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:06.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:06.103+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:06.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:06.103+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:06.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:06.104+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:06.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:06.254+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:06.255504+0000) 2024-04-27T03:43:06.585 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:06.338+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:06.339253+0000) 2024-04-27T03:43:06.954 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:43:06.706+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70210 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:43:06.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:06.703+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:06.703932+0000) 2024-04-27T03:43:06.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:06.578+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:06.579427+0000) 2024-04-27T03:43:06.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:06.702+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:06.703061+0000) 2024-04-27T03:43:06.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:06.695+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:06.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:06.695+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:06.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:06.695+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:06.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:06.696+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:06.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:06.702+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:06.703179+0000) 2024-04-27T03:43:07.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:06.948+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:07.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:06.948+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:07.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:06.948+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:07.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:06.948+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:07.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:07.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:07.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:07.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:07.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:07.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:07.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:07.113+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:07.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:07.257+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:07.258198+0000) 2024-04-27T03:43:07.587 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:07.339+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:07.340617+0000) 2024-04-27T03:43:07.908 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:07.580+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:07.581526+0000) 2024-04-27T03:43:07.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:07.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:07.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:07.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:07.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:07.697+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:07.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:07.697+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:08.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:07.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:08.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:07.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:08.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:07.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:08.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:07.952+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:08.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:08.154+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:08.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:08.154+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:08.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:08.154+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:08.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:08.154+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:08.590 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:08.344+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:08.345660+0000) 2024-04-27T03:43:08.590 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:08.259+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:08.260252+0000) 2024-04-27T03:43:08.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:08.583+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:08.584688+0000) 2024-04-27T03:43:08.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:08.735+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:08.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:08.735+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:08.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:08.735+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:08.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:08.736+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:09.268 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:08.978+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:09.268 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:08.978+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:09.268 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:08.978+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:09.268 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:08.978+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:09.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:09.186+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:09.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:09.186+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:09.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:09.186+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:09.268 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:09.186+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:09.592 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:09.261+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:09.262342+0000) 2024-04-27T03:43:09.593 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:09.347+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:09.347951+0000) 2024-04-27T03:43:09.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:09.586+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:09.587254+0000) 2024-04-27T03:43:09.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:09.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:09.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:09.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:09.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:09.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:09.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:09.712+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:10.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:09.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:10.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:09.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:10.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:09.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:10.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:09.953+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:10.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:10.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:10.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:10.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:10.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:10.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:10.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:10.174+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:10.595 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:10.264+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:10.265352+0000) 2024-04-27T03:43:10.595 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:10.349+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:10.350149+0000) 2024-04-27T03:43:10.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:10.588+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:10.589567+0000) 2024-04-27T03:43:10.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:10.742+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:10.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:10.742+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:10.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:10.742+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:10.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:10.743+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:11.270 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:10.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:11.271 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:10.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:11.271 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:10.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:11.271 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:10.980+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:11.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:11.197+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:11.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:11.197+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:11.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:11.197+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:11.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:11.198+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:11.597 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:11.266+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:11.267503+0000) 2024-04-27T03:43:11.597 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:11.352+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:11.353277+0000) 2024-04-27T03:43:11.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:43:11.715+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70251 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:43:11.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:11.712+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:11.713700+0000) 2024-04-27T03:43:11.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:11.591+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:11.591969+0000) 2024-04-27T03:43:11.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:11.711+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:11.712788+0000) 2024-04-27T03:43:11.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:11.710+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:11.711505+0000) 2024-04-27T03:43:11.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:11.776+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:11.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:11.776+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:11.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:11.776+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:11.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:11.777+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:12.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:11.937+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:12.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:11.937+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:12.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:11.937+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:12.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:11.937+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:12.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:12.268+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:12.269010+0000) 2024-04-27T03:43:12.600 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:12.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:12.600 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:12.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:12.600 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:12.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:12.601 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:12.222+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:12.601 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:12.354+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:12.354820+0000) 2024-04-27T03:43:12.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:12.593+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:12.594621+0000) 2024-04-27T03:43:12.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:12.810+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:12.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:12.810+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:12.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:12.810+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:12.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:12.811+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:13.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:12.910+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:13.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:12.910+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:13.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:12.910+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:13.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:12.911+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:13.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:13.270+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:13.271139+0000) 2024-04-27T03:43:13.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:13.359+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:13.359831+0000) 2024-04-27T03:43:13.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:13.186+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:13.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:13.187+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:13.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:13.187+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:13.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:13.187+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:13.853 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:13.596+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:13.597089+0000) 2024-04-27T03:43:14.155 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:13.847+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:14.155 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:13.847+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:14.155 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:13.847+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:14.155 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:13.849+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:14.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:13.941+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:14.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:13.941+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:14.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:13.941+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:14.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:13.941+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:14.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:14.272+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:14.273227+0000) 2024-04-27T03:43:14.432 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:14.361+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:14.362090+0000) 2024-04-27T03:43:14.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:14.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:14.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:14.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:14.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:14.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:14.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:14.183+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:14.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:14.598+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:14.599215+0000) 2024-04-27T03:43:15.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:14.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:15.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:14.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:15.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:14.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:15.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:14.886+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:15.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:14.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:15.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:14.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:15.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:14.898+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:15.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:14.899+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:15.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:15.276+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:15.277468+0000) 2024-04-27T03:43:15.432 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:15.363+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:15.364404+0000) 2024-04-27T03:43:15.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:15.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:15.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:15.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:15.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:15.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:15.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:15.136+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:15.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:15.602+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:15.603467+0000) 2024-04-27T03:43:16.155 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:15.917+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:16.155 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:15.917+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:16.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:15.917+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:16.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:15.918+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:16.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:15.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:16.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:15.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:16.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:15.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:16.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:15.887+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:16.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:16.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:16.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:16.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:16.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:16.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:16.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:16.108+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:16.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:16.278+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:16.279597+0000) 2024-04-27T03:43:16.432 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:16.365+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:16.366512+0000) 2024-04-27T03:43:16.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:16.606+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:16.607768+0000) 2024-04-27T03:43:16.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:43:16.724+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70285 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:43:16.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:16.720+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:16.721655+0000) 2024-04-27T03:43:16.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:16.720+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:16.721644+0000) 2024-04-27T03:43:16.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:16.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:16.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:16.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:16.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:16.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:16.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:16.894+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:16.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:16.720+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:16.721285+0000) 2024-04-27T03:43:16.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:16.866+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:16.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:16.866+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:16.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:16.866+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:16.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:16.866+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:17.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:17.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:17.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:17.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:17.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:17.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:17.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:17.080+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:17.615 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:17.367+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:17.368125+0000) 2024-04-27T03:43:17.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:17.281+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:17.282318+0000) 2024-04-27T03:43:17.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:17.608+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:17.609748+0000) 2024-04-27T03:43:18.155 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:17.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:18.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:17.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:18.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:17.933+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:18.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:17.934+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:18.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:17.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:18.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:17.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:18.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:17.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:18.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:17.890+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:18.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:18.092+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:18.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:18.092+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:18.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:18.092+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:18.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:18.093+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:18.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:18.286+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:18.286840+0000) 2024-04-27T03:43:18.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:18.370+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:18.371191+0000) 2024-04-27T03:43:18.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:18.613+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:18.613949+0000) 2024-04-27T03:43:19.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:18.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:19.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:18.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:19.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:18.925+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:19.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:18.926+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:19.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:19.076+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:19.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:19.076+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:19.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:19.076+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:19.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:19.077+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:19.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:18.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:19.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:18.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:19.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:18.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:19.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:18.845+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:19.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:19.290+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:19.291025+0000) 2024-04-27T03:43:19.438 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:19.372+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:19.373487+0000) 2024-04-27T03:43:19.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:19.615+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:19.616474+0000) 2024-04-27T03:43:20.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:19.961+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:20.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:19.961+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:20.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:19.961+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:20.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:19.962+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:20.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:19.893+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:20.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:19.893+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:20.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:19.893+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:20.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:19.894+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:20.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:20.292+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:20.293171+0000) 2024-04-27T03:43:20.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:20.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:20.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:20.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:20.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:20.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:20.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:20.112+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:20.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:20.617+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:20.618759+0000) 2024-04-27T03:43:20.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:20.375+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:20.376353+0000) 2024-04-27T03:43:21.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:20.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:21.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:20.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:21.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:20.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:21.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:20.924+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:21.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:21.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:21.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:21.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:21.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:21.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:21.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:21.066+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:21.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:20.932+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:21.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:20.932+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:21.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:20.932+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:21.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:20.933+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:21.626 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:21.294+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:21.295383+0000) 2024-04-27T03:43:21.627 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:21.377+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:21.378759+0000) 2024-04-27T03:43:21.903 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:43:21.733+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70326 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:43:21.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:21.729+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:21.730124+0000) 2024-04-27T03:43:21.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:21.730+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:21.731290+0000) 2024-04-27T03:43:21.904 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:21.620+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:21.621056+0000) 2024-04-27T03:43:21.904 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:21.730+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:21.731647+0000) 2024-04-27T03:43:22.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:21.897+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:22.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:21.897+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:22.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:21.897+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:22.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:21.898+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:22.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:21.897+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:22.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:21.897+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:22.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:21.897+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:22.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:21.898+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:22.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:22.112+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:22.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:22.112+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:22.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:22.112+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:22.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:22.113+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:22.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:22.296+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:22.297163+0000) 2024-04-27T03:43:22.438 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:22.380+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:22.381564+0000) 2024-04-27T03:43:22.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:22.622+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:22.622971+0000) 2024-04-27T03:43:23.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:22.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:23.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:22.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:23.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:22.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:23.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:22.922+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:23.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:22.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:23.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:22.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:23.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:22.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:23.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:22.940+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:23.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:23.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:23.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:23.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:23.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:23.155+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:23.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:23.156+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:23.630 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:23.300+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:23.301200+0000) 2024-04-27T03:43:23.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:23.383+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:23.384369+0000) 2024-04-27T03:43:23.883 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:23.624+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:23.625059+0000) 2024-04-27T03:43:24.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:23.877+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:24.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:23.877+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:24.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:23.877+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:24.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:23.878+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:24.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:23.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:24.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:23.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:24.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:23.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:24.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:23.909+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:24.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:24.126+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:24.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:24.126+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:24.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:24.126+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:24.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:24.126+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:24.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:24.302+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:24.303283+0000) 2024-04-27T03:43:24.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:24.385+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:24.386664+0000) 2024-04-27T03:43:24.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:24.628+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:24.629285+0000) 2024-04-27T03:43:25.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:24.866+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:25.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:24.866+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:25.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:24.866+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:25.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:24.867+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:25.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:24.903+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:25.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:24.903+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:25.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:24.903+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:25.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:24.904+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:25.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:25.304+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:25.305649+0000) 2024-04-27T03:43:25.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:25.390+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:25.391017+0000) 2024-04-27T03:43:25.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:25.144+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:25.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:25.144+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:25.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:25.144+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:25.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:25.144+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:25.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:25.633+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:25.633838+0000) 2024-04-27T03:43:26.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:25.846+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:26.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:25.846+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:26.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:25.846+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:26.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:25.847+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:26.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:25.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:26.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:25.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:26.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:25.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:26.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:25.950+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:26.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:26.307+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:26.307885+0000) 2024-04-27T03:43:26.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:26.394+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:26.395372+0000) 2024-04-27T03:43:26.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:26.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:26.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:26.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:26.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:26.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:26.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:26.189+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:26.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:26.635+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:26.636219+0000) 2024-04-27T03:43:27.089 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:43:26.741+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70364 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:43:27.089 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:26.738+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:26.739213+0000) 2024-04-27T03:43:27.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:26.737+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:26.738636+0000) 2024-04-27T03:43:27.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:26.825+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:27.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:26.825+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:27.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:26.825+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:27.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:26.825+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:27.090 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:26.739+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:26.740128+0000) 2024-04-27T03:43:27.090 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:26.912+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:27.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:26.912+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:27.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:26.912+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:27.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:26.913+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:27.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:27.309+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:27.309907+0000) 2024-04-27T03:43:27.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:27.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:27.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:27.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:27.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:27.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:27.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:27.198+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:27.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:27.638+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:27.639013+0000) 2024-04-27T03:43:27.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:27.397+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:27.398297+0000) 2024-04-27T03:43:28.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:27.822+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:28.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:27.822+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:28.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:27.822+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:28.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:27.823+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:28.090 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:27.878+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:28.090 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:27.878+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:28.090 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:27.878+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:28.090 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:27.878+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:28.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:28.313+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:28.314087+0000) 2024-04-27T03:43:28.408 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:28.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:28.408 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:28.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:28.409 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:28.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:28.409 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:28.170+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:28.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:28.402+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:28.402886+0000) 2024-04-27T03:43:28.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:28.640+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:28.641132+0000) 2024-04-27T03:43:29.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:28.822+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:29.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:28.822+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:29.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:28.822+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:29.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:28.823+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:29.090 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:28.909+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:29.090 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:28.909+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:29.090 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:28.909+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:29.090 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:28.910+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:29.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:29.315+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:29.316334+0000) 2024-04-27T03:43:29.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:29.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:29.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:29.191+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:29.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:29.191+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:29.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:29.191+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:29.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:29.404+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:29.405232+0000) 2024-04-27T03:43:29.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:29.642+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:29.643382+0000) 2024-04-27T03:43:30.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:29.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:30.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:29.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:30.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:29.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:30.090 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:29.908+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:30.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:29.796+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:30.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:29.796+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:30.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:29.796+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:30.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:29.797+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:30.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:30.318+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:30.319029+0000) 2024-04-27T03:43:30.418 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:30.211+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:30.418 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:30.211+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:30.418 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:30.211+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:30.418 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:30.212+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:30.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:30.409+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:30.409932+0000) 2024-04-27T03:43:30.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:30.645+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:30.646378+0000) 2024-04-27T03:43:31.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:30.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:31.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:30.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:31.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:30.925+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:31.090 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:30.925+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:31.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:30.778+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:31.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:30.778+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:31.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:30.778+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:31.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:30.779+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:31.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:31.320+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:31.321308+0000) 2024-04-27T03:43:31.418 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:31.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:31.418 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:31.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:31.418 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:31.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:31.418 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:31.203+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:31.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:31.647+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:31.648628+0000) 2024-04-27T03:43:31.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:31.411+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:31.412767+0000) 2024-04-27T03:43:32.089 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:43:31.750+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70405 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:43:32.090 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:31.746+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:31.747461+0000) 2024-04-27T03:43:32.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:31.746+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:31.747762+0000) 2024-04-27T03:43:32.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:31.770+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:32.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:31.770+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:32.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:31.770+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:32.091 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:31.771+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:32.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:31.746+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:31.747333+0000) 2024-04-27T03:43:32.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:31.944+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:32.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:31.944+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:32.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:31.944+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:32.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:31.944+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:32.421 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:32.322+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:32.323040+0000) 2024-04-27T03:43:32.421 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:32.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:32.421 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:32.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:32.421 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:32.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:32.421 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:32.161+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:32.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:32.649+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:32.650082+0000) 2024-04-27T03:43:32.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:32.414+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:32.415571+0000) 2024-04-27T03:43:33.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:32.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:33.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:32.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:33.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:32.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:33.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:32.732+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:33.090 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:32.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:33.090 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:32.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:33.090 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:32.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:33.090 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:32.968+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:33.424 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:33.325+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:33.327748+0000) 2024-04-27T03:43:33.424 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:33.200+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:33.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:33.200+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:33.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:33.200+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:33.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:33.201+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:33.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:33.416+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:33.418539+0000) 2024-04-27T03:43:33.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:33.652+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:33.654521+0000) 2024-04-27T03:43:33.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:33.714+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:33.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:33.714+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:33.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:33.714+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:33.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:33.714+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:33.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:33.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:33.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:33.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:33.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:33.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:33.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:33.956+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:34.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:34.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:34.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:34.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:34.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:34.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:34.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:34.172+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:34.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:34.328+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:34.330389+0000) 2024-04-27T03:43:34.663 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:34.419+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:34.421767+0000) 2024-04-27T03:43:34.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:34.655+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:34.657512+0000) 2024-04-27T03:43:34.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:34.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:34.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:34.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:34.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:34.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:34.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:34.712+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:35.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:35.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:35.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:35.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:35.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:35.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:35.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:35.170+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:35.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:34.953+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:35.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:34.953+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:35.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:34.953+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:35.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:34.954+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:35.555 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:35.330+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:35.332575+0000) 2024-04-27T03:43:35.555 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:35.422+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:35.424524+0000) 2024-04-27T03:43:35.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:35.754+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:35.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:35.754+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:35.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:35.754+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:35.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:35.756+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:35.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:35.658+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:35.659826+0000) 2024-04-27T03:43:36.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:35.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:36.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:35.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:36.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:35.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:36.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:35.968+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:36.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:36.192+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:36.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:36.192+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:36.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:36.192+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:36.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:36.193+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:36.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:36.335+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:36.336887+0000) 2024-04-27T03:43:36.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:36.424+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:36.426712+0000) 2024-04-27T03:43:36.975 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:43:36.758+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70443 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:43:36.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:36.662+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:36.664119+0000) 2024-04-27T03:43:36.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:36.755+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:36.758505+0000) 2024-04-27T03:43:36.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:36.750+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:36.755825+0000) 2024-04-27T03:43:36.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:36.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:36.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:36.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:36.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:36.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:36.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:36.755+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:36.758442+0000) 2024-04-27T03:43:36.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:36.804+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:36.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:36.804+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:36.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:36.804+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:36.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:36.805+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:37.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:37.183+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:37.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:37.183+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:37.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:37.183+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:37.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:37.184+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:37.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:36.969+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:37.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:37.336+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:37.338520+0000) 2024-04-27T03:43:37.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:37.427+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:37.429573+0000) 2024-04-27T03:43:37.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:37.663+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:37.665728+0000) 2024-04-27T03:43:37.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:37.766+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:37.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:37.766+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:37.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:37.766+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:37.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:37.767+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:38.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:38.015+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:38.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:38.015+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:38.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:38.015+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:38.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:38.016+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:38.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:38.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:38.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:38.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:38.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:38.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:38.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:38.169+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:38.674 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:38.339+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:38.340860+0000) 2024-04-27T03:43:38.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:38.430+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:38.432362+0000) 2024-04-27T03:43:38.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:38.666+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:38.668575+0000) 2024-04-27T03:43:38.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:38.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:38.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:38.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:38.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:38.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:38.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:38.769+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:39.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:39.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:39.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:39.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:39.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:39.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:39.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:39.026+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:39.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:39.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:39.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:39.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:39.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:39.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:39.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:39.191+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:39.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:39.342+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:39.344031+0000) 2024-04-27T03:43:39.678 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:39.434+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:39.436769+0000) 2024-04-27T03:43:39.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:39.670+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:39.672374+0000) 2024-04-27T03:43:39.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:39.790+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:39.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:39.790+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:39.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:39.790+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:39.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:39.792+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:40.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:40.056+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:40.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:40.056+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:40.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:40.056+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:40.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:40.056+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:40.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:40.177+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:40.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:40.177+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:40.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:40.177+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:40.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:40.177+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:40.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:40.344+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:40.346201+0000) 2024-04-27T03:43:40.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:40.438+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:40.440045+0000) 2024-04-27T03:43:40.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:40.674+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:40.676497+0000) 2024-04-27T03:43:40.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:40.757+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:40.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:40.758+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:40.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:40.758+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:40.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:40.758+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:41.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:41.168+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:41.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:41.168+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:41.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:41.168+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:41.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:41.169+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:41.282 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:41.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:41.282 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:41.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:41.282 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:41.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:41.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:41.060+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:41.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:41.346+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:41.348676+0000) 2024-04-27T03:43:41.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:41.440+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:41.442517+0000) 2024-04-27T03:43:41.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:43:41.766+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70484 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:43:41.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:41.762+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:41.764708+0000) 2024-04-27T03:43:41.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:41.677+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:41.679351+0000) 2024-04-27T03:43:41.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:41.762+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:41.764604+0000) 2024-04-27T03:43:41.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:41.762+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:41.764456+0000) 2024-04-27T03:43:41.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:41.803+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:41.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:41.803+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:41.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:41.803+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:41.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:41.804+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:42.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:42.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:42.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:42.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:42.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:42.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:42.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:42.082+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:42.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:42.164+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:42.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:42.164+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:42.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:42.164+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:42.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:42.164+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:42.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:42.348+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:42.350117+0000) 2024-04-27T03:43:42.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:42.442+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:42.444575+0000) 2024-04-27T03:43:42.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:42.679+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:42.681527+0000) 2024-04-27T03:43:42.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:42.836+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:42.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:42.836+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:42.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:42.836+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:42.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:42.838+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:43.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:43.074+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:43.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:43.074+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:43.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:43.074+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:43.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:43.075+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:43.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:43.141+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:43.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:43.141+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:43.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:43.141+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:43.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:43.142+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:43.690 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:43.350+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:43.352263+0000) 2024-04-27T03:43:43.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:43.447+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:43.449051+0000) 2024-04-27T03:43:43.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:43.683+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:43.684843+0000) 2024-04-27T03:43:43.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:43.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:43.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:43.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:43.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:43.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:43.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:43.875+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:44.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:44.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:44.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:44.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:44.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:44.162+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:44.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:44.163+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:44.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:44.056+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:44.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:44.056+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:44.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:44.056+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:44.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:44.057+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:44.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:44.354+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:44.356536+0000) 2024-04-27T03:43:44.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:44.452+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:44.453873+0000) 2024-04-27T03:43:44.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:44.687+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:44.689190+0000) 2024-04-27T03:43:44.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:44.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:44.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:44.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:44.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:44.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:44.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:44.894+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:45.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:45.037+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:45.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:45.037+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:45.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:45.037+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:45.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:45.037+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:45.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:45.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:45.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:45.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:45.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:45.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:45.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:45.150+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:45.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:45.357+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:45.358715+0000) 2024-04-27T03:43:45.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:45.455+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:45.456778+0000) 2024-04-27T03:43:45.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:45.693+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:45.693907+0000) 2024-04-27T03:43:45.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:45.861+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:45.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:45.861+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:45.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:45.861+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:45.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:45.863+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:46.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:46.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:46.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:46.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:46.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:46.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:46.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:46.161+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:46.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:46.034+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:46.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:46.035+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:46.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:46.035+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:46.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:46.035+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:46.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:46.361+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:46.361955+0000) 2024-04-27T03:43:46.702 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:46.460+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:46.461159+0000) 2024-04-27T03:43:46.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:46.773+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:46.774278+0000) 2024-04-27T03:43:46.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:43:46.776+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70522 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:43:46.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:46.695+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:46.696666+0000) 2024-04-27T03:43:46.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:46.772+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:46.773275+0000) 2024-04-27T03:43:46.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:46.773+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:46.774086+0000) 2024-04-27T03:43:46.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:46.883+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:46.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:46.883+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:46.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:46.883+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:46.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:46.883+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:47.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:47.025+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:47.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:47.025+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:47.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:47.025+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:47.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:47.026+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:47.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:47.199+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:47.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:47.199+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:47.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:47.199+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:47.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:47.200+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:47.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:47.362+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:47.363550+0000) 2024-04-27T03:43:47.704 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:47.461+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:47.462743+0000) 2024-04-27T03:43:47.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:47.698+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:47.698842+0000) 2024-04-27T03:43:47.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:47.886+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:47.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:47.886+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:47.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:47.886+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:47.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:47.887+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:48.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:48.072+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:48.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:48.072+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:48.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:48.072+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:48.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:48.073+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:48.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:48.199+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:48.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:48.199+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:48.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:48.199+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:48.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:48.200+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:48.709 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:48.466+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:48.467045+0000) 2024-04-27T03:43:48.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:48.367+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:48.368601+0000) 2024-04-27T03:43:48.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:48.703+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:48.703881+0000) 2024-04-27T03:43:48.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:48.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:48.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:48.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:48.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:48.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:48.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:48.895+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:49.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:49.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:49.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:49.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:49.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:49.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:49.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:49.063+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:49.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:49.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:49.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:49.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:49.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:49.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:49.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:49.203+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:49.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:49.372+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:49.373469+0000) 2024-04-27T03:43:49.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:49.469+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:49.470149+0000) 2024-04-27T03:43:49.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:49.708+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:49.708946+0000) 2024-04-27T03:43:49.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:49.905+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:49.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:49.905+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:49.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:49.905+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:49.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:49.907+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:50.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:50.080+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:50.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:50.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:50.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:50.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:50.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:50.082+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:50.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:50.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:50.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:50.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:50.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:50.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:50.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:50.233+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:50.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:50.377+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:50.378586+0000) 2024-04-27T03:43:50.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:50.471+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:50.472388+0000) 2024-04-27T03:43:50.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:50.713+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:50.714049+0000) 2024-04-27T03:43:50.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:50.900+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:50.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:50.900+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:50.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:50.900+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:50.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:50.901+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:51.273 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:51.047+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:51.273 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:51.047+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:51.273 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:51.047+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:51.273 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:51.048+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:51.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:51.382+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:51.383611+0000) 2024-04-27T03:43:51.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:51.474+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:51.475277+0000) 2024-04-27T03:43:51.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:51.266+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:51.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:51.266+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:51.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:51.266+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:51.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:51.266+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:51.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:43:51.785+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70563 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:43:51.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:51.781+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:51.782752+0000) 2024-04-27T03:43:51.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:51.718+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:51.719173+0000) 2024-04-27T03:43:51.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:51.781+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:51.782260+0000) 2024-04-27T03:43:51.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:51.781+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:51.781868+0000) 2024-04-27T03:43:51.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:51.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:51.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:51.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:51.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:51.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:51.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:51.875+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:52.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:52.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:52.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:52.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:52.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:52.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:52.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:52.091+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:52.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:52.384+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:52.385196+0000) 2024-04-27T03:43:52.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:52.475+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:52.476739+0000) 2024-04-27T03:43:52.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:52.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:52.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:52.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:52.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:52.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:52.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:52.298+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:52.990 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:52.721+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:52.722587+0000) 2024-04-27T03:43:52.990 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:52.833+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:52.990 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:52.833+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:52.990 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:52.833+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:52.990 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:52.833+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:53.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:53.090+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:53.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:53.090+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:53.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:53.090+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:53.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:53.091+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:53.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:53.480+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:53.481000+0000) 2024-04-27T03:43:53.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:53.386+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:53.387395+0000) 2024-04-27T03:43:53.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:53.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:53.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:53.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:53.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:53.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:53.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:53.270+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:53.987 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:53.724+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:53.724847+0000) 2024-04-27T03:43:53.987 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:53.815+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:53.988 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:53.815+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:53.988 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:53.815+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:53.988 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:53.816+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:54.271 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:54.138+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:54.271 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:54.138+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:54.272 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:54.138+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:54.272 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:54.138+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:54.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:54.388+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:54.389605+0000) 2024-04-27T03:43:54.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:54.482+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:54.483489+0000) 2024-04-27T03:43:54.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:54.265+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:54.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:54.265+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:54.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:54.265+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:54.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:54.265+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:54.990 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:54.726+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:54.727096+0000) 2024-04-27T03:43:54.990 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:54.865+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:54.991 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:54.865+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:54.991 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:54.865+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:54.991 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:54.866+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:55.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:55.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:55.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:55.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:55.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:55.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:55.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:55.122+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:55.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:55.487+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:55.487824+0000) 2024-04-27T03:43:55.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:55.390+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:55.391686+0000) 2024-04-27T03:43:55.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:55.291+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:55.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:55.291+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:55.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:55.291+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:55.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:55.291+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:55.990 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:55.728+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:55.729256+0000) 2024-04-27T03:43:55.990 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:55.863+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:55.990 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:55.863+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:55.991 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:55.863+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:55.991 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:55.864+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:56.255 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:56.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:56.256 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:56.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:56.256 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:56.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:56.256 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:56.099+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:56.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:56.393+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:56.394400+0000) 2024-04-27T03:43:56.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:56.489+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:56.489931+0000) 2024-04-27T03:43:56.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:56.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:56.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:56.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:56.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:56.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:56.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:56.249+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:56.990 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:56.790+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:56.791160+0000) 2024-04-27T03:43:56.991 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:43:56.794+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70601 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:43:56.991 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:56.790+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:56.791733+0000) 2024-04-27T03:43:56.991 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:56.894+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:56.991 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:56.894+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:56.991 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:56.894+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:56.991 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:56.895+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:56.992 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:56.730+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:56.731372+0000) 2024-04-27T03:43:56.992 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:56.790+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:56.790919+0000) 2024-04-27T03:43:57.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:57.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:57.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:57.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:57.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:57.232+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:57.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:57.233+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:57.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:57.087+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:57.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:57.087+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:57.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:57.087+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:57.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:57.087+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:57.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:57.395+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:57.396495+0000) 2024-04-27T03:43:57.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:57.490+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:57.491550+0000) 2024-04-27T03:43:57.990 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:57.732+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:57.733307+0000) 2024-04-27T03:43:57.991 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:57.940+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:57.991 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:57.940+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:57.991 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:57.940+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:57.991 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:57.941+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:58.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:58.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:58.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:58.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:58.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:58.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:58.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:58.095+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:58.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:58.187+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:58.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:58.187+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:58.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:58.187+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:58.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:58.187+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:58.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:58.398+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:58.398917+0000) 2024-04-27T03:43:58.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:58.492+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:58.493665+0000) 2024-04-27T03:43:59.093 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:58.734+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:58.735429+0000) 2024-04-27T03:43:59.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:58.903+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:59.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:58.903+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:59.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:58.903+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:43:59.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:58.905+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:59.409 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:59.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:59.409 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:59.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:59.409 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:59.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:43:59.409 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:59.214+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:59.409 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:59.086+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:59.409 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:59.086+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:59.409 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:59.086+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:43:59.409 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:59.087+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:43:59.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:59.402+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:59.403155+0000) 2024-04-27T03:43:59.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:43:59.497+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:59.497966+0000) 2024-04-27T03:44:00.093 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:43:59.738+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:43:59.739660+0000) 2024-04-27T03:44:00.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:59.943+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:00.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:59.943+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:00.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:59.943+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:00.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:43:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:43:59.943+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:00.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:00.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:00.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:00.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:00.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:00.206+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:00.411 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:00.206+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:00.411 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:00.099+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:00.411 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:00.099+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:00.412 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:00.099+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:00.412 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:00.100+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:00.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:00.404+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:00.405723+0000) 2024-04-27T03:44:00.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:00.500+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:00.501453+0000) 2024-04-27T03:44:01.093 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:00.741+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:00.742419+0000) 2024-04-27T03:44:01.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:00.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:01.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:00.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:01.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:00.922+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:01.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:00.923+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:01.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:01.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:01.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:01.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:01.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:01.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:01.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:01.123+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:01.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:01.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:01.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:01.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:01.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:01.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:01.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:01.203+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:01.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:01.407+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:01.408257+0000) 2024-04-27T03:44:01.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:01.503+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:01.504366+0000) 2024-04-27T03:44:02.093 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:44:01.803+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70642 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:44:02.093 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:01.799+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:01.799973+0000) 2024-04-27T03:44:02.094 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:01.744+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:01.745283+0000) 2024-04-27T03:44:02.094 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:01.799+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:01.800564+0000) 2024-04-27T03:44:02.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:01.799+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:01.800728+0000) 2024-04-27T03:44:02.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:01.948+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:02.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:01.948+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:02.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:01.948+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:02.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:01.949+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:02.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:02.168+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:02.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:02.168+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:02.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:02.168+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:02.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:02.169+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:02.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:02.210+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:02.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:02.210+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:02.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:02.210+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:02.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:02.210+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:02.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:02.505+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:02.506492+0000) 2024-04-27T03:44:02.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:02.409+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:02.410569+0000) 2024-04-27T03:44:03.093 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:02.746+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:02.747109+0000) 2024-04-27T03:44:03.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:02.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:03.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:02.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:03.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:02.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:03.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:02.973+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:03.418 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:03.141+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:03.418 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:03.141+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:03.419 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:03.141+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:03.419 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:03.143+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:03.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:03.244+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:03.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:03.244+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:03.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:03.244+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:03.419 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:03.245+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:03.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:03.411+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:03.412787+0000) 2024-04-27T03:44:03.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:03.507+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:03.508751+0000) 2024-04-27T03:44:04.093 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:03.748+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:03.749464+0000) 2024-04-27T03:44:04.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:03.962+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:04.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:03.962+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:04.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:03.962+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:04.093 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:03.962+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:04.421 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:04.293+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:04.421 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:04.293+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:04.421 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:04.293+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:04.421 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:04.293+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:04.421 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:04.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:04.421 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:04.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:04.421 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:04.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:04.422 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:04.097+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:04.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:04.511+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:04.511969+0000) 2024-04-27T03:44:04.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:04.414+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:04.415498+0000) 2024-04-27T03:44:05.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:04.750+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:04.751601+0000) 2024-04-27T03:44:05.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:04.950+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:05.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:04.950+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:05.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:04.950+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:05.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:04.951+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:05.424 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:05.080+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:05.425 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:05.080+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:05.425 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:05.080+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:05.425 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:05.081+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:05.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:05.296+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:05.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:05.296+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:05.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:05.296+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:05.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:05.296+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:05.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:05.418+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:05.418931+0000) 2024-04-27T03:44:05.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:05.513+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:05.514788+0000) 2024-04-27T03:44:06.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:05.754+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:05.755732+0000) 2024-04-27T03:44:06.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:05.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:06.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:05.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:06.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:05.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:06.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:05.993+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:06.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:06.063+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:06.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:06.063+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:06.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:06.063+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:06.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:06.063+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:06.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:06.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:06.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:06.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:06.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:06.308+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:06.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:06.308+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:06.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:06.420+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:06.421323+0000) 2024-04-27T03:44:06.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:06.517+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:06.517996+0000) 2024-04-27T03:44:07.047 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:44:06.811+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70680 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:44:07.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:06.757+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:06.758091+0000) 2024-04-27T03:44:07.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:06.808+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:06.809344+0000) 2024-04-27T03:44:07.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:06.808+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:06.809268+0000) 2024-04-27T03:44:07.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:06.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:07.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:06.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:07.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:06.996+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:07.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:06.998+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:07.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:06.807+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:06.808620+0000) 2024-04-27T03:44:07.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:07.041+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:07.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:07.041+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:07.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:07.041+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:07.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:07.042+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:07.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:07.422+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:07.423293+0000) 2024-04-27T03:44:07.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:07.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:07.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:07.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:07.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:07.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:07.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:07.296+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:07.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:07.519+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:07.520032+0000) 2024-04-27T03:44:08.093 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:07.758+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:07.759749+0000) 2024-04-27T03:44:08.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:08.003+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:08.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:08.003+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:08.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:08.003+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:08.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:08.005+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:08.431 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:08.275+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:08.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:08.275+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:08.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:08.275+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:08.432 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:08.275+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:08.432 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:08.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:08.432 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:08.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:08.432 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:08.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:08.432 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:08.093+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:08.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:08.424+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:08.425542+0000) 2024-04-27T03:44:08.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:08.521+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:08.522629+0000) 2024-04-27T03:44:09.093 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:08.761+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:08.762442+0000) 2024-04-27T03:44:09.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:08.962+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:09.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:08.962+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:09.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:08.962+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:09.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:08.963+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:09.348 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:09.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:09.348 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:09.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:09.348 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:09.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:09.349 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:09.269+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:09.349 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:09.089+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:09.349 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:09.089+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:09.349 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:09.089+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:09.349 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:09.090+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:09.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:09.427+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:09.428444+0000) 2024-04-27T03:44:09.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:09.526+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:09.526965+0000) 2024-04-27T03:44:10.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:10.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:10.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:10.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:10.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:10.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:10.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:10.013+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:10.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:09.764+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:09.765434+0000) 2024-04-27T03:44:10.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:10.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:10.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:10.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:10.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:10.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:10.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:10.265+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:10.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:10.080+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:10.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:10.080+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:10.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:10.080+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:10.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:10.081+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:10.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:10.432+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:10.432931+0000) 2024-04-27T03:44:10.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:10.528+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:10.529273+0000) 2024-04-27T03:44:11.067 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:10.767+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:10.768404+0000) 2024-04-27T03:44:11.067 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:11.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:11.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:11.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:11.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:11.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:11.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:11.009+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:11.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:11.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:11.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:11.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:11.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:11.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:11.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:11.286+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:11.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:11.061+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:11.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:11.061+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:11.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:11.061+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:11.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:11.061+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:11.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:11.434+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:11.435307+0000) 2024-04-27T03:44:11.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:11.531+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:11.532352+0000) 2024-04-27T03:44:12.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:44:11.820+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70721 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:44:12.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:11.772+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:11.772837+0000) 2024-04-27T03:44:12.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:11.816+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:11.817038+0000) 2024-04-27T03:44:12.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:11.815+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:11.816447+0000) 2024-04-27T03:44:12.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:11.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:12.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:11.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:12.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:11.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:12.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:12.000+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:11.816+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:11.817622+0000) 2024-04-27T03:44:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:12.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:12.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:12.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:12.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:12.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:12.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:12.063+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:12.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:12.305+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:12.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:12.305+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:12.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:12.305+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:12.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:12.306+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:12.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:12.436+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:12.436973+0000) 2024-04-27T03:44:12.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:12.534+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:12.535577+0000) 2024-04-27T03:44:13.061 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:12.773+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:12.774598+0000) 2024-04-27T03:44:13.061 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:12.990+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:13.061 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:12.990+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:13.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:12.990+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:13.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:12.991+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:13.311 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:13.055+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:13.312 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:13.055+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:13.312 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:13.055+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:13.312 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:13.055+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:13.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:13.439+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:13.440076+0000) 2024-04-27T03:44:13.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:13.536+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:13.537668+0000) 2024-04-27T03:44:13.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:13.305+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:13.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:13.305+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:13.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:13.305+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:13.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:13.306+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:14.043 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:13.776+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:13.777577+0000) 2024-04-27T03:44:14.301 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:14.105+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:14.301 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:14.105+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:14.301 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:14.105+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:14.301 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:14.105+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:14.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:14.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:14.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:14.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:14.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:14.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:14.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:14.037+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:14.302 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:14.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:14.302 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:14.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:14.302 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:14.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:14.302 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:14.277+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:14.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:14.539+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:14.540439+0000) 2024-04-27T03:44:14.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:14.441+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:14.442708+0000) 2024-04-27T03:44:15.082 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:14.778+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:14.779737+0000) 2024-04-27T03:44:15.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:15.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:15.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:15.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:15.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:15.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:15.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:15.077+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:15.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:15.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:15.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:15.261+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:15.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:15.261+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:15.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:15.261+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:15.453 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:15.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:15.453 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:15.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:15.453 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:15.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:15.453 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:15.130+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:15.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:15.444+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:15.444979+0000) 2024-04-27T03:44:15.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:15.543+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:15.543943+0000) 2024-04-27T03:44:16.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:15.781+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:15.782003+0000) 2024-04-27T03:44:16.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:16.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:16.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:16.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:16.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:16.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:16.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:16.047+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:16.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:16.176+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:16.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:16.176+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:16.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:16.176+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:16.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:16.177+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:16.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:16.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:16.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:16.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:16.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:16.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:16.376 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:16.250+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:16.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:16.446+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:16.447563+0000) 2024-04-27T03:44:16.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:16.545+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:16.546623+0000) 2024-04-27T03:44:17.094 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:44:16.829+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70759 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:44:17.095 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:16.825+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:16.826295+0000) 2024-04-27T03:44:17.095 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:16.783+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:16.784375+0000) 2024-04-27T03:44:17.096 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:16.825+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:16.826422+0000) 2024-04-27T03:44:17.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:16.825+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:16.826058+0000) 2024-04-27T03:44:17.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:17.019+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:17.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:17.019+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:17.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:17.019+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:17.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:17.020+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:17.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:17.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:17.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:17.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:17.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:17.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:17.395 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:17.215+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:17.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:17.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:17.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:17.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:17.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:17.203+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:17.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:17.203+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:17.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:17.448+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:17.449151+0000) 2024-04-27T03:44:17.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:17.547+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:17.548355+0000) 2024-04-27T03:44:18.095 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:17.785+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:17.785958+0000) 2024-04-27T03:44:18.095 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:18.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:18.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:18.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:18.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:18.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:18.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:18.014+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:18.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:18.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:18.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:18.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:18.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:18.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:18.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:18.180+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:18.459 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:18.194+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:18.459 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:18.194+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:18.459 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:18.194+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:18.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:18.195+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:18.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:18.549+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:18.550435+0000) 2024-04-27T03:44:18.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:18.452+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:18.453334+0000) 2024-04-27T03:44:19.095 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:18.787+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:18.788223+0000) 2024-04-27T03:44:19.095 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:18.990+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:19.095 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:18.990+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:19.095 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:18.990+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:19.095 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:18.991+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:19.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:19.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:19.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:19.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:19.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:19.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:19.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:19.150+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:19.462 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:19.197+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:19.462 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:19.197+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:19.462 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:19.197+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:19.462 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:19.198+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:19.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:19.551+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:19.552571+0000) 2024-04-27T03:44:19.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:19.455+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:19.456087+0000) 2024-04-27T03:44:20.095 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:19.789+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:19.790568+0000) 2024-04-27T03:44:20.095 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:19.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:20.095 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:19.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:20.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:19.967+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:20.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:19.969+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:20.464 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:20.195+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:20.464 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:20.195+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:20.464 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:20.195+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:20.465 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:20.196+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:20.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:20.186+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:20.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:20.186+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:20.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:20.186+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:20.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:20.186+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:20.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:20.458+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:20.458885+0000) 2024-04-27T03:44:20.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:20.554+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:20.555388+0000) 2024-04-27T03:44:21.095 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:20.794+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:20.794862+0000) 2024-04-27T03:44:21.095 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:20.979+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:21.095 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:20.979+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:21.095 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:20.979+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:21.095 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:20.980+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:21.467 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:21.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:21.467 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:21.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:21.467 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:21.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:21.467 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:21.222+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:21.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:21.210+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:21.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:21.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:21.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:21.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:21.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:21.211+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:21.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:21.460+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:21.461246+0000) 2024-04-27T03:44:21.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:21.557+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:21.558717+0000) 2024-04-27T03:44:22.095 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:44:21.838+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70800 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:44:22.095 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:21.836+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:21.837757+0000) 2024-04-27T03:44:22.096 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:21.798+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:21.799625+0000) 2024-04-27T03:44:22.096 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:21.834+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:21.835306+0000) 2024-04-27T03:44:22.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:21.833+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:21.834272+0000) 2024-04-27T03:44:22.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:22.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:22.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:22.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:22.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:22.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:22.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:22.014+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:22.442 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:22.217+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:22.442 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:22.217+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:22.442 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:22.217+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:22.442 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:22.218+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:22.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:22.200+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:22.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:22.200+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:22.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:22.200+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:22.443 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:22.201+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:22.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:22.462+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:22.463084+0000) 2024-04-27T03:44:22.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:22.561+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:22.562014+0000) 2024-04-27T03:44:23.095 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:22.800+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:22.801431+0000) 2024-04-27T03:44:23.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:23.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:23.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:23.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:23.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:23.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:23.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:23.041+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:23.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:23.251+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:23.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:23.251+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:23.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:23.251+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:23.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:23.252+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:23.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:23.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:23.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:23.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:23.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:23.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:23.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:23.241+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:23.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:23.466+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:23.466886+0000) 2024-04-27T03:44:23.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:23.563+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:23.564255+0000) 2024-04-27T03:44:24.095 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:23.802+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:23.803622+0000) 2024-04-27T03:44:24.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:24.025+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:24.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:24.025+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:24.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:24.025+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:24.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:24.026+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:24.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:24.263+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:24.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:24.263+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:24.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:24.263+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:24.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:24.264+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:24.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:24.290+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:24.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:24.290+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:24.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:24.290+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:24.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:24.291+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:24.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:24.468+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:24.469372+0000) 2024-04-27T03:44:24.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:24.566+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:24.567299+0000) 2024-04-27T03:44:25.095 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:24.805+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:24.805945+0000) 2024-04-27T03:44:25.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:25.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:25.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:25.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:25.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:25.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:25.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:25.004+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:25.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:25.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:25.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:25.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:25.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:25.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:25.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:25.289+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:25.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:25.235+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:25.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:25.235+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:25.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:25.236+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:25.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:25.236+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:25.813 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:25.473+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:25.474480+0000) 2024-04-27T03:44:25.814 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:25.568+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:25.569504+0000) 2024-04-27T03:44:26.096 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:25.807+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:25.808173+0000) 2024-04-27T03:44:26.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:25.987+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:26.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:25.987+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:26.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:25.987+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:26.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:25.988+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:26.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:26.258+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:26.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:26.258+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:26.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:26.258+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:26.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:26.258+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:26.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:26.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:26.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:26.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:26.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:26.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:26.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:26.262+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:26.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:26.478+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:26.479586+0000) 2024-04-27T03:44:26.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:26.570+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:26.571727+0000) 2024-04-27T03:44:27.096 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:44:26.847+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70838 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:44:27.096 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:26.844+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:26.845189+0000) 2024-04-27T03:44:27.096 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:26.809+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:26.810594+0000) 2024-04-27T03:44:27.096 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:26.843+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:26.843844+0000) 2024-04-27T03:44:27.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:26.842+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:26.842876+0000) 2024-04-27T03:44:27.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:27.030+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:27.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:27.030+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:27.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:27.030+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:27.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:27.031+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:27.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:27.296+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:27.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:27.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:27.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:27.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:27.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:27.298+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:27.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:27.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:27.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:27.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:27.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:27.209+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:27.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:27.209+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:27.780 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:27.572+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:27.573280+0000) 2024-04-27T03:44:27.780 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:27.480+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:27.481286+0000) 2024-04-27T03:44:28.096 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:27.811+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:27.812298+0000) 2024-04-27T03:44:28.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:28.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:28.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:28.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:28.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:28.004+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:28.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:28.005+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:28.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:28.322+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:28.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:28.322+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:28.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:28.322+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:28.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:28.324+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:28.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:28.200+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:28.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:28.200+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:28.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:28.200+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:28.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:28.201+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:28.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:28.485+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:28.486170+0000) 2024-04-27T03:44:28.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:28.577+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:28.578356+0000) 2024-04-27T03:44:29.096 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:28.814+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:28.815254+0000) 2024-04-27T03:44:29.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:28.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:29.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:28.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:29.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:28.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:29.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:28.974+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:29.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:29.295+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:29.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:29.295+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:29.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:29.295+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:29.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:29.297+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:29.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:29.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:29.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:29.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:29.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:29.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:29.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:29.151+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:29.825 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:29.487+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:29.488283+0000) 2024-04-27T03:44:29.825 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:29.580+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:29.581590+0000) 2024-04-27T03:44:30.096 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:29.818+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:29.819471+0000) 2024-04-27T03:44:30.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:30.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:30.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:30.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:30.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:30.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:30.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:30.010+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:30.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:30.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:30.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:30.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:30.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:30.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:30.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:30.314+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:30.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:30.109+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:30.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:30.109+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:30.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:30.109+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:30.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:30.109+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:30.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:30.489+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:30.490415+0000) 2024-04-27T03:44:30.830 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:30.583+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:30.584385+0000) 2024-04-27T03:44:31.096 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:30.821+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:30.822021+0000) 2024-04-27T03:44:31.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:31.049+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:31.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:31.049+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:31.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:31.049+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:31.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:31.049+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:31.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:31.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:31.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:31.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:31.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:31.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:31.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:31.314+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:31.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:31.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:31.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:31.091+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:31.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:31.091+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:31.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:31.091+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:31.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:31.494+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:31.494923+0000) 2024-04-27T03:44:31.830 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:31.585+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:31.586675+0000) 2024-04-27T03:44:32.083 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:44:31.856+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70879 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:44:32.083 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:31.853+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:31.854712+0000) 2024-04-27T03:44:32.084 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:31.853+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:31.854176+0000) 2024-04-27T03:44:32.084 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:32.077+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:32.084 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:32.077+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:32.084 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:32.077+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:32.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:31.823+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:31.824328+0000) 2024-04-27T03:44:32.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:31.852+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:31.853703+0000) 2024-04-27T03:44:32.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:32.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:32.085 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:32.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:32.085 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:32.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:32.085 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:32.056+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:32.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:32.078+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:32.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:32.318+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:32.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:32.318+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:32.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:32.318+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:32.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:32.319+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:32.833 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:32.496+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:32.497092+0000) 2024-04-27T03:44:32.833 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:32.587+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:32.588399+0000) 2024-04-27T03:44:33.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:33.038+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:33.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:33.038+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:33.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:33.038+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:33.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:33.039+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:33.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:32.826+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:32.827306+0000) 2024-04-27T03:44:33.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:33.059+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:33.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:33.059+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:33.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:33.059+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:33.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:33.060+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:33.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:33.303+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:33.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:33.303+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:33.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:33.303+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:33.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:33.304+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:33.836 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:33.498+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:33.499357+0000) 2024-04-27T03:44:33.836 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:33.589+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:33.590707+0000) 2024-04-27T03:44:34.090 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:33.829+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:33.830275+0000) 2024-04-27T03:44:34.090 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:34.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:34.090 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:34.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:34.090 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:34.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:34.090 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:34.037+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:34.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:34.083+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:34.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:34.083+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:34.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:34.083+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:34.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:34.085+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:34.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:34.260+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:34.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:34.260+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:34.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:34.260+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:34.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:34.261+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:34.840 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:34.501+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:34.502484+0000) 2024-04-27T03:44:34.840 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:34.592+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:34.593385+0000) 2024-04-27T03:44:35.096 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:34.834+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:34.834935+0000) 2024-04-27T03:44:35.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:35.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:35.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:35.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:35.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:35.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:35.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:35.028+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:35.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:35.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:35.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:35.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:35.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:35.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:35.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:35.093+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:35.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:35.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:35.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:35.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:35.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:35.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:35.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:35.250+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:35.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:35.504+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:35.506547+0000) 2024-04-27T03:44:35.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:35.593+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:35.595583+0000) 2024-04-27T03:44:36.096 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:35.835+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:35.837278+0000) 2024-04-27T03:44:36.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:36.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:36.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:36.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:36.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:36.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:36.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:36.066+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:36.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:36.100+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:36.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:36.100+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:36.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:36.100+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:36.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:36.101+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:36.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:36.263+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:36.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:36.263+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:36.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:36.263+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:36.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:36.264+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:36.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:36.507+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:36.508830+0000) 2024-04-27T03:44:36.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:36.596+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:36.597799+0000) 2024-04-27T03:44:37.097 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:44:36.864+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70917 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:44:37.097 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:36.861+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:36.863165+0000) 2024-04-27T03:44:37.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:36.837+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:36.839530+0000) 2024-04-27T03:44:37.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:36.859+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:36.861564+0000) 2024-04-27T03:44:37.098 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:36.860+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:36.862614+0000) 2024-04-27T03:44:37.098 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:37.067+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:37.098 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:37.067+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:37.098 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:37.067+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:37.098 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:37.068+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:37.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:37.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:37.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:37.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:37.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:37.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:37.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:37.311+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:37.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:37.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:37.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:37.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:37.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:37.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:37.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:37.100+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:37.846 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:37.509+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:37.510894+0000) 2024-04-27T03:44:37.847 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:37.597+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:37.599432+0000) 2024-04-27T03:44:38.097 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:37.839+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:37.841194+0000) 2024-04-27T03:44:38.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:38.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:38.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:38.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:38.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:38.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:38.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:38.075+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:38.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:38.295+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:38.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:38.295+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:38.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:38.295+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:38.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:38.296+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:38.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:38.112+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:38.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:38.112+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:38.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:38.112+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:38.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:38.113+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:38.849 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:38.511+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:38.512949+0000) 2024-04-27T03:44:38.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:38.601+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:38.603792+0000) 2024-04-27T03:44:39.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:38.841+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:38.843469+0000) 2024-04-27T03:44:39.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:39.077+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:39.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:39.077+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:39.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:39.077+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:39.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:39.077+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:39.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:39.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:39.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:39.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:39.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:39.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:39.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:39.315+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:39.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:39.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:39.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:39.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:39.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:39.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:39.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:39.151+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:39.851 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:39.513+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:39.515042+0000) 2024-04-27T03:44:39.851 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:39.604+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:39.605926+0000) 2024-04-27T03:44:40.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:39.843+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:39.845713+0000) 2024-04-27T03:44:40.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:40.057+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:40.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:40.057+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:40.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:40.057+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:40.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:40.059+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:40.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:40.341+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:40.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:40.341+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:40.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:40.341+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:40.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:40.342+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:40.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:40.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:40.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:40.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:40.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:40.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:40.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:40.175+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:40.858 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:40.606+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:40.607994+0000) 2024-04-27T03:44:40.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:40.515+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:40.517731+0000) 2024-04-27T03:44:41.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:40.848+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:40.849843+0000) 2024-04-27T03:44:41.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:41.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:41.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:41.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:41.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:41.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:41.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:41.076+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:41.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:41.211+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:41.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:41.211+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:41.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:41.211+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:41.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:41.211+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:41.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:41.299+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:41.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:41.299+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:41.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:41.299+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:41.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:41.300+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:41.858 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:41.608+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:41.610141+0000) 2024-04-27T03:44:41.859 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:41.520+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:41.521985+0000) 2024-04-27T03:44:42.166 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:44:41.873+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70958 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:44:42.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:41.851+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:41.852860+0000) 2024-04-27T03:44:42.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:41.870+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:41.871880+0000) 2024-04-27T03:44:42.166 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:41.869+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:41.871113+0000) 2024-04-27T03:44:42.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:41.868+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:41.870747+0000) 2024-04-27T03:44:42.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:42.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:42.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:42.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:42.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:42.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:42.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:42.043+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:42.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:42.286+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:42.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:42.286+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:42.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:42.286+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:42.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:42.287+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:42.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:42.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:42.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:42.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:42.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:42.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:42.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:42.206+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:42.860 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:42.521+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:42.523632+0000) 2024-04-27T03:44:42.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:42.609+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:42.611692+0000) 2024-04-27T03:44:43.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:42.852+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:42.854697+0000) 2024-04-27T03:44:43.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:43.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:43.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:43.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:43.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:43.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:43.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:43.086+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:43.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:43.318+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:43.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:43.318+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:43.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:43.318+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:43.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:43.319+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:43.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:43.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:43.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:43.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:43.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:43.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:43.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:43.254+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:43.863 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:43.524+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:43.526381+0000) 2024-04-27T03:44:43.863 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:43.613+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:43.614914+0000) 2024-04-27T03:44:44.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:43.855+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:43.857473+0000) 2024-04-27T03:44:44.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:44.057+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:44.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:44.057+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:44.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:44.057+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:44.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:44.058+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:44.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:44.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:44.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:44.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:44.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:44.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:44.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:44.294+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:44.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:44.359+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:44.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:44.359+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:44.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:44.359+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:44.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:44.360+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:44.865 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:44.527+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:44.529251+0000) 2024-04-27T03:44:44.866 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:44.615+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:44.617718+0000) 2024-04-27T03:44:45.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:44.858+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:44.859824+0000) 2024-04-27T03:44:45.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:45.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:45.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:45.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:45.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:45.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:45.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:45.041+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:45.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:45.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:45.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:45.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:45.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:45.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:45.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:45.326+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:45.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:45.402+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:45.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:45.402+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:45.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:45.402+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:45.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:45.403+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:45.868 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:45.619+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:45.620989+0000) 2024-04-27T03:44:45.869 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:45.530+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:45.532164+0000) 2024-04-27T03:44:46.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:45.861+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:45.863019+0000) 2024-04-27T03:44:46.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:46.034+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:46.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:46.034+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:46.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:46.034+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:46.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:46.035+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:46.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:46.405+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:46.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:46.405+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:46.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:46.405+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:46.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:46.405+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:46.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:46.309+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:46.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:46.309+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:46.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:46.309+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:46.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:46.310+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:46.871 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:46.533+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:46.535272+0000) 2024-04-27T03:44:46.871 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:46.622+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:46.624497+0000) 2024-04-27T03:44:47.166 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:44:46.882+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 70996 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:44:47.167 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:46.879+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:46.880984+0000) 2024-04-27T03:44:47.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:46.864+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:46.865843+0000) 2024-04-27T03:44:47.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:46.878+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:46.880546+0000) 2024-04-27T03:44:47.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:46.877+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:46.879584+0000) 2024-04-27T03:44:47.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:47.027+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:47.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:47.027+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:47.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:47.027+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:47.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:47.029+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:47.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:47.272+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:47.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:47.272+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:47.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:47.272+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:47.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:47.272+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:47.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:47.535+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:47.537448+0000) 2024-04-27T03:44:47.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:47.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:47.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:47.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:47.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:47.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:47.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:47.434+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:47.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:47.624+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:47.626119+0000) 2024-04-27T03:44:47.996 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:47.868+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:47.868976+0000) 2024-04-27T03:44:48.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:48.037+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:48.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:48.037+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:48.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:48.037+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:48.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:48.038+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:48.540 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:48.271+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:48.541 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:48.271+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:48.541 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:48.271+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:48.541 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:48.272+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:48.541 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:48.471+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:48.541 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:48.471+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:48.541 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:48.471+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:48.541 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:48.471+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:48.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:48.540+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:48.541745+0000) 2024-04-27T03:44:48.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:48.627+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:48.628243+0000) 2024-04-27T03:44:49.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:48.871+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:48.872100+0000) 2024-04-27T03:44:49.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:48.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:49.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:48.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:49.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:48.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:49.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:48.989+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:49.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:49.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:49.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:49.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:49.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:49.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:49.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:49.311+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:49.880 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:49.543+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:49.544485+0000) 2024-04-27T03:44:49.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:49.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:49.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:49.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:49.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:49.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:49.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:49.496+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:49.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:49.630+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:49.631223+0000) 2024-04-27T03:44:50.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:49.874+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:49.875132+0000) 2024-04-27T03:44:50.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:49.995+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:50.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:49.995+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:50.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:49.995+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:50.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:49.996+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:50.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:50.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:50.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:50.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:50.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:50.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:50.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:50.293+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:50.883 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:50.547+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:50.547918+0000) 2024-04-27T03:44:50.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:50.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:50.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:50.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:50.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:50.474+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:50.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:50.475+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:50.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:50.632+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:50.633473+0000) 2024-04-27T03:44:51.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:51.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:51.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:51.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:51.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:51.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:51.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:51.015+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:51.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:50.876+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:50.877395+0000) 2024-04-27T03:44:51.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:51.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:51.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:51.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:51.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:51.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:51.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:51.282+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:51.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:51.549+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:51.550613+0000) 2024-04-27T03:44:51.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:51.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:51.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:51.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:51.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:51.502+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:51.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:51.503+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:51.887 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:51.637+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:51.637811+0000) 2024-04-27T03:44:52.167 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:44:51.893+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71037 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:44:52.167 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:51.887+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:51.888126+0000) 2024-04-27T03:44:52.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:51.880+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:51.881586+0000) 2024-04-27T03:44:52.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:51.887+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:51.888105+0000) 2024-04-27T03:44:52.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:51.888+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:51.889131+0000) 2024-04-27T03:44:52.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:51.995+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:52.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:51.995+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:52.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:51.995+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:52.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:51.995+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:52.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:52.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:52.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:52.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:52.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:52.331+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:52.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:52.331+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:52.889 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:52.551+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:52.552756+0000) 2024-04-27T03:44:52.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:52.541+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:52.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:52.541+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:52.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:52.541+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:52.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:52.543+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:52.890 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:52.640+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:52.640973+0000) 2024-04-27T03:44:53.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:52.883+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:52.883906+0000) 2024-04-27T03:44:53.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:52.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:53.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:52.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:53.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:52.954+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:53.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:52.954+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:53.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:53.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:53.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:53.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:53.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:53.289+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:53.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:53.289+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:53.891 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:53.556+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:53.556982+0000) 2024-04-27T03:44:53.892 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:53.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:53.892 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:53.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:53.892 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:53.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:53.892 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:53.527+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:53.892 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:53.642+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:53.643098+0000) 2024-04-27T03:44:54.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:53.885+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:53.886100+0000) 2024-04-27T03:44:54.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:53.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:54.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:53.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:54.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:53.986+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:54.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:53.987+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:54.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:54.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:54.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:54.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:54.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:54.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:54.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:54.257+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:54.894 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:54.559+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:54.560744+0000) 2024-04-27T03:44:54.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:54.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:54.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:54.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:54.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:54.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:54.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:54.555+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:54.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:54.644+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:54.645191+0000) 2024-04-27T03:44:55.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:54.888+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:54.888848+0000) 2024-04-27T03:44:55.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:54.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:55.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:54.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:55.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:54.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:55.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:54.991+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:55.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:55.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:55.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:55.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:55.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:55.276+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:55.428 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:55.276+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:55.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:55.562+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:55.562840+0000) 2024-04-27T03:44:55.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:55.515+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:55.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:55.515+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:55.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:55.515+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:55.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:55.516+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:55.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:55.646+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:55.647499+0000) 2024-04-27T03:44:56.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:55.893+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:55.893851+0000) 2024-04-27T03:44:56.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:55.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:56.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:55.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:56.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:55.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:56.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:55.946+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:56.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:56.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:56.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:56.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:56.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:56.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:56.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:56.253+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:56.902 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:56.564+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:56.564937+0000) 2024-04-27T03:44:56.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:56.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:56.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:56.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:56.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:56.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:56.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:56.560+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:56.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:56.648+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:56.649644+0000) 2024-04-27T03:44:57.167 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:44:56.902+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71075 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:44:57.168 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:56.897+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:56.898375+0000) 2024-04-27T03:44:57.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:56.897+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:56.898256+0000) 2024-04-27T03:44:57.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:56.897+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:56.898409+0000) 2024-04-27T03:44:57.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:56.899+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:56.900679+0000) 2024-04-27T03:44:57.169 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:56.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:57.169 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:56.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:57.169 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:56.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:57.169 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:56.975+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:57.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:57.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:57.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:57.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:57.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:57.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:57.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:57.229+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:57.824 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:57.566+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:57.567064+0000) 2024-04-27T03:44:57.825 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:57.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:57.825 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:57.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:57.825 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:57.574+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:57.825 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:57.575+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:57.825 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:57.650+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:57.651390+0000) 2024-04-27T03:44:58.099 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:57.900+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:57.901467+0000) 2024-04-27T03:44:58.100 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:58.023+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:58.100 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:58.023+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:58.100 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:58.023+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:58.100 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:58.023+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:58.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:58.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:58.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:58.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:58.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:58.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:58.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:58.263+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:58.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:58.568+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:58.569251+0000) 2024-04-27T03:44:58.909 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:58.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:58.909 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:58.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:58.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:58.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:58.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:58.603+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:58.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:58.655+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:58.656342+0000) 2024-04-27T03:44:59.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:58.903+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:58.903799+0000) 2024-04-27T03:44:59.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:58.978+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:59.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:58.978+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:59.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:58.978+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:44:59.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:58.979+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:59.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:59.271+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:59.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:59.271+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:59.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:59.271+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:44:59.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:59.272+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:59.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:59.570+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:59.571350+0000) 2024-04-27T03:44:59.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:59.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:59.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:59.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:59.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:59.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:44:59.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:59.578+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:44:59.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:44:59.660+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:59.661376+0000) 2024-04-27T03:45:00.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:44:59.905+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:44:59.906609+0000) 2024-04-27T03:45:00.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:59.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:00.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:59.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:00.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:59.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:00.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:44:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:44:59.973+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:00.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:00.314+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:00.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:00.314+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:00.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:00.314+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:00.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:00.315+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:00.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:00.575+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:00.576458+0000) 2024-04-27T03:45:00.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:00.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:00.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:00.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:00.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:00.604+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:00.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:00.605+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:00.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:00.665+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:00.666424+0000) 2024-04-27T03:45:01.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:00.908+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:00.909793+0000) 2024-04-27T03:45:01.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:01.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:01.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:01.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:01.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:01.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:01.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:01.017+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:01.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:01.265+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:01.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:01.265+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:01.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:01.265+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:01.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:01.266+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:01.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:01.580+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:01.581584+0000) 2024-04-27T03:45:01.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:01.561+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:01.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:01.561+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:01.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:01.561+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:01.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:01.562+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:01.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:01.670+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:01.671563+0000) 2024-04-27T03:45:02.168 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:01.908+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:01.909155+0000) 2024-04-27T03:45:02.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:45:01.910+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71117 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:45:02.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:01.906+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:01.907568+0000) 2024-04-27T03:45:02.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:01.911+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:01.911953+0000) 2024-04-27T03:45:02.169 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:01.907+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:01.907955+0000) 2024-04-27T03:45:02.169 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:02.037+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:02.169 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:02.037+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:02.169 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:02.037+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:02.169 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:02.038+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:02.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:02.301+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:02.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:02.301+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:02.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:02.301+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:02.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:02.301+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:02.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:02.584+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:02.585283+0000) 2024-04-27T03:45:02.891 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:02.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:02.891 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:02.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:02.891 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:02.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:02.891 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:02.573+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:02.891 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:02.674+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:02.675324+0000) 2024-04-27T03:45:03.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:02.914+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:02.915181+0000) 2024-04-27T03:45:03.170 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:03.058+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:03.170 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:03.058+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:03.170 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:03.058+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:03.170 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:03.058+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:03.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:03.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:03.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:03.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:03.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:03.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:03.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:03.280+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:03.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:03.588+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:03.589508+0000) 2024-04-27T03:45:03.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:03.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:03.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:03.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:03.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:03.607+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:03.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:03.609+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:03.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:03.676+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:03.677417+0000) 2024-04-27T03:45:04.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:03.917+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:03.917855+0000) 2024-04-27T03:45:04.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:04.035+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:04.169 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:04.035+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:04.169 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:04.035+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:04.169 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:04.036+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:04.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:04.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:04.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:04.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:04.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:04.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:04.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:04.255+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:04.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:04.590+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:04.591618+0000) 2024-04-27T03:45:04.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:04.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:04.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:04.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:04.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:04.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:04.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:04.634+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:04.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:04.680+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:04.681752+0000) 2024-04-27T03:45:05.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:04.919+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:04.920633+0000) 2024-04-27T03:45:05.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:05.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:05.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:05.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:05.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:05.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:05.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:05.040+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:05.562 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:05.267+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:05.562 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:05.267+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:05.562 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:05.267+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:05.562 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:05.268+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:05.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:05.595+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:05.595841+0000) 2024-04-27T03:45:05.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:05.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:05.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:05.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:05.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:05.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:05.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:05.670+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:05.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:05.685+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:05.686138+0000) 2024-04-27T03:45:06.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:05.922+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:05.923058+0000) 2024-04-27T03:45:06.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:06.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:06.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:06.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:06.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:06.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:06.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:06.039+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:06.562 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:06.299+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:06.562 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:06.299+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:06.563 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:06.299+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:06.563 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:06.300+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:06.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:06.597+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:06.598581+0000) 2024-04-27T03:45:06.918 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:06.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:06.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:06.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:06.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:06.651+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:06.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:06.651+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:06.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:06.688+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:06.689290+0000) 2024-04-27T03:45:07.226 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:45:06.919+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71155 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:45:07.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:06.916+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:06.916842+0000) 2024-04-27T03:45:07.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:06.916+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:06.917387+0000) 2024-04-27T03:45:07.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:06.924+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:06.924846+0000) 2024-04-27T03:45:07.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:06.915+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:06.915939+0000) 2024-04-27T03:45:07.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:07.044+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:07.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:07.044+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:07.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:07.044+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:07.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:07.045+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:07.562 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:07.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:07.563 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:07.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:07.563 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:07.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:07.563 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:07.351+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:07.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:07.599+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:07.600189+0000) 2024-04-27T03:45:07.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:07.639+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:07.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:07.639+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:07.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:07.639+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:07.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:07.640+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:07.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:07.692+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:07.692901+0000) 2024-04-27T03:45:08.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:07.927+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:07.928341+0000) 2024-04-27T03:45:08.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:08.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:08.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:08.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:08.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:08.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:08.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:08.054+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:08.565 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:08.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:08.565 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:08.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:08.565 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:08.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:08.565 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:08.336+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:08.937 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:08.601+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:08.602267+0000) 2024-04-27T03:45:08.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:08.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:08.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:08.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:08.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:08.653+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:08.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:08.654+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:08.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:08.696+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:08.697272+0000) 2024-04-27T03:45:09.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:08.930+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:08.931393+0000) 2024-04-27T03:45:09.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:09.038+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:09.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:09.038+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:09.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:09.038+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:09.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:09.039+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:09.568 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:09.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:09.568 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:09.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:09.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:09.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:09.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:09.362+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:09.939 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:09.603+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:09.604392+0000) 2024-04-27T03:45:09.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:09.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:09.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:09.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:09.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:09.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:09.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:09.683+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:09.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:09.698+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:09.699401+0000) 2024-04-27T03:45:10.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:09.933+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:09.933917+0000) 2024-04-27T03:45:10.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:10.017+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:10.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:10.017+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:10.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:10.017+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:10.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:10.019+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:10.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:10.385+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:10.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:10.385+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:10.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:10.385+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:10.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:10.386+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:10.942 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:10.605+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:10.606489+0000) 2024-04-27T03:45:10.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:10.701+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:10.702295+0000) 2024-04-27T03:45:10.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:10.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:10.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:10.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:10.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:10.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:10.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:10.725+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:11.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:10.935+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:10.936464+0000) 2024-04-27T03:45:11.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:11.015+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:11.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:11.015+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:11.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:11.015+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:11.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:11.016+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:11.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:11.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:11.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:11.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:11.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:11.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:11.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:11.368+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:11.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:11.607+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:11.608627+0000) 2024-04-27T03:45:11.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:11.704+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:11.704944+0000) 2024-04-27T03:45:11.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:11.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:11.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:11.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:11.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:11.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:11.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:11.718+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:12.226 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:45:11.928+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71196 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:45:12.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:11.924+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:11.925192+0000) 2024-04-27T03:45:12.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:11.925+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:11.926282+0000) 2024-04-27T03:45:12.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:11.937+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:11.938355+0000) 2024-04-27T03:45:12.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:11.924+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:11.925569+0000) 2024-04-27T03:45:12.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:11.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:12.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:11.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:12.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:11.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:12.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:11.977+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:12.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:12.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:12.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:12.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:12.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:12.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:12.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:12.392+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:12.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:12.611+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:12.612127+0000) 2024-04-27T03:45:12.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:12.690+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:12.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:12.690+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:12.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:12.690+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:12.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:12.691+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:12.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:12.707+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:12.708654+0000) 2024-04-27T03:45:13.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:12.941+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:12.942358+0000) 2024-04-27T03:45:13.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:12.975+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:13.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:12.975+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:13.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:12.975+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:13.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:12.976+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:13.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:13.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:13.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:13.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:13.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:13.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:13.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:13.350+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:13.860 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:13.613+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:13.614356+0000) 2024-04-27T03:45:13.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:13.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:13.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:13.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:13.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:13.683+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:13.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:13.684+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:13.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:13.712+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:13.712977+0000) 2024-04-27T03:45:14.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:13.945+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:13.946571+0000) 2024-04-27T03:45:14.170 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:14.018+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:14.170 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:14.018+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:14.170 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:14.018+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:14.170 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:14.018+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:14.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:14.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:14.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:14.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:14.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:14.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:14.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:14.308+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:14.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:14.616+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:14.617224+0000) 2024-04-27T03:45:14.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:14.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:14.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:14.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:14.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:14.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:14.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:14.639+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:14.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:14.714+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:14.715140+0000) 2024-04-27T03:45:15.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:14.947+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:14.948724+0000) 2024-04-27T03:45:15.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:14.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:15.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:14.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:15.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:14.971+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:15.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:14.972+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:15.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:15.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:15.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:15.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:15.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:15.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:15.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:15.283+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:15.957 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:15.619+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:15.619839+0000) 2024-04-27T03:45:15.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:15.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:15.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:15.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:15.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:15.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:15.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:15.618+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:15.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:15.718+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:15.719528+0000) 2024-04-27T03:45:16.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:15.950+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:15.951368+0000) 2024-04-27T03:45:16.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:15.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:16.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:15.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:16.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:15.976+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:16.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:15.978+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:16.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:16.283+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:16.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:16.283+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:16.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:16.283+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:16.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:16.283+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:16.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:16.621+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:16.621980+0000) 2024-04-27T03:45:16.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:16.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:16.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:16.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:16.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:16.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:16.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:16.644+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:16.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:16.723+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:16.723917+0000) 2024-04-27T03:45:17.226 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:45:16.936+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71234 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:45:17.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:16.932+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:16.933419+0000) 2024-04-27T03:45:17.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:16.934+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:16.935349+0000) 2024-04-27T03:45:17.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:16.952+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:16.953676+0000) 2024-04-27T03:45:17.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:16.932+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:16.933712+0000) 2024-04-27T03:45:17.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:16.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:17.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:16.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:17.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:16.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:17.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:16.999+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:17.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:17.247+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:17.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:17.247+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:17.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:17.247+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:17.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:17.247+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:17.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:17.623+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:17.624267+0000) 2024-04-27T03:45:17.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:17.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:17.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:17.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:17.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:17.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:17.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:17.670+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:17.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:17.724+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:17.725590+0000) 2024-04-27T03:45:18.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:17.999+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:18.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:17.999+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:18.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:17.999+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:18.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:18.000+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:18.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:17.955+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:17.955892+0000) 2024-04-27T03:45:18.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:18.204+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:18.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:18.204+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:18.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:18.204+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:18.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:18.204+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:18.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:18.625+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:18.626504+0000) 2024-04-27T03:45:18.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:18.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:18.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:18.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:18.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:18.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:18.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:18.640+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:18.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:18.727+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:18.728585+0000) 2024-04-27T03:45:19.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:19.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:19.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:19.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:19.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:19.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:19.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:19.025+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:19.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:18.957+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:18.957847+0000) 2024-04-27T03:45:19.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:19.242+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:19.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:19.242+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:19.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:19.242+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:19.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:19.243+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:19.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:19.629+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:19.630739+0000) 2024-04-27T03:45:19.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:19.641+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:19.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:19.641+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:19.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:19.641+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:19.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:19.642+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:19.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:19.731+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:19.732778+0000) 2024-04-27T03:45:20.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:19.959+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:19.960440+0000) 2024-04-27T03:45:20.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:20.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:20.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:20.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:20.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:20.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:20.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:20.021+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:20.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:20.266+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:20.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:20.266+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:20.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:20.266+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:20.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:20.266+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:20.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:20.632+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:20.633491+0000) 2024-04-27T03:45:20.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:20.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:20.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:20.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:20.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:20.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:20.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:20.603+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:20.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:20.734+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:20.735437+0000) 2024-04-27T03:45:21.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:20.961+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:20.962401+0000) 2024-04-27T03:45:21.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:20.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:21.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:20.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:21.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:20.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:21.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:20.982+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:21.561 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:21.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:21.561 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:21.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:21.561 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:21.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:21.561 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:21.222+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:21.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:21.637+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:21.638622+0000) 2024-04-27T03:45:21.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:21.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:21.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:21.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:21.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:21.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:21.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:21.555+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:21.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:21.739+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:21.739817+0000) 2024-04-27T03:45:22.226 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:45:21.945+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71275 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:45:22.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:21.942+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:21.943192+0000) 2024-04-27T03:45:22.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:21.942+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:21.942926+0000) 2024-04-27T03:45:22.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:21.964+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:21.965353+0000) 2024-04-27T03:45:22.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:21.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:22.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:21.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:22.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:21.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:22.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:21.942+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:22.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:21.944+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:21.944845+0000) 2024-04-27T03:45:22.525 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:22.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:22.526 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:22.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:22.526 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:22.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:22.526 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:22.254+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:22.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:22.639+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:22.640347+0000) 2024-04-27T03:45:22.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:22.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:22.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:22.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:22.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:22.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:22.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:22.520+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:22.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:22.741+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:22.742132+0000) 2024-04-27T03:45:23.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:22.966+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:22.967404+0000) 2024-04-27T03:45:23.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:22.895+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:23.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:22.895+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:23.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:22.895+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:23.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:22.897+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:23.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:23.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:23.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:23.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:23.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:23.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:23.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:23.215+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:23.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:23.641+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:23.642434+0000) 2024-04-27T03:45:23.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:23.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:23.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:23.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:23.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:23.519+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:23.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:23.519+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:23.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:23.743+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:23.744266+0000) 2024-04-27T03:45:24.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:23.968+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:23.969523+0000) 2024-04-27T03:45:24.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:23.929+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:24.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:23.929+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:24.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:23.929+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:24.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:23.930+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:24.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:24.177+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:24.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:24.177+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:24.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:24.177+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:24.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:24.177+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:24.586 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:24.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:24.587 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:24.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:24.587 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:24.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:24.587 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:24.500+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:24.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:24.645+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:24.646673+0000) 2024-04-27T03:45:24.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:24.897+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:24.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:24.745+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:24.746397+0000) 2024-04-27T03:45:25.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:24.970+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:24.971788+0000) 2024-04-27T03:45:25.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:24.897+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:25.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:24.897+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:25.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:24.899+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:25.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:25.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:25.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:25.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:25.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:25.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:25.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:25.181+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:25.754 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:25.648+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:25.649407+0000) 2024-04-27T03:45:25.754 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:25.496+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:25.754 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:25.496+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:25.754 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:25.496+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:25.755 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:25.497+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:26.102 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:25.747+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:25.748630+0000) 2024-04-27T03:45:26.102 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:25.975+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:25.976616+0000) 2024-04-27T03:45:26.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:25.932+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:26.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:25.932+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:26.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:25.932+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:26.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:25.933+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:26.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:26.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:26.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:26.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:26.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:26.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:26.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:26.202+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:26.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:26.481+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:26.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:26.481+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:26.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:26.481+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:26.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:26.482+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:26.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:26.752+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:26.753013+0000) 2024-04-27T03:45:26.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:26.651+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:26.652290+0000) 2024-04-27T03:45:26.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:26.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:26.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:26.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:26.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:26.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:26.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:26.900+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:27.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:26.949+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:26.950474+0000) 2024-04-27T03:45:27.227 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:45:26.955+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71313 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:45:27.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:26.950+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:26.951713+0000) 2024-04-27T03:45:27.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:26.951+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:26.952372+0000) 2024-04-27T03:45:27.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:26.978+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:26.978931+0000) 2024-04-27T03:45:27.524 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:27.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:27.525 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:27.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:27.525 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:27.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:27.525 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:27.225+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:27.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:27.654+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:27.655558+0000) 2024-04-27T03:45:27.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:27.906+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:27.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:27.518+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:27.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:27.518+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:27.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:27.518+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:27.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:27.519+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:27.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:27.753+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:27.754646+0000) 2024-04-27T03:45:28.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:27.980+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:27.981618+0000) 2024-04-27T03:45:28.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:27.906+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:28.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:27.906+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:28.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:27.907+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:28.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:28.274+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:28.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:28.274+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:28.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:28.274+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:28.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:28.274+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:28.763 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:28.658+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:28.659733+0000) 2024-04-27T03:45:28.763 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:28.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:28.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:28.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:28.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:28.483+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:28.764 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:28.484+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:29.102 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:28.756+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:28.757425+0000) 2024-04-27T03:45:29.103 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:28.982+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:28.983778+0000) 2024-04-27T03:45:29.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:28.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:29.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:28.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:29.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:28.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:29.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:28.922+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:29.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:29.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:29.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:29.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:29.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:29.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:29.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:29.293+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:29.765 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:29.663+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:29.663945+0000) 2024-04-27T03:45:29.765 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:29.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:29.766 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:29.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:29.766 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:29.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:29.766 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:29.474+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:30.102 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:29.759+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:29.759813+0000) 2024-04-27T03:45:30.103 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:29.985+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:29.986067+0000) 2024-04-27T03:45:30.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:29.924+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:30.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:29.924+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:30.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:29.924+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:30.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:29.925+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:30.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:30.314+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:30.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:30.314+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:30.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:30.314+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:30.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:30.314+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:30.768 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:30.665+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:30.666122+0000) 2024-04-27T03:45:30.768 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:30.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:30.768 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:30.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:30.769 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:30.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:30.769 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:30.495+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:31.102 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:30.761+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:30.762235+0000) 2024-04-27T03:45:31.103 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:30.987+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:30.988345+0000) 2024-04-27T03:45:31.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:30.957+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:31.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:30.957+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:31.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:30.957+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:31.103 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:30.957+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:31.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:31.299+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:31.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:31.299+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:31.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:31.299+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:31.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:31.300+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:31.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:31.668+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:31.669143+0000) 2024-04-27T03:45:31.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:31.453+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:31.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:31.453+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:31.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:31.453+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:31.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:31.455+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:32.088 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:45:31.964+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71354 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:45:32.088 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:31.764+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:31.764960+0000) 2024-04-27T03:45:32.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:31.960+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:31.960849+0000) 2024-04-27T03:45:32.089 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:31.959+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:31.960626+0000) 2024-04-27T03:45:32.089 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:31.989+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:31.990730+0000) 2024-04-27T03:45:32.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:31.961+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:31.961957+0000) 2024-04-27T03:45:32.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:32.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:32.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:32.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:32.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:32.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:32.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:32.003+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:32.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:32.313+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:32.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:32.313+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:32.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:32.313+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:32.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:32.313+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:32.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:32.671+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:32.672424+0000) 2024-04-27T03:45:32.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:32.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:32.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:32.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:32.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:32.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:32.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:32.444+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:33.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:33.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:33.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:33.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:33.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:33.009+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:33.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:33.010+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:33.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:32.992+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:32.992920+0000) 2024-04-27T03:45:33.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:32.766+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:32.767186+0000) 2024-04-27T03:45:33.403 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:33.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:33.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:33.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:33.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:33.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:33.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:33.293+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:33.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:33.396+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:33.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:33.396+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:33.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:33.396+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:33.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:33.398+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:33.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:33.771+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:33.771827+0000) 2024-04-27T03:45:33.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:33.673+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:33.674643+0000) 2024-04-27T03:45:34.264 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:33.994+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:33.995162+0000) 2024-04-27T03:45:34.265 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:34.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:34.265 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:34.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:34.265 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:34.014+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:34.265 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:34.016+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:34.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:34.258+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:34.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:34.258+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:34.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:34.258+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:34.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:34.258+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:34.604 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:34.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:34.604 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:34.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:34.604 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:34.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:34.604 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:34.385+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:34.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:34.677+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:34.677928+0000) 2024-04-27T03:45:34.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:34.776+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:34.776922+0000) 2024-04-27T03:45:35.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:34.997+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:34.997828+0000) 2024-04-27T03:45:35.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:34.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:35.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:34.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:35.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:34.968+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:35.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:34.969+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:35.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:35.300+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:35.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:35.300+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:35.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:35.300+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:35.603 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:35.301+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:35.604 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:35.378+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:35.604 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:35.378+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:35.604 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:35.378+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:35.604 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:35.379+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:35.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:35.781+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:35.782054+0000) 2024-04-27T03:45:35.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:35.681+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:35.681967+0000) 2024-04-27T03:45:36.291 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:35.999+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:36.000628+0000) 2024-04-27T03:45:36.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:36.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:36.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:36.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:36.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:36.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:36.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:36.012+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:36.604 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:36.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:36.604 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:36.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:36.604 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:36.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:36.604 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:36.423+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:36.605 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:36.313+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:36.605 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:36.313+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:36.605 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:36.313+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:36.605 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:36.314+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:36.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:36.685+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:36.686565+0000) 2024-04-27T03:45:36.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:36.786+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:36.787065+0000) 2024-04-27T03:45:37.226 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:45:36.973+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71392 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:45:37.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:36.970+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:36.970853+0000) 2024-04-27T03:45:37.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:36.969+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:36.970360+0000) 2024-04-27T03:45:37.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:37.002+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:37.002953+0000) 2024-04-27T03:45:37.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:36.969+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:36.970149+0000) 2024-04-27T03:45:37.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:37.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:37.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:37.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:37.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:37.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:37.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:37.012+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:37.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:37.268+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:37.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:37.268+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:37.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:37.268+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:37.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:37.269+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:37.605 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:37.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:37.605 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:37.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:37.605 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:37.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:37.605 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:37.466+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:37.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:37.689+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:37.690550+0000) 2024-04-27T03:45:37.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:37.789+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:37.790006+0000) 2024-04-27T03:45:38.291 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:38.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:38.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:38.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:38.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:38.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:38.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:38.024+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:38.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:38.005+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:38.007217+0000) 2024-04-27T03:45:38.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:38.233+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:38.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:38.233+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:38.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:38.233+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:38.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:38.233+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:38.606 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:38.487+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:38.606 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:38.487+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:38.607 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:38.487+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:38.607 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:38.488+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:38.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:38.790+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:38.792099+0000) 2024-04-27T03:45:38.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:38.691+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:38.692799+0000) 2024-04-27T03:45:39.291 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:39.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:39.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:39.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:39.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:39.013+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:39.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:39.013+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:39.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:39.007+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:39.009430+0000) 2024-04-27T03:45:39.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:39.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:39.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:39.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:39.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:39.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:39.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:39.188+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:39.609 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:39.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:39.609 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:39.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:39.609 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:39.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:39.609 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:39.498+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:39.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:39.694+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:39.696012+0000) 2024-04-27T03:45:39.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:39.792+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:39.794211+0000) 2024-04-27T03:45:40.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:40.010+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:40.012474+0000) 2024-04-27T03:45:40.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:40.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:40.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:40.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:40.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:40.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:40.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:40.236+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:40.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:40.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:40.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:40.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:40.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:40.039+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:40.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:40.041+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:40.609 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:40.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:40.609 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:40.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:40.609 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:40.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:40.610 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:40.486+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:40.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:40.696+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:40.698276+0000) 2024-04-27T03:45:40.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:40.794+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:40.796307+0000) 2024-04-27T03:45:41.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:41.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:41.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:41.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:41.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:41.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:41.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:41.083+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:41.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:41.012+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:41.014641+0000) 2024-04-27T03:45:41.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:41.187+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:41.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:41.187+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:41.293 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:41.187+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:41.293 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:41.187+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:41.609 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:41.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:41.609 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:41.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:41.609 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:41.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:41.610 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:41.529+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:41.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:41.799+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:41.801047+0000) 2024-04-27T03:45:41.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:41.701+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:41.702992+0000) 2024-04-27T03:45:42.292 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:45:41.981+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71433 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:45:42.292 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:41.977+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:41.979134+0000) 2024-04-27T03:45:42.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:41.976+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:41.978520+0000) 2024-04-27T03:45:42.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:42.016+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:42.018365+0000) 2024-04-27T03:45:42.293 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:42.157+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:42.293 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:42.157+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:42.293 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:42.157+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:42.293 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:42.157+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:42.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:41.978+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:41.980020+0000) 2024-04-27T03:45:42.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:42.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:42.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:42.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:42.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:42.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:42.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:42.056+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:42.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:42.702+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:42.704690+0000) 2024-04-27T03:45:42.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:42.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:42.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:42.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:42.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:42.562+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:42.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:42.563+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:42.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:42.800+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:42.802701+0000) 2024-04-27T03:45:43.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:43.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:43.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:43.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:43.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:43.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:43.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:43.092+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:43.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:43.018+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:43.020558+0000) 2024-04-27T03:45:43.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:43.128+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:43.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:43.128+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:43.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:43.128+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:43.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:43.128+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:43.812 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:43.705+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:43.706921+0000) 2024-04-27T03:45:43.813 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:43.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:43.813 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:43.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:43.813 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:43.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:43.813 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:43.523+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:44.104 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:44.021+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:44.023756+0000) 2024-04-27T03:45:44.104 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:43.805+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:43.806998+0000) 2024-04-27T03:45:44.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:44.126+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:44.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:44.126+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:44.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:44.126+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:44.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:44.127+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:44.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:44.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:44.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:44.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:44.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:44.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:44.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:44.170+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:44.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:44.707+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:44.709019+0000) 2024-04-27T03:45:44.815 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:44.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:44.815 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:44.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:44.815 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:44.522+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:44.815 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:44.524+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:45.104 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:45.024+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:45.026028+0000) 2024-04-27T03:45:45.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:44.807+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:44.809281+0000) 2024-04-27T03:45:45.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:45.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:45.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:45.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:45.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:45.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:45.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:45.125+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:45.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:45.149+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:45.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:45.149+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:45.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:45.149+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:45.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:45.150+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:45.817 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:45.712+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:45.713845+0000) 2024-04-27T03:45:45.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:45.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:45.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:45.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:45.818 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:45.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:45.818 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:45.537+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:46.104 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:45.809+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:45.811604+0000) 2024-04-27T03:45:46.104 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:46.026+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:46.028170+0000) 2024-04-27T03:45:46.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:46.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:46.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:46.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:46.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:46.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:46.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:46.144+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:46.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:46.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:46.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:46.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:46.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:46.118+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:46.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:46.119+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:46.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:46.541+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:46.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:46.541+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:46.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:46.541+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:46.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:46.541+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:46.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:46.715+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:46.717036+0000) 2024-04-27T03:45:46.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:46.812+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:46.813909+0000) 2024-04-27T03:45:47.292 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:45:46.989+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71471 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:45:47.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:46.985+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:46.987793+0000) 2024-04-27T03:45:47.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:46.987+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:46.988980+0000) 2024-04-27T03:45:47.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:47.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:47.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:47.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:47.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:47.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:47.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:47.148+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:47.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:46.985+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:46.987539+0000) 2024-04-27T03:45:47.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:47.028+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:47.029825+0000) 2024-04-27T03:45:47.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:47.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:47.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:47.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:47.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:47.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:47.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:47.179+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:47.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:47.717+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:47.718849+0000) 2024-04-27T03:45:47.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:47.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:47.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:47.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:47.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:47.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:47.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:47.557+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:48.104 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:48.032+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:48.034561+0000) 2024-04-27T03:45:48.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:47.813+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:47.815630+0000) 2024-04-27T03:45:48.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:48.154+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:48.154+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:48.154+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:48.155+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:48.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:48.100+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:48.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:48.100+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:48.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:48.100+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:48.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:48.102+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:48.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:48.719+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:48.720920+0000) 2024-04-27T03:45:48.824 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:48.525+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:48.824 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:48.525+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:48.824 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:48.525+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:48.824 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:48.526+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:49.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:49.035+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:49.036816+0000) 2024-04-27T03:45:49.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:48.816+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:48.817846+0000) 2024-04-27T03:45:49.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:49.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:49.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:49.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:49.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:49.101+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:49.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:49.102+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:49.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:49.187+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:49.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:49.187+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:49.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:49.187+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:49.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:49.187+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:49.825 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:49.721+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:49.723681+0000) 2024-04-27T03:45:49.826 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:49.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:49.826 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:49.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:49.826 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:49.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:49.826 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:49.495+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:50.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:50.038+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:50.040092+0000) 2024-04-27T03:45:50.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:49.819+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:49.819868+0000) 2024-04-27T03:45:50.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:50.073+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:50.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:50.073+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:50.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:50.073+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:50.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:50.074+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:50.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:50.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:50.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:50.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:50.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:50.160+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:50.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:50.161+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:50.828 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:50.725+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:50.726031+0000) 2024-04-27T03:45:50.828 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:50.489+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:50.828 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:50.489+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:50.828 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:50.489+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:50.828 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:50.490+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:51.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:50.821+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:50.822511+0000) 2024-04-27T03:45:51.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:51.041+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:51.042581+0000) 2024-04-27T03:45:51.105 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:51.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:51.106 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:51.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:51.106 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:51.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:51.106 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:51.050+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:51.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:51.153+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:51.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:51.153+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:51.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:51.153+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:51.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:51.154+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:51.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:51.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:51.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:51.462+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:51.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:51.462+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:51.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:51.462+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:51.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:51.727+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:51.728247+0000) 2024-04-27T03:45:51.998 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:51.824+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:51.824800+0000) 2024-04-27T03:45:52.293 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:45:52.001+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71512 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:45:52.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:51.996+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:51.997181+0000) 2024-04-27T03:45:52.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:51.995+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:51.996234+0000) 2024-04-27T03:45:52.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:52.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:52.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:52.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:52.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:52.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:52.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:52.024+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:52.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:51.995+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:51.996045+0000) 2024-04-27T03:45:52.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:52.043+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:52.044337+0000) 2024-04-27T03:45:52.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:52.120+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:52.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:52.120+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:52.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:52.120+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:52.295 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:52.120+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:52.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:52.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:52.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:52.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:52.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:52.446+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:52.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:52.448+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:52.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:52.825+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:52.826533+0000) 2024-04-27T03:45:52.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:52.730+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:52.731713+0000) 2024-04-27T03:45:53.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:53.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:53.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:53.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:53.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:53.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:53.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:53.002+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:53.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:53.046+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:53.047346+0000) 2024-04-27T03:45:53.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:53.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:53.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:53.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:53.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:53.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:53.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:53.146+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:53.627 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:53.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:53.627 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:53.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:53.627 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:53.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:53.627 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:53.473+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:53.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:53.734+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:53.734980+0000) 2024-04-27T03:45:53.908 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:53.830+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:53.831600+0000) 2024-04-27T03:45:54.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:53.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:54.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:53.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:54.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:53.981+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:54.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:53.982+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:54.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:54.048+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:54.049658+0000) 2024-04-27T03:45:54.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:54.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:54.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:54.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:54.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:54.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:54.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:54.101+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:54.743 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:54.471+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:54.743 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:54.471+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:54.744 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:54.471+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:54.744 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:54.472+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:55.003 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:54.736+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:54.737706+0000) 2024-04-27T03:45:55.004 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:54.835+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:54.836546+0000) 2024-04-27T03:45:55.293 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:55.052+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:55.052939+0000) 2024-04-27T03:45:55.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:55.126+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:55.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:55.126+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:55.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:55.126+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:55.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:55.127+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:55.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:55.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:55.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:55.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:55.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:55.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:55.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:55.012+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:55.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:55.435+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:55.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:55.435+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:55.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:55.435+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:55.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:55.436+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:55.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:55.739+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:55.740201+0000) 2024-04-27T03:45:55.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:55.840+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:55.841707+0000) 2024-04-27T03:45:56.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:55.980+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:56.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:55.980+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:56.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:55.980+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:56.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:55.981+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:56.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:56.055+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:56.055840+0000) 2024-04-27T03:45:56.295 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:56.131+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:56.295 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:56.131+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:56.295 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:56.131+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:56.295 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:56.131+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:56.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:56.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:56.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:56.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:56.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:56.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:56.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:56.440+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:56.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:56.741+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:56.742319+0000) 2024-04-27T03:45:56.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:56.845+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:56.846709+0000) 2024-04-27T03:45:57.226 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:45:57.009+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71550 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:45:57.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:57.006+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:57.007468+0000) 2024-04-27T03:45:57.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:56.949+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:57.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:56.949+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:57.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:56.949+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:57.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:56.951+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:57.228 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:57.006+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:57.007468+0000) 2024-04-27T03:45:57.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:57.006+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:57.007010+0000) 2024-04-27T03:45:57.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:57.057+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:57.057821+0000) 2024-04-27T03:45:57.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:57.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:57.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:57.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:57.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:57.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:57.228 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:57.161+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:57.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:57.405+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:57.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:57.405+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:57.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:57.405+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:57.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:57.406+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:57.743+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:57.744052+0000) 2024-04-27T03:45:57.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:57.849+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:57.850662+0000) 2024-04-27T03:45:58.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:58.060+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:58.061035+0000) 2024-04-27T03:45:58.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:58.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:58.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:58.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:58.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:58.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:58.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:58.179+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:58.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:57.978+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:58.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:57.978+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:58.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:57.978+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:58.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:57.979+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:58.633 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:58.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:58.633 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:58.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:58.633 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:58.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:58.634 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:58.455+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:58.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:58.852+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:58.853401+0000) 2024-04-27T03:45:58.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:58.745+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:58.746776+0000) 2024-04-27T03:45:59.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:59.063+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:59.063824+0000) 2024-04-27T03:45:59.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:59.139+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:59.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:59.139+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:59.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:59.139+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:45:59.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:45:59.139+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:59.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:58.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:59.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:58.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:59.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:58.974+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:45:59.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:58.974+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:59.636 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:59.504+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:59.637 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:59.504+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:59.637 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:59.504+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:45:59.637 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:59.505+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:45:59.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:59.748+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:59.748918+0000) 2024-04-27T03:45:59.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:45:59.855+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:45:59.856657+0000) 2024-04-27T03:46:00.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:59.944+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:00.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:59.944+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:00.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:59.944+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:00.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:45:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:45:59.945+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:00.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:00.066+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:00.067081+0000) 2024-04-27T03:46:00.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:00.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:00.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:00.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:00.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:00.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:00.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:00.112+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:00.637 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:00.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:00.637 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:00.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:00.637 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:00.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:00.637 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:00.466+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:00.908 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:00.858+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:00.859424+0000) 2024-04-27T03:46:00.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:00.752+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:00.753118+0000) 2024-04-27T03:46:00.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:00.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:00.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:00.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:01.175 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:00.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:01.175 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:00.902+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:01.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:01.068+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:01.069204+0000) 2024-04-27T03:46:01.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:01.138+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:01.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:01.138+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:01.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:01.138+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:01.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:01.139+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:01.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:01.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:01.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:01.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:01.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:01.433+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:01.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:01.434+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:02.017 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:01.861+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:01.862457+0000) 2024-04-27T03:46:02.017 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:01.754+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:01.755323+0000) 2024-04-27T03:46:02.017 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:01.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:02.017 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:01.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:02.018 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:01.860+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:02.018 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:01.862+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:02.294 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:46:02.018+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71591 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:46:02.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:02.016+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:02.017516+0000) 2024-04-27T03:46:02.295 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:02.014+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:02.015379+0000) 2024-04-27T03:46:02.295 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:02.014+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:02.015499+0000) 2024-04-27T03:46:02.295 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:02.070+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:02.071618+0000) 2024-04-27T03:46:02.295 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:02.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:02.295 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:02.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:02.295 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:02.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:02.295 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:02.185+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:02.637 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:02.416+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:02.637 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:02.416+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:02.637 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:02.416+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:02.637 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:02.417+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:02.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:02.756+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:02.757617+0000) 2024-04-27T03:46:02.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:02.908+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:02.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:02.908+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:02.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:02.865+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:02.865841+0000) 2024-04-27T03:46:03.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:03.073+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:03.074381+0000) 2024-04-27T03:46:03.175 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:02.908+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:03.175 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:02.909+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:03.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:03.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:03.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:03.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:03.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:03.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:03.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:03.370+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:03.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:03.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:03.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:03.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:03.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:03.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:03.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:03.221+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:04.085 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:03.867+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:03.868484+0000) 2024-04-27T03:46:04.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:03.759+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:03.760353+0000) 2024-04-27T03:46:04.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:03.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:04.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:03.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:04.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:03.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:04.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:03.921+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:04.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:04.078+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:04.079172+0000) 2024-04-27T03:46:04.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:04.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:04.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:04.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:04.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:04.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:04.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:04.237+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:04.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:04.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:04.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:04.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:04.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:04.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:04.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:04.345+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:04.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:04.872+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:04.873219+0000) 2024-04-27T03:46:04.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:04.761+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:04.762514+0000) 2024-04-27T03:46:04.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:04.883+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:04.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:04.883+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:04.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:04.883+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:04.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:04.884+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:05.285 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:05.080+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:05.081484+0000) 2024-04-27T03:46:05.644 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:05.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:05.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:05.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:05.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:05.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:05.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:05.279+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:05.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:05.320+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:05.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:05.320+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:05.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:05.320+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:05.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:05.321+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:05.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:05.875+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:05.876214+0000) 2024-04-27T03:46:05.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:05.764+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:05.765559+0000) 2024-04-27T03:46:05.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:05.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:05.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:05.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:05.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:05.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:05.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:05.839+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:06.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:06.083+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:06.084323+0000) 2024-04-27T03:46:06.295 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:06.234+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:06.295 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:06.234+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:06.295 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:06.234+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:06.295 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:06.235+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:06.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:06.332+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:06.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:06.332+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:06.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:06.332+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:06.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:06.333+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:06.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:06.878+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:06.878886+0000) 2024-04-27T03:46:06.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:06.767+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:06.768339+0000) 2024-04-27T03:46:06.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:06.867+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:06.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:06.867+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:06.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:06.867+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:06.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:06.869+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:07.275 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:46:07.026+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71629 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:46:07.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:07.023+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:07.024345+0000) 2024-04-27T03:46:07.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:07.023+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:07.024308+0000) 2024-04-27T03:46:07.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:07.022+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:07.022921+0000) 2024-04-27T03:46:07.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:07.085+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:07.086679+0000) 2024-04-27T03:46:07.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:07.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:07.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:07.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:07.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:07.269+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:07.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:07.269+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:07.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:07.381+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:07.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:07.381+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:07.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:07.381+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:07.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:07.382+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:07.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:07.881+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:07.882422+0000) 2024-04-27T03:46:07.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:07.769+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:07.770649+0000) 2024-04-27T03:46:07.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:07.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:07.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:07.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:07.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:07.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:07.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:07.874+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:08.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:08.088+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:08.088826+0000) 2024-04-27T03:46:08.552 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:08.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:08.552 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:08.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:08.552 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:08.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:08.552 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:08.348+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:08.552 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:08.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:08.552 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:08.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:08.552 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:08.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:08.552 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:08.312+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:08.849 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:08.771+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:08.772726+0000) 2024-04-27T03:46:08.849 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:08.842+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:08.849 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:08.842+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:09.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:08.884+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:08.885270+0000) 2024-04-27T03:46:09.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:09.090+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:09.091478+0000) 2024-04-27T03:46:09.107 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:08.842+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:09.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:08.844+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:09.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:09.321+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:09.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:09.321+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:09.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:09.321+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:09.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:09.321+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:09.652 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:09.375+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:09.652 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:09.375+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:09.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:09.375+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:09.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:09.376+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:09.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:09.889+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:09.890037+0000) 2024-04-27T03:46:09.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:09.776+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:09.776953+0000) 2024-04-27T03:46:09.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:09.799+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:09.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:09.799+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:09.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:09.799+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:09.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:09.800+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:10.284 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:10.092+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:10.093628+0000) 2024-04-27T03:46:10.652 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:10.362+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:10.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:10.362+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:10.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:10.362+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:10.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:10.363+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:10.653 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:10.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:10.653 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:10.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:10.653 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:10.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:10.653 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:10.278+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:10.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:10.891+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:10.892293+0000) 2024-04-27T03:46:10.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:10.778+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:10.779236+0000) 2024-04-27T03:46:10.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:10.797+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:10.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:10.797+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:10.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:10.797+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:10.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:10.798+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:11.295 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:11.097+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:11.098420+0000) 2024-04-27T03:46:11.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:11.335+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:11.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:11.335+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:11.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:11.335+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:11.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:11.336+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:11.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:11.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:11.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:11.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:11.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:11.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:11.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:11.297+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:11.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:11.895+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:11.896631+0000) 2024-04-27T03:46:11.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:11.780+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:11.781544+0000) 2024-04-27T03:46:11.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:11.797+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:11.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:11.797+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:11.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:11.797+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:11.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:11.798+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:12.259 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:46:12.034+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71670 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:46:12.259 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:12.031+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:12.031855+0000) 2024-04-27T03:46:12.260 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:12.032+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:12.033466+0000) 2024-04-27T03:46:12.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:12.033+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:12.033847+0000) 2024-04-27T03:46:12.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:12.099+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:12.100391+0000) 2024-04-27T03:46:12.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:12.252+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:12.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:12.252+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:12.260 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:12.252+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:12.653 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:12.253+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:12.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:12.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:12.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:12.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:12.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:12.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:12.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:12.367+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:12.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:12.783+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:12.784040+0000) 2024-04-27T03:46:12.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:12.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:12.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:12.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:12.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:12.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:12.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:12.838+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:13.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:12.898+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:12.898989+0000) 2024-04-27T03:46:13.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:13.102+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:13.103156+0000) 2024-04-27T03:46:13.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:13.259+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:13.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:13.259+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:13.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:13.259+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:13.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:13.259+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:13.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:13.351+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:13.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:13.351+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:13.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:13.351+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:13.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:13.351+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:13.846 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:13.786+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:13.787187+0000) 2024-04-27T03:46:13.846 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:13.839+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:13.847 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:13.839+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:13.847 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:13.839+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:14.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:13.840+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:14.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:13.901+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:13.901819+0000) 2024-04-27T03:46:14.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:14.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:14.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:14.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:14.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:14.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:14.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:14.349+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:14.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:14.105+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:14.105932+0000) 2024-04-27T03:46:14.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:14.305+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:14.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:14.305+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:14.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:14.305+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:14.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:14.306+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:14.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:14.788+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:14.789488+0000) 2024-04-27T03:46:14.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:14.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:14.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:14.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:14.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:14.887+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:14.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:14.888+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:15.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:14.903+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:14.903952+0000) 2024-04-27T03:46:15.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:15.108+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:15.109099+0000) 2024-04-27T03:46:15.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:15.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:15.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:15.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:15.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:15.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:15.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:15.366+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:15.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:15.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:15.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:15.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:15.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:15.311+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:15.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:15.311+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:15.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:15.790+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:15.791694+0000) 2024-04-27T03:46:16.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:15.906+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:15.907127+0000) 2024-04-27T03:46:16.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:16.110+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:16.111301+0000) 2024-04-27T03:46:16.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:15.918+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:16.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:15.918+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:16.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:15.918+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:16.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:15.919+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:16.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:16.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:16.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:16.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:16.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:16.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:16.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:16.398+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:16.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:16.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:16.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:16.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:16.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:16.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:16.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:16.340+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:16.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:16.793+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:16.793802+0000) 2024-04-27T03:46:16.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:16.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:16.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:16.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:16.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:16.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:16.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:16.891+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:17.174 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:46:17.045+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71708 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:46:17.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:17.040+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:17.041404+0000) 2024-04-27T03:46:17.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:17.040+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:17.041274+0000) 2024-04-27T03:46:17.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:17.113+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:17.114784+0000) 2024-04-27T03:46:17.175 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:16.909+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:16.910533+0000) 2024-04-27T03:46:17.175 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:17.040+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:17.041054+0000) 2024-04-27T03:46:17.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:17.410+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:17.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:17.410+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:17.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:17.410+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:17.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:17.411+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:17.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:17.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:17.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:17.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:17.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:17.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:17.479 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:17.330+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:17.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:17.794+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:17.795558+0000) 2024-04-27T03:46:18.177 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:17.912+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:17.913100+0000) 2024-04-27T03:46:18.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:18.116+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:18.116964+0000) 2024-04-27T03:46:18.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:17.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:18.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:17.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:18.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:17.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:18.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:17.932+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:18.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:18.305+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:18.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:18.305+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:18.459 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:18.305+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:18.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:18.306+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:18.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:18.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:18.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:18.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:18.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:18.452+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:18.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:18.453+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:19.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:18.914+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:18.915387+0000) 2024-04-27T03:46:19.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:18.799+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:18.800546+0000) 2024-04-27T03:46:19.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:18.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:19.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:18.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:19.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:18.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:19.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:18.942+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:19.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:19.118+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:19.119134+0000) 2024-04-27T03:46:19.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:19.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:19.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:19.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:19.464 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:19.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:19.465 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:19.331+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:19.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:19.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:19.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:19.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:19.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:19.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:19.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:19.459+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:20.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:19.918+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:19.918848+0000) 2024-04-27T03:46:20.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:19.804+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:19.805640+0000) 2024-04-27T03:46:20.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:19.979+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:20.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:19.979+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:20.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:19.979+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:20.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:19.981+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:20.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:20.120+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:20.121365+0000) 2024-04-27T03:46:20.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:20.322+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:20.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:20.322+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:20.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:20.322+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:20.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:20.323+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:20.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:20.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:20.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:20.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:20.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:20.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:20.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:20.435+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:21.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:20.923+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:20.923914+0000) 2024-04-27T03:46:21.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:20.806+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:20.807784+0000) 2024-04-27T03:46:21.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:20.993+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:21.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:20.993+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:21.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:20.993+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:21.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:20.995+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:21.444 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:21.125+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:21.126469+0000) 2024-04-27T03:46:21.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:21.333+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:21.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:21.333+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:21.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:21.333+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:21.445 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:21.333+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:21.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:21.449+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:21.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:21.449+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:21.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:21.449+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:21.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:21.449+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:22.067 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:46:22.053+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71749 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:46:22.068 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:22.049+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:22.050255+0000) 2024-04-27T03:46:22.068 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:21.928+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:21.929028+0000) 2024-04-27T03:46:22.068 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:22.049+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:22.050139+0000) 2024-04-27T03:46:22.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:21.809+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:21.809907+0000) 2024-04-27T03:46:22.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:21.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:22.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:21.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:22.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:21.989+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:22.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:21.991+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:22.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:22.049+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:22.050488+0000) 2024-04-27T03:46:22.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:22.129+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:22.130742+0000) 2024-04-27T03:46:22.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:22.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:22.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:22.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:22.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:22.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:22.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:22.380+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:22.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:22.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:22.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:22.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:22.669 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:22.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:22.669 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:22.470+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:22.938 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:22.810+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:22.811745+0000) 2024-04-27T03:46:23.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:23.132+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:23.133480+0000) 2024-04-27T03:46:23.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:22.932+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:22.933243+0000) 2024-04-27T03:46:23.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:22.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:23.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:22.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:23.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:22.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:23.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:22.946+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:23.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:23.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:23.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:23.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:23.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:23.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:23.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:23.407+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:23.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:23.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:23.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:23.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:23.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:23.472+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:23.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:23.473+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:23.942 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:23.813+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:23.813957+0000) 2024-04-27T03:46:23.942 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:23.909+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:23.942 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:23.909+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:23.942 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:23.909+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:23.942 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:23.910+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:24.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:24.135+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:24.136586+0000) 2024-04-27T03:46:24.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:23.935+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:23.936123+0000) 2024-04-27T03:46:24.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:24.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:24.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:24.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:24.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:24.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:24.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:24.435+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:24.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:24.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:24.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:24.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:24.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:24.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:24.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:24.368+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:24.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:24.817+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:24.818172+0000) 2024-04-27T03:46:24.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:24.911+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:24.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:24.911+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:24.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:24.911+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:24.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:24.912+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:25.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:24.940+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:24.940872+0000) 2024-04-27T03:46:25.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:25.138+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:25.139623+0000) 2024-04-27T03:46:25.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:25.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:25.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:25.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:25.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:25.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:25.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:25.409+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:25.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:25.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:25.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:25.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:25.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:25.470+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:25.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:25.471+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:25.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:25.820+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:25.820820+0000) 2024-04-27T03:46:26.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:26.142+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:26.142990+0000) 2024-04-27T03:46:26.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:25.943+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:25.944059+0000) 2024-04-27T03:46:26.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:25.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:26.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:25.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:26.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:25.959+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:26.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:25.960+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:26.516 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:26.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:26.516 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:26.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:26.516 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:26.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:26.516 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:26.403+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:26.829 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:26.509+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:26.829 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:26.509+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:26.829 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:26.509+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:26.829 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:26.509+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:27.086 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:46:27.061+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71787 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:46:27.086 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:27.058+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:27.059049+0000) 2024-04-27T03:46:27.087 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:26.945+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:26.946179+0000) 2024-04-27T03:46:27.087 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:27.058+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:27.059125+0000) 2024-04-27T03:46:27.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:26.822+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:26.823163+0000) 2024-04-27T03:46:27.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:26.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:27.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:26.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:27.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:26.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:27.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:26.972+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:27.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:27.058+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:27.059080+0000) 2024-04-27T03:46:27.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:27.144+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:27.144910+0000) 2024-04-27T03:46:27.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:27.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:27.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:27.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:27.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:27.418+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:27.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:27.418+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:27.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:27.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:27.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:27.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:27.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:27.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:27.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:27.522+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:28.109 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:27.947+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:27.948481+0000) 2024-04-27T03:46:28.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:27.826+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:27.827338+0000) 2024-04-27T03:46:28.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:28.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:28.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:28.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:28.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:28.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:28.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:28.017+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:28.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:28.147+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:28.148109+0000) 2024-04-27T03:46:28.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:28.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:28.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:28.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:28.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:28.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:28.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:28.384+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:28.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:28.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:28.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:28.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:28.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:28.535+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:28.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:28.536+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:28.957 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:28.829+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:28.830002+0000) 2024-04-27T03:46:29.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:29.150+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:29.151172+0000) 2024-04-27T03:46:29.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:28.950+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:28.951371+0000) 2024-04-27T03:46:29.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:29.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:29.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:29.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:29.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:29.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:29.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:29.042+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:29.578 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:29.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:29.578 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:29.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:29.578 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:29.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:29.578 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:29.429+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:29.838 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:29.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:29.838 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:29.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:29.838 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:29.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:29.838 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:29.572+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:30.109 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:29.953+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:29.954527+0000) 2024-04-27T03:46:30.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:29.831+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:29.832309+0000) 2024-04-27T03:46:30.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:30.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:30.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:30.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:30.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:30.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:30.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:30.060+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:30.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:30.153+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:30.154260+0000) 2024-04-27T03:46:30.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:30.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:30.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:30.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:30.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:30.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:30.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:30.436+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:30.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:30.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:30.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:30.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:30.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:30.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:30.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:30.536+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:31.109 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:30.956+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:30.957352+0000) 2024-04-27T03:46:31.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:30.834+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:30.835234+0000) 2024-04-27T03:46:31.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:31.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:31.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:31.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:31.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:31.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:31.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:31.055+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:31.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:31.156+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:31.157251+0000) 2024-04-27T03:46:31.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:31.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:31.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:31.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:31.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:31.551+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:31.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:31.552+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:31.844 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:31.471+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:31.844 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:31.471+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:31.844 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:31.471+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:31.844 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:31.471+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:32.095 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:46:32.071+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71828 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:46:32.096 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:32.066+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:32.067312+0000) 2024-04-27T03:46:32.096 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:31.959+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:31.960388+0000) 2024-04-27T03:46:32.096 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:32.066+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:32.067220+0000) 2024-04-27T03:46:32.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:31.836+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:31.837749+0000) 2024-04-27T03:46:32.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:32.067+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:32.068136+0000) 2024-04-27T03:46:32.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:32.159+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:32.160626+0000) 2024-04-27T03:46:32.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:32.092+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:32.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:32.092+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:32.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:32.092+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:32.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:32.093+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:32.846 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:32.478+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:32.846 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:32.478+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:32.846 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:32.478+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:32.846 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:32.479+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:32.847 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:32.541+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:32.847 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:32.541+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:32.847 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:32.541+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:32.847 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:32.542+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:33.105 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:32.839+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:32.840016+0000) 2024-04-27T03:46:33.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:32.962+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:32.962930+0000) 2024-04-27T03:46:33.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:33.161+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:33.162762+0000) 2024-04-27T03:46:33.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:33.098+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:33.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:33.098+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:33.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:33.098+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:33.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:33.100+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:33.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:33.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:33.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:33.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:33.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:33.576+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:33.789 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:33.577+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:33.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:33.513+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:33.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:33.513+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:33.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:33.513+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:33.790 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:33.513+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:34.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:33.964+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:33.965030+0000) 2024-04-27T03:46:34.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:33.844+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:33.844966+0000) 2024-04-27T03:46:34.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:34.165+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:34.166018+0000) 2024-04-27T03:46:34.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:34.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:34.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:34.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:34.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:34.113+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:34.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:34.114+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:34.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:34.528+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:34.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:34.528+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:34.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:34.528+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:34.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:34.529+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:34.856 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:34.547+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:34.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:34.547+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:34.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:34.547+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:34.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:34.547+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:35.110 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:34.966+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:34.967179+0000) 2024-04-27T03:46:35.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:34.849+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:34.850087+0000) 2024-04-27T03:46:35.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:35.167+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:35.168134+0000) 2024-04-27T03:46:35.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:35.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:35.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:35.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:35.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:35.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:35.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:35.109+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:35.861 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:35.511+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:35.861 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:35.511+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:35.861 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:35.511+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:35.861 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:35.511+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:35.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:35.552+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:35.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:35.552+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:35.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:35.552+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:35.862 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:35.554+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:36.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:35.968+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:35.969319+0000) 2024-04-27T03:46:36.139 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:35.854+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:35.855029+0000) 2024-04-27T03:46:36.139 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:36.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:36.139 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:36.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:36.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:36.172+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:36.173264+0000) 2024-04-27T03:46:36.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:36.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:36.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:36.133+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:36.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:36.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:36.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:36.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:36.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:36.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:36.804 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:36.592+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:36.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:36.481+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:36.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:36.481+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:36.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:36.481+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:36.804 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:36.482+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:37.079 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:36.859+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:36.859878+0000) 2024-04-27T03:46:37.079 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:36.971+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:36.972273+0000) 2024-04-27T03:46:37.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:46:37.081+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71867 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:46:37.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:37.075+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:37.076784+0000) 2024-04-27T03:46:37.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:37.077+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:37.078317+0000) 2024-04-27T03:46:37.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:37.176+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:37.177617+0000) 2024-04-27T03:46:37.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:37.453+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:37.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:37.453+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:37.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:37.453+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:37.461 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:37.076+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:37.077588+0000) 2024-04-27T03:46:37.461 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:37.087+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:37.461 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:37.087+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:37.461 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:37.087+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:37.461 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:37.088+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:37.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:37.453+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:37.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:37.609+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:37.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:37.609+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:37.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:37.609+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:37.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:37.609+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:38.179 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:37.973+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:37.974671+0000) 2024-04-27T03:46:38.179 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:37.861+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:37.862254+0000) 2024-04-27T03:46:38.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:38.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:38.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:38.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:38.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:38.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:38.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:38.041+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:38.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:38.179+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:38.179838+0000) 2024-04-27T03:46:38.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:38.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:38.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:38.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:38.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:38.433+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:38.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:38.433+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:38.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:38.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:38.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:38.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:38.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:38.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:38.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:38.643+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:38.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:38.864+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:38.864898+0000) 2024-04-27T03:46:39.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:38.975+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:38.976769+0000) 2024-04-27T03:46:39.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:39.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:39.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:39.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:39.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:39.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:39.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:39.054+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:39.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:39.181+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:39.181901+0000) 2024-04-27T03:46:39.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:39.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:39.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:39.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:39.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:39.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:39.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:39.416+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:39.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:39.867+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:39.868596+0000) 2024-04-27T03:46:39.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:39.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:39.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:39.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:39.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:39.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:39.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:39.669+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:40.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:40.183+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:40.184652+0000) 2024-04-27T03:46:40.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:40.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:40.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:40.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:40.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:40.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:40.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:40.009+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:40.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:39.979+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:39.979995+0000) 2024-04-27T03:46:40.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:40.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:40.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:40.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:40.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:40.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:40.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:40.415+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:40.965 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:40.871+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:40.872920+0000) 2024-04-27T03:46:40.965 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:40.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:40.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:40.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:40.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:40.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:40.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:40.670+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:41.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:40.980+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:40.982729+0000) 2024-04-27T03:46:41.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:41.187+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:41.188984+0000) 2024-04-27T03:46:41.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:41.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:41.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:41.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:41.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:41.008+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:41.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:41.008+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:41.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:41.411+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:41.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:41.411+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:41.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:41.411+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:41.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:41.411+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:41.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:41.874+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:41.876159+0000) 2024-04-27T03:46:41.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:41.641+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:41.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:41.641+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:41.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:41.641+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:41.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:41.642+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:42.299 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:46:42.088+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71907 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:46:42.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:42.086+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:42.087865+0000) 2024-04-27T03:46:42.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:42.189+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:42.191780+0000) 2024-04-27T03:46:42.300 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:41.984+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:41.986014+0000) 2024-04-27T03:46:42.300 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:42.081+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:42.087453+0000) 2024-04-27T03:46:42.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:41.991+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:42.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:41.991+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:42.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:41.991+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:42.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:41.992+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:42.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:42.081+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:42.087235+0000) 2024-04-27T03:46:42.632 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:42.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:42.632 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:42.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:42.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:42.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:42.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:42.373+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:42.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:42.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:42.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:42.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:42.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:42.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:42.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:42.626+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:43.180 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:42.986+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:42.988411+0000) 2024-04-27T03:46:43.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:42.876+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:42.878474+0000) 2024-04-27T03:46:43.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:42.965+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:43.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:42.965+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:43.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:42.965+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:43.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:42.966+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:43.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:43.192+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:43.193994+0000) 2024-04-27T03:46:43.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:43.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:43.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:43.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:43.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:43.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:43.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:43.344+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:43.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:43.878+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:43.880702+0000) 2024-04-27T03:46:43.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:43.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:43.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:43.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:43.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:43.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:43.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:43.634+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:44.179 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:43.989+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:43.991199+0000) 2024-04-27T03:46:44.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:43.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:44.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:43.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:44.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:43.941+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:44.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:43.942+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:44.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:44.194+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:44.196138+0000) 2024-04-27T03:46:44.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:44.313+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:44.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:44.313+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:44.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:44.313+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:44.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:44.314+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:44.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:44.882+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:44.883907+0000) 2024-04-27T03:46:44.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:44.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:44.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:44.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:44.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:44.899+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:44.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:44.900+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:44.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:44.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:44.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:44.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:44.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:44.658+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:44.978 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:44.658+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:45.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:44.991+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:44.993327+0000) 2024-04-27T03:46:45.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:45.197+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:45.198827+0000) 2024-04-27T03:46:45.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:45.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:45.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:45.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:45.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:45.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:45.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:45.302+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:45.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:45.675+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:45.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:45.675+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:45.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:45.675+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:45.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:45.676+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:45.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:45.884+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:45.885838+0000) 2024-04-27T03:46:45.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:45.892+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:45.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:45.892+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:45.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:45.892+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:45.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:45.893+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:46.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:45.993+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:45.995458+0000) 2024-04-27T03:46:46.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:46.199+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:46.200986+0000) 2024-04-27T03:46:46.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:46.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:46.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:46.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:46.655 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:46.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:46.655 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:46.292+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:46.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:46.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:46.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:46.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:46.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:46.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:46.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:46.647+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:46.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:46.886+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:46.888112+0000) 2024-04-27T03:46:46.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:46.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:46.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:46.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:46.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:46.902+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:47.180 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:46:47.098+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71946 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:46:47.181 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:47.094+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:47.096187+0000) 2024-04-27T03:46:47.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:46.903+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:47.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:47.098+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:47.100434+0000) 2024-04-27T03:46:47.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:46.996+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:46.998299+0000) 2024-04-27T03:46:47.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:47.094+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:47.096590+0000) 2024-04-27T03:46:47.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:47.201+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:47.203039+0000) 2024-04-27T03:46:47.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:47.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:47.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:47.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:47.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:47.249+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:47.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:47.250+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:47.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:47.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:47.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:47.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:47.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:47.619+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:47.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:47.620+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:48.180 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:48.000+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:48.001977+0000) 2024-04-27T03:46:48.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:47.888+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:47.889953+0000) 2024-04-27T03:46:48.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:47.919+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:48.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:47.919+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:48.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:47.919+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:48.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:47.920+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:48.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:48.203+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:48.205266+0000) 2024-04-27T03:46:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:48.242+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:48.242+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:48.242+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:48.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:48.243+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:48.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:48.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:48.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:48.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:48.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:48.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:48.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:48.614+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:49.180 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:49.002+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:49.004187+0000) 2024-04-27T03:46:49.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:48.890+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:48.892613+0000) 2024-04-27T03:46:49.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:48.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:49.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:48.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:49.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:48.890+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:49.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:48.892+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:49.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:49.205+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:49.207482+0000) 2024-04-27T03:46:49.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:49.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:49.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:49.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:49.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:49.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:49.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:49.240+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:49.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-27T03:46:49.610+0000 7ff0e4ed9700 0 monclient(hunting): authenticate timed out after 300 2024-04-27T03:46:49.618 INFO:teuthology.orchestra.run.smithi029.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-04-27T03:46:49.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:49.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:49.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:49.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:49.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:49.577+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:49.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:49.578+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:50.112 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:50.004+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:50.006569+0000) 2024-04-27T03:46:50.112 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:49.853+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:50.112 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:49.853+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:50.112 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:49.853+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:50.112 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:49.854+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:50.112 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:49.895+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:49.897380+0000) 2024-04-27T03:46:50.405 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:50.208+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:50.210383+0000) 2024-04-27T03:46:50.405 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:50.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:50.405 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:50.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:50.406 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:50.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:50.406 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:50.260+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:50.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T03:46:50.426 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/cephadm.py", line 1469, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/cephadm.py", line 110, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi029 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph orch ps'" 2024-04-27T03:46:50.671 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3fadf09fe1454846857965df30295752 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/cephadm.py", line 1469, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/cephadm.py", line 110, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi029 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph orch ps'" 2024-04-27T03:46:50.675 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-27T03:46:50.687 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-27T03:46:50.687 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-27T03:46:50.687 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-27T03:46:50.707 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:50.529+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:50.707 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:50.529+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:50.707 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:50.529+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:50.707 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:50.530+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:50.712 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-27T03:46:50.712 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-27T03:46:50.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T03:46:50.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:50.848+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:50.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:50.848+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:50.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:50.848+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:50.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:50.849+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:50.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:50.897+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:50.899661+0000) 2024-04-27T03:46:51.297 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:51.007+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:51.008720+0000) 2024-04-27T03:46:51.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:51.211+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:51.213239+0000) 2024-04-27T03:46:51.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:51.291+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:51.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:51.291+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:51.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:51.291+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:51.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:51.291+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:51.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:51.481+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:51.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:51.481+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:51.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:51.481+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:51.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:51.481+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:51.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:51.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:51.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:51.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:51.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:51.827+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:51.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:51.827+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:51.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:51.902+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:51.904510+0000) 2024-04-27T03:46:52.268 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:52.104+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:52.105847+0000) 2024-04-27T03:46:52.268 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:46:52.107+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 71986 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:46:52.269 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:52.102+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:52.104797+0000) 2024-04-27T03:46:52.269 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:52.214+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:52.215918+0000) 2024-04-27T03:46:52.269 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:52.010+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:52.011975+0000) 2024-04-27T03:46:52.269 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:52.103+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:52.105112+0000) 2024-04-27T03:46:52.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:52.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:52.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:52.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:52.708 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:52.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:52.708 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:52.261+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:52.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:52.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:52.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:52.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:52.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:52.456+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:52.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:52.457+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:52.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:52.814+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:52.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:52.814+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:52.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:52.814+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:52.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:52.815+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:52.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:52.905+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:52.906551+0000) 2024-04-27T03:46:53.300 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:53.013+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:53.014351+0000) 2024-04-27T03:46:53.300 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:53.219+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:53.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:53.219+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:53.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:53.219+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:53.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:53.219+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:53.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:53.220+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:53.220979+0000) 2024-04-27T03:46:53.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:53.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:53.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:53.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:53.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:53.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:53.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:53.443+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:53.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:53.812+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:53.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:53.812+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:53.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:53.812+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:53.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:53.813+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:53.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:53.908+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:53.908860+0000) 2024-04-27T03:46:54.300 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:54.016+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:54.017353+0000) 2024-04-27T03:46:54.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:54.223+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:54.224068+0000) 2024-04-27T03:46:54.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:54.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:54.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:54.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:54.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:54.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:54.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:54.228+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:54.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:54.399+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:54.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:54.399+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:54.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:54.399+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:54.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:54.399+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:54.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:54.785+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:54.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:54.785+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:54.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:54.786+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:54.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:54.786+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:54.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:54.910+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:54.911599+0000) 2024-04-27T03:46:55.300 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:55.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:55.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:55.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:55.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:55.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:55.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:55.216+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:55.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:55.225+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:55.226254+0000) 2024-04-27T03:46:55.301 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:55.019+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:55.020293+0000) 2024-04-27T03:46:55.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:55.355+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:55.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:55.355+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:55.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:55.355+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:55.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:55.355+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:55.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:55.754+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:55.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:55.754+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:55.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:55.754+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:55.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:55.755+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:55.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:55.913+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:55.913882+0000) 2024-04-27T03:46:56.272 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:56.228+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:56.228912+0000) 2024-04-27T03:46:56.272 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:56.022+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:56.022979+0000) 2024-04-27T03:46:56.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:56.265+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:56.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:56.265+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:56.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:56.265+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:56.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:56.266+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:56.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:56.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:56.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:56.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:56.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:56.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:56.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:56.315+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:56.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:56.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:56.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:56.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:56.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:56.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:56.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:56.719+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:56.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:56.915+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:56.916070+0000) 2024-04-27T03:46:57.279 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:46:57.117+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72025 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:46:57.280 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:57.112+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:57.113743+0000) 2024-04-27T03:46:57.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:57.025+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:57.026202+0000) 2024-04-27T03:46:57.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:57.113+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:57.113833+0000) 2024-04-27T03:46:57.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:57.112+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:57.113359+0000) 2024-04-27T03:46:57.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:57.230+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:57.231620+0000) 2024-04-27T03:46:57.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:57.272+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:57.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:57.272+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:57.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:57.272+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:57.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:57.273+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:57.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:57.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:57.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:57.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:57.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:57.292+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:57.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:57.292+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:57.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:57.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:57.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:57.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:57.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:57.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:57.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:57.688+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:57.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:57.917+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:57.918757+0000) 2024-04-27T03:46:58.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:58.232+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:58.233772+0000) 2024-04-27T03:46:58.301 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:58.028+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:58.028823+0000) 2024-04-27T03:46:58.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:58.342+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:58.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:58.342+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:58.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:58.342+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:58.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:58.342+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:58.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:58.319+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:58.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:58.319+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:58.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:58.319+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:58.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:58.319+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:58.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:58.717+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:58.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:58.717+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:58.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:58.717+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:58.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:58.718+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:58.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:58.920+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:58.921444+0000) 2024-04-27T03:46:59.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:59.237+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:59.238109+0000) 2024-04-27T03:46:59.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:59.031+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:59.032185+0000) 2024-04-27T03:46:59.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:59.349+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:59.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:59.349+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:59.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:59.349+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:46:59.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:46:59.350+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:59.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:59.272+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:59.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:59.272+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:59.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:59.272+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:46:59.678 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:46:59.273+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:46:59.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:59.668+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:59.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:59.668+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:59.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:59.668+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:46:59.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:59.669+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:00.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:00.033+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:00.034397+0000) 2024-04-27T03:47:00.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:46:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:46:59.922+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:46:59.923528+0000) 2024-04-27T03:47:00.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:00.251+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:00.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:00.251+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:00.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:00.251+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:00.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:00.252+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:00.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:00.239+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:00.240287+0000) 2024-04-27T03:47:00.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:00.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:00.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:00.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:00.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:00.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:00.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:00.358+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:00.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:00.621+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:00.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:00.621+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:00.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:00.621+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:00.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:00.622+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:01.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:01.035+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:01.036667+0000) 2024-04-27T03:47:01.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:00.926+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:00.927788+0000) 2024-04-27T03:47:01.623 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:01.241+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:01.242560+0000) 2024-04-27T03:47:01.624 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:01.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:01.624 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:01.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:01.624 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:01.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:01.624 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:01.383+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:01.624 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:01.254+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:01.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:01.254+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:01.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:01.254+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:01.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:01.255+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:01.937 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:01.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:01.938 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:01.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:01.938 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:01.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:01.938 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:01.617+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:02.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:02.121+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:02.122474+0000) 2024-04-27T03:47:02.227 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:47:02.125+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72066 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:47:02.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:01.931+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:01.931911+0000) 2024-04-27T03:47:02.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:02.122+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:02.123591+0000) 2024-04-27T03:47:02.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:02.039+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:02.039903+0000) 2024-04-27T03:47:02.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:02.121+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:02.122597+0000) 2024-04-27T03:47:02.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:02.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:02.654 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:02.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:02.654 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:02.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:02.654 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:02.251+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:02.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:02.244+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:02.245114+0000) 2024-04-27T03:47:02.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:02.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:02.655 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:02.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:02.655 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:02.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:02.655 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:02.397+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:02.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:02.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:02.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:02.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:02.941 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:02.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:02.941 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:02.647+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:03.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:02.933+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:02.934440+0000) 2024-04-27T03:47:03.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:03.040+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:03.041713+0000) 2024-04-27T03:47:03.632 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:03.265+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:03.632 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:03.265+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:03.632 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:03.265+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:03.632 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:03.266+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:03.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:03.246+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:03.247370+0000) 2024-04-27T03:47:03.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:03.385+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:03.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:03.385+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:03.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:03.385+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:03.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:03.385+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:03.914 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:03.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:03.914 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:03.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:03.914 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:03.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:03.914 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:03.626+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:04.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:04.045+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:04.045978+0000) 2024-04-27T03:47:04.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:03.936+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:03.937757+0000) 2024-04-27T03:47:04.452 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:04.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:04.452 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:04.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:04.452 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:04.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:04.452 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:04.227+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:04.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:04.250+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:04.251484+0000) 2024-04-27T03:47:04.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:04.398+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:04.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:04.398+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:04.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:04.398+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:04.453 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:04.399+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:04.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:04.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:04.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:04.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:04.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:04.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:04.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:04.676+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:05.012 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:04.939+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:04.939934+0000) 2024-04-27T03:47:05.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:05.253+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:05.254388+0000) 2024-04-27T03:47:05.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:05.047+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:05.048648+0000) 2024-04-27T03:47:05.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:05.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:05.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:05.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:05.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:05.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:05.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:05.239+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:05.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:05.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:05.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:05.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:05.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:05.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:05.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:05.430+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:05.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:05.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:05.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:05.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:05.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:05.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:05.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:05.668+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:06.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:05.941+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:05.942694+0000) 2024-04-27T03:47:06.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:06.051+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:06.052045+0000) 2024-04-27T03:47:06.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:06.191+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:06.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:06.191+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:06.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:06.191+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:06.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:06.191+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:06.631 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:06.255+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:06.256743+0000) 2024-04-27T03:47:06.631 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:06.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:06.631 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:06.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:06.632 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:06.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:06.632 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:06.384+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:06.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:06.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:06.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:06.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:06.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:06.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:06.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:06.625+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:07.226 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:47:07.134+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72106 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:47:07.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:07.129+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:07.130559+0000) 2024-04-27T03:47:07.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:06.945+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:06.946020+0000) 2024-04-27T03:47:07.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:07.130+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:07.131245+0000) 2024-04-27T03:47:07.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:07.053+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:07.054209+0000) 2024-04-27T03:47:07.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:07.130+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:07.131681+0000) 2024-04-27T03:47:07.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:07.142+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:07.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:07.142+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:07.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:07.142+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:07.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:07.143+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:07.525 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:07.258+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:07.259610+0000) 2024-04-27T03:47:07.525 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:07.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:07.525 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:07.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:07.526 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:07.392+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:07.526 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:07.393+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:07.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:07.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:07.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:07.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:07.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:07.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:07.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:07.635+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:08.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:07.947+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:07.948440+0000) 2024-04-27T03:47:08.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:08.055+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:08.056671+0000) 2024-04-27T03:47:08.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:08.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:08.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:08.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:08.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:08.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:08.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:08.147+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:08.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:08.261+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:08.262377+0000) 2024-04-27T03:47:08.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:08.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:08.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:08.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:08.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:08.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:08.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:08.389+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:08.957 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:08.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:08.957 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:08.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:08.957 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:08.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:08.957 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:08.639+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:09.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:08.950+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:08.951357+0000) 2024-04-27T03:47:09.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:09.059+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:09.059991+0000) 2024-04-27T03:47:09.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:09.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:09.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:09.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:09.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:09.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:09.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:09.145+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:09.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:09.264+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:09.265079+0000) 2024-04-27T03:47:09.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:09.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:09.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:09.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:09.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:09.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:09.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:09.420+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:09.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:09.600+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:09.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:09.600+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:09.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:09.600+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:09.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:09.601+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:10.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:09.954+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:09.955628+0000) 2024-04-27T03:47:10.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:10.061+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:10.062699+0000) 2024-04-27T03:47:10.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:10.157+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:10.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:10.157+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:10.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:10.157+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:10.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:10.159+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:10.596 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:10.267+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:10.268152+0000) 2024-04-27T03:47:10.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:10.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:10.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:10.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:10.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:10.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:10.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:10.459+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:10.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:10.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:10.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:10.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:10.965 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:10.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:10.965 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:10.590+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:11.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:10.957+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:10.958453+0000) 2024-04-27T03:47:11.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:11.065+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:11.066036+0000) 2024-04-27T03:47:11.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:11.120+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:11.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:11.120+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:11.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:11.120+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:11.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:11.121+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:11.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:11.269+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:11.270319+0000) 2024-04-27T03:47:11.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:11.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:11.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:11.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:11.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:11.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:11.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:11.475+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:11.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:11.562+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:11.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:11.562+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:11.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:11.562+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:11.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:11.564+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:12.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:12.139+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:12.140055+0000) 2024-04-27T03:47:12.227 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:47:12.143+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72146 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:47:12.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:11.960+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:11.961125+0000) 2024-04-27T03:47:12.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:12.140+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:12.141365+0000) 2024-04-27T03:47:12.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:12.067+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:12.068371+0000) 2024-04-27T03:47:12.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:12.072+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:12.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:12.072+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:12.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:12.072+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:12.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:12.072+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:12.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:12.138+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:12.139662+0000) 2024-04-27T03:47:12.523 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:12.272+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:12.272966+0000) 2024-04-27T03:47:12.523 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:12.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:12.524 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:12.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:12.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:12.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:12.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:12.517+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:12.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:12.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:12.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:12.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:12.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:12.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:12.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:12.557+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:13.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:12.962+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:12.963139+0000) 2024-04-27T03:47:13.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:13.070+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:13.071059+0000) 2024-04-27T03:47:13.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:13.106+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:13.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:13.106+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:13.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:13.106+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:13.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:13.107+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:13.534 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:13.276+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:13.277792+0000) 2024-04-27T03:47:13.535 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:13.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:13.535 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:13.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:13.535 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:13.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:13.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:13.528+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:13.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:13.592+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:13.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:13.592+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:13.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:13.592+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:13.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:13.593+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:14.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:13.966+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:13.967750+0000) 2024-04-27T03:47:14.184 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:14.072+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:14.073088+0000) 2024-04-27T03:47:14.184 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:14.099+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:14.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:14.099+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:14.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:14.099+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:14.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:14.100+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:14.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:14.281+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:14.282437+0000) 2024-04-27T03:47:14.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:14.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:14.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:14.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:14.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:14.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:14.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:14.593+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:14.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:14.548+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:14.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:14.548+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:14.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:14.548+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:14.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:14.549+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:15.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:14.971+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:14.971973+0000) 2024-04-27T03:47:15.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:15.076+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:15.077456+0000) 2024-04-27T03:47:15.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:15.087+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:15.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:15.087+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:15.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:15.087+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:15.294 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:15.087+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:15.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:15.285+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:15.286744+0000) 2024-04-27T03:47:15.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:15.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:15.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:15.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:15.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:15.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:15.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:15.549+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:15.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:15.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:15.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:15.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:15.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:15.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:15.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:15.585+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:16.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:15.973+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:15.974042+0000) 2024-04-27T03:47:16.296 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:16.060+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:16.296 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:16.060+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:16.296 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:16.060+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:16.296 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:16.061+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:16.297 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:16.078+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:16.079660+0000) 2024-04-27T03:47:16.596 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:16.289+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:16.289978+0000) 2024-04-27T03:47:16.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:16.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:16.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:16.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:16.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:16.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:16.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:16.591+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:16.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:16.593+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:16.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:16.593+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:16.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:16.593+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:16.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:16.594+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:17.277 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:47:17.152+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72185 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:47:17.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:17.147+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:17.147876+0000) 2024-04-27T03:47:17.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:16.977+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:16.978324+0000) 2024-04-27T03:47:17.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:17.147+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:17.148000+0000) 2024-04-27T03:47:17.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:17.081+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:17.082420+0000) 2024-04-27T03:47:17.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:17.110+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:17.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:17.110+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:17.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:17.110+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:17.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:17.110+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:17.279 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:17.147+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:17.148166+0000) 2024-04-27T03:47:17.554 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:17.291+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:17.292408+0000) 2024-04-27T03:47:17.555 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:17.548+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:17.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:17.548+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:17.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:17.548+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:17.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:17.548+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:17.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:17.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:17.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:17.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:17.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:17.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:17.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:17.629+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:18.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:17.981+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:17.982791+0000) 2024-04-27T03:47:18.300 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:18.083+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:18.084497+0000) 2024-04-27T03:47:18.301 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:18.107+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:18.301 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:18.107+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:18.301 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:18.107+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:18.301 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:18.108+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:18.555 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:18.293+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:18.294625+0000) 2024-04-27T03:47:18.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:18.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:18.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:18.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:18.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:18.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:18.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:18.658+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:18.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:18.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:18.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:18.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:18.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:18.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:18.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:18.549+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:19.243 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:18.984+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:18.984822+0000) 2024-04-27T03:47:19.243 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:19.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:19.243 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:19.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:19.243 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:19.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:19.243 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:19.085+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:19.244 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:19.086+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:19.086838+0000) 2024-04-27T03:47:19.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:19.296+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:19.297430+0000) 2024-04-27T03:47:19.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:19.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:20.017 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:19.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:20.017 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:19.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:20.017 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:19.676+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:20.017 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:19.565+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:20.017 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:19.565+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:20.018 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:19.565+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:20.018 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:19.565+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:20.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:19.988+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:19.989608+0000) 2024-04-27T03:47:20.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:20.089+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:20.090055+0000) 2024-04-27T03:47:20.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:20.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:20.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:20.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:20.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:20.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:20.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:20.126+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:20.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:20.299+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:20.300364+0000) 2024-04-27T03:47:20.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:20.531+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:20.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:20.531+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:20.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:20.531+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:20.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:20.532+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:20.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:20.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:20.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:20.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:20.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:20.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:20.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:20.712+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:21.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:20.990+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:20.991678+0000) 2024-04-27T03:47:21.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:21.092+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:21.093291+0000) 2024-04-27T03:47:21.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:21.138+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:21.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:21.138+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:21.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:21.138+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:21.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:21.139+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:21.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:21.303+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:21.304637+0000) 2024-04-27T03:47:21.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:21.485+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:21.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:21.485+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:21.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:21.485+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:21.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:21.486+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:21.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:21.703+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:21.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:21.703+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:21.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:21.703+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:21.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:21.704+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:22.285 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:47:22.160+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72225 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:47:22.286 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:22.156+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:22.157502+0000) 2024-04-27T03:47:22.286 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:21.995+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:21.996442+0000) 2024-04-27T03:47:22.286 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:22.157+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:22.157802+0000) 2024-04-27T03:47:22.286 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:22.094+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:22.095680+0000) 2024-04-27T03:47:22.287 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:22.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:22.287 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:22.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:22.287 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:22.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:22.287 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:22.099+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:22.287 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:22.158+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:22.158978+0000) 2024-04-27T03:47:22.688 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:22.306+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:22.307150+0000) 2024-04-27T03:47:22.689 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:22.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:22.689 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:22.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:22.689 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:22.443+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:22.689 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:22.443+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:22.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:22.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:22.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:22.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:22.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:22.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:22.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:22.683+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:23.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:22.998+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:22.999044+0000) 2024-04-27T03:47:23.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:23.098+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:23.099564+0000) 2024-04-27T03:47:23.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:23.115+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:23.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:23.116+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:23.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:23.116+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:23.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:23.116+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:23.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:23.310+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:23.311392+0000) 2024-04-27T03:47:23.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:23.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:23.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:23.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:23.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:23.422+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:23.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:23.422+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:23.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:23.668+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:23.979 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:23.668+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:23.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:23.668+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:23.980 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:23.669+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:24.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:24.000+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:24.001645+0000) 2024-04-27T03:47:24.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:24.101+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:24.101844+0000) 2024-04-27T03:47:24.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:24.133+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:24.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:24.133+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:24.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:24.133+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:24.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:24.134+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:24.680 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:24.312+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:24.313669+0000) 2024-04-27T03:47:24.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:24.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:24.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:24.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:24.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:24.402+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:24.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:24.402+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:24.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:24.674+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:24.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:24.674+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:24.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:24.674+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:24.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:24.675+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:25.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:25.003+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:25.003881+0000) 2024-04-27T03:47:25.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:25.105+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:25.106614+0000) 2024-04-27T03:47:25.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:25.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:25.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:25.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:25.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:25.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:25.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:25.162+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:25.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:25.316+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:25.317021+0000) 2024-04-27T03:47:25.662 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:25.439+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:25.662 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:25.439+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:25.662 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:25.439+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:25.662 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:25.440+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:25.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:25.655+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:25.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:25.655+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:25.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:25.655+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:25.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:25.656+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:26.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:26.005+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:26.006185+0000) 2024-04-27T03:47:26.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:26.110+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:26.111348+0000) 2024-04-27T03:47:26.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:26.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:26.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:26.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:26.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:26.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:26.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:26.176+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:26.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:26.318+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:26.319714+0000) 2024-04-27T03:47:26.615 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:26.468+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:26.615 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:26.468+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:26.615 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:26.468+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:26.615 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:26.469+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:26.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:26.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:26.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:26.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:26.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:26.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:26.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:26.608+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:27.295 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:47:27.169+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72264 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:47:27.296 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:27.007+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:27.008360+0000) 2024-04-27T03:47:27.296 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:27.165+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:27.166648+0000) 2024-04-27T03:47:27.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:27.165+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:27.165976+0000) 2024-04-27T03:47:27.296 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:27.114+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:27.115728+0000) 2024-04-27T03:47:27.297 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:27.165+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:27.165890+0000) 2024-04-27T03:47:27.297 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:27.209+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:27.297 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:27.209+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:27.297 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:27.209+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:27.297 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:27.209+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:27.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:27.320+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:27.321791+0000) 2024-04-27T03:47:27.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:27.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:27.604 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:27.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:27.605 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:27.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:27.605 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:27.431+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:27.605 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:27.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:27.605 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:27.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:27.605 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:27.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:27.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:27.598+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:28.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:28.010+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:28.011072+0000) 2024-04-27T03:47:28.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:28.116+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:28.117590+0000) 2024-04-27T03:47:28.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:28.194+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:28.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:28.195+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:28.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:28.195+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:28.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:28.195+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:28.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:28.323+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:28.323937+0000) 2024-04-27T03:47:28.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:28.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:28.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:28.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:28.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:28.405+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:28.598 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:28.405+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:28.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:28.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:28.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:28.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:28.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:28.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:28.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:28.591+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:29.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:29.014+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:29.014810+0000) 2024-04-27T03:47:29.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:29.119+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:29.119821+0000) 2024-04-27T03:47:29.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:29.209+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:29.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:29.209+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:29.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:29.209+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:29.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:29.210+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:29.566 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:29.325+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:29.326057+0000) 2024-04-27T03:47:29.567 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:29.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:29.567 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:29.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:29.567 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:29.449+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:29.567 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:29.450+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:29.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:29.559+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:29.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:29.559+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:29.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:29.559+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:29.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:29.560+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:30.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:30.016+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:30.016964+0000) 2024-04-27T03:47:30.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:30.121+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:30.122080+0000) 2024-04-27T03:47:30.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:30.236+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:30.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:30.236+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:30.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:30.236+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:30.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:30.237+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:30.594 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:30.327+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:30.328185+0000) 2024-04-27T03:47:30.594 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:30.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:30.595 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:30.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:30.595 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:30.459+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:30.595 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:30.460+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:30.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:30.587+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:30.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:30.587+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:30.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:30.587+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:30.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:30.588+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:31.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:31.018+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:31.019099+0000) 2024-04-27T03:47:31.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:31.123+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:31.124171+0000) 2024-04-27T03:47:31.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:31.230+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:31.306 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:31.230+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:31.306 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:31.230+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:31.306 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:31.231+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:31.582 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:31.329+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:31.330532+0000) 2024-04-27T03:47:31.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:31.502+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:31.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:31.502+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:31.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:31.502+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:31.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:31.502+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:31.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:31.576+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:31.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:31.576+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:31.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:31.576+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:31.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:31.577+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:32.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:32.175+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:32.176224+0000) 2024-04-27T03:47:32.281 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:47:32.177+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72304 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:47:32.281 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:32.023+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:32.024123+0000) 2024-04-27T03:47:32.281 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:32.175+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:32.176040+0000) 2024-04-27T03:47:32.282 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:32.125+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:32.126261+0000) 2024-04-27T03:47:32.282 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:32.173+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:32.174673+0000) 2024-04-27T03:47:32.282 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:32.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:32.282 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:32.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:32.282 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:32.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:32.532 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:32.274+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:32.533 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:32.331+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:32.332476+0000) 2024-04-27T03:47:32.533 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:32.525+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:32.533 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:32.525+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:32.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:32.525+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:32.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:32.526+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:32.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:32.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:32.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:32.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:32.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:32.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:32.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:32.598+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:33.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:33.027+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:33.028659+0000) 2024-04-27T03:47:33.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:33.127+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:33.128114+0000) 2024-04-27T03:47:33.571 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:33.333+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:33.334632+0000) 2024-04-27T03:47:33.572 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:33.301+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:33.572 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:33.301+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:33.572 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:33.301+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:33.572 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:33.302+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:33.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:33.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:33.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:33.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:33.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:33.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:33.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:33.566+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:33.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:33.565+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:33.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:33.565+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:33.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:33.565+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:33.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:33.565+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:34.186 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:34.129+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:34.130202+0000) 2024-04-27T03:47:34.186 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:34.029+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:34.030741+0000) 2024-04-27T03:47:34.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:34.335+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:34.336776+0000) 2024-04-27T03:47:34.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:34.319+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:34.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:34.319+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:34.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:34.319+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:34.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:34.320+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:34.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:34.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:34.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:34.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:34.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:34.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:34.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:34.576+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:34.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:34.599+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:34.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:34.599+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:34.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:34.599+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:34.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:34.599+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:35.305 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:35.032+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:35.032807+0000) 2024-04-27T03:47:35.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:35.131+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:35.132323+0000) 2024-04-27T03:47:35.556 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:35.340+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:35.341090+0000) 2024-04-27T03:47:35.557 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:35.349+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:35.557 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:35.349+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:35.557 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:35.349+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:35.557 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:35.350+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:35.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:35.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:35.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:35.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:35.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:35.550+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:35.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:35.550+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:35.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:35.630+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:35.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:35.630+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:35.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:35.630+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:35.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:35.631+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:36.305 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:36.034+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:36.034908+0000) 2024-04-27T03:47:36.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:36.134+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:36.135378+0000) 2024-04-27T03:47:36.624 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:36.343+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:36.344196+0000) 2024-04-27T03:47:36.624 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:36.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:36.624 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:36.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:36.625 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:36.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:36.625 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:36.548+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:36.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:36.303+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:36.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:36.303+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:36.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:36.303+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:36.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:36.304+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:36.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:36.617+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:36.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:36.617+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:36.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:36.617+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:36.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:36.618+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:37.302 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:47:37.185+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72343 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:47:37.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:37.182+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:37.183530+0000) 2024-04-27T03:47:37.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:37.036+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:37.037014+0000) 2024-04-27T03:47:37.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:37.181+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:37.182629+0000) 2024-04-27T03:47:37.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:37.137+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:37.138393+0000) 2024-04-27T03:47:37.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:37.181+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:37.182679+0000) 2024-04-27T03:47:37.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:37.296+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:37.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:37.296+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:37.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:37.296+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:37.554 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:37.346+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:37.346993+0000) 2024-04-27T03:47:37.554 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:37.297+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:37.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:37.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:37.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:37.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:37.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:37.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:37.978 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:37.548+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:37.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:37.570+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:37.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:37.570+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:37.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:37.570+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:37.979 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:37.571+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:38.293 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:38.040+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:38.040817+0000) 2024-04-27T03:47:38.294 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:38.139+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:38.140345+0000) 2024-04-27T03:47:38.549 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:38.348+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:38.349111+0000) 2024-04-27T03:47:38.549 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:38.287+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:38.549 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:38.287+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:38.549 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:38.287+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:38.549 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:38.288+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:38.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:38.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:38.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:38.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:38.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:38.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:38.977 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:38.543+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:38.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:38.578+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:38.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:38.578+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:38.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:38.578+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:38.978 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:38.579+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:39.305 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:39.042+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:39.043016+0000) 2024-04-27T03:47:39.306 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:39.141+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:39.142574+0000) 2024-04-27T03:47:39.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:39.317+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:39.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:39.317+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:39.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:39.317+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:39.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:39.318+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:39.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:39.350+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:39.351364+0000) 2024-04-27T03:47:39.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:39.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:39.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:39.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:39.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:39.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:39.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:39.537+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:39.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:39.543+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:39.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:39.543+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:39.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:39.543+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:39.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:39.545+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:40.305 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:40.044+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:40.045330+0000) 2024-04-27T03:47:40.306 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:40.144+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:40.144843+0000) 2024-04-27T03:47:40.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:40.515+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:40.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:40.515+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:40.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:40.515+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:40.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:40.516+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:40.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:40.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:40.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:40.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:40.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:40.347+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:40.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:40.347+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:40.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:40.353+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:40.354652+0000) 2024-04-27T03:47:40.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:40.519+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:40.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:40.519+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:40.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:40.519+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:40.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:40.519+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:41.306 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:41.046+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:41.047539+0000) 2024-04-27T03:47:41.306 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:41.148+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:41.149198+0000) 2024-04-27T03:47:41.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:41.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:41.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:41.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:41.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:41.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:41.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:41.477+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:41.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:41.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:41.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:41.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:41.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:41.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:41.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:41.374+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:41.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:41.356+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:41.356997+0000) 2024-04-27T03:47:41.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:41.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:41.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:41.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:41.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:41.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:41.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:41.524+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:42.306 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:47:42.195+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72383 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:47:42.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:42.191+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:42.192077+0000) 2024-04-27T03:47:42.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:42.049+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:42.049810+0000) 2024-04-27T03:47:42.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:42.194+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:42.195298+0000) 2024-04-27T03:47:42.307 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:42.152+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:42.153578+0000) 2024-04-27T03:47:42.307 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:42.191+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:42.192230+0000) 2024-04-27T03:47:42.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:42.367+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:42.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:42.367+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:42.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:42.367+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:42.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:42.368+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:42.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:42.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:42.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:42.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:42.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:42.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:42.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:42.483+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:42.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:42.359+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:42.359971+0000) 2024-04-27T03:47:42.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:42.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:42.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:42.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:42.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:42.494+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:42.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:42.495+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:43.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:43.154+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:43.156166+0000) 2024-04-27T03:47:43.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:43.052+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:43.054066+0000) 2024-04-27T03:47:43.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:43.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:43.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:43.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:43.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:43.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:43.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:43.455+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:43.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:43.337+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:43.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:43.337+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:43.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:43.337+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:43.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:43.337+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:43.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:43.361+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:43.363025+0000) 2024-04-27T03:47:43.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:43.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:43.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:43.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:43.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:43.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:43.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:43.452+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:44.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:44.056+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:44.058184+0000) 2024-04-27T03:47:44.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:44.157+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:44.159250+0000) 2024-04-27T03:47:44.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:44.290+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:44.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:44.290+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:44.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:44.290+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:44.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:44.291+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:44.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:44.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:44.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:44.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:44.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:44.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:44.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:44.446+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:44.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:44.363+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:44.365235+0000) 2024-04-27T03:47:44.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:44.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:44.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:44.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:44.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:44.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:44.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:44.431+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:45.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:45.059+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:45.061130+0000) 2024-04-27T03:47:45.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:45.160+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:45.162456+0000) 2024-04-27T03:47:45.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:45.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:45.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:45.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:45.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:45.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:45.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:45.271+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:45.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:45.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:45.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:45.473+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:45.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:45.366+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:45.368061+0000) 2024-04-27T03:47:45.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:45.475+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:45.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:45.476+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:45.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:45.476+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:45.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:45.476+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:46.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:46.061+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:46.063419+0000) 2024-04-27T03:47:46.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:46.163+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:46.165345+0000) 2024-04-27T03:47:46.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:46.265+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:46.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:46.265+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:46.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:46.265+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:46.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:46.267+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:46.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:46.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:46.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:46.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:46.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:46.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:46.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:46.461+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:46.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:46.368+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:46.370408+0000) 2024-04-27T03:47:46.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:46.429+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:46.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:46.429+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:46.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:46.429+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:46.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:46.429+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:47.323 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:47:47.202+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72422 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:47:47.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:47.063+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:47.065663+0000) 2024-04-27T03:47:47.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:47.199+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:47.201493+0000) 2024-04-27T03:47:47.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:47.199+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:47.201274+0000) 2024-04-27T03:47:47.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:47.166+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:47.168127+0000) 2024-04-27T03:47:47.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:47.198+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:47.200476+0000) 2024-04-27T03:47:47.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:47.266+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:47.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:47.266+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:47.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:47.266+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:47.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:47.268+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:47.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:47.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:47.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:47.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:47.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:47.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:47.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:47.459+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:47.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:47.371+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:47.373191+0000) 2024-04-27T03:47:47.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:47.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:47.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:47.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:47.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:47.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:47.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:47.407+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:48.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:48.066+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:48.067824+0000) 2024-04-27T03:47:48.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:48.169+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:48.170982+0000) 2024-04-27T03:47:48.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:48.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:48.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:48.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:48.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:48.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:48.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:48.248+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:48.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:48.412+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:48.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:48.412+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:48.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:48.412+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:48.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:48.413+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:48.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:48.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:48.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:48.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:48.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:48.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:48.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:48.360+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:48.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:48.374+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:48.375920+0000) 2024-04-27T03:47:49.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:49.070+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:49.072522+0000) 2024-04-27T03:47:49.333 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:49.172+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:49.174071+0000) 2024-04-27T03:47:49.333 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:49.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:49.333 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:49.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:49.334 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:49.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:49.334 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:49.274+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:49.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:49.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:49.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:49.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:49.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:49.325+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:49.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:49.325+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:49.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:49.377+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:49.379127+0000) 2024-04-27T03:47:49.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:49.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:49.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:49.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:49.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:49.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:49.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:49.404+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:50.379 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:50.074+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:50.076724+0000) 2024-04-27T03:47:50.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:50.371+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:50.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:50.371+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:50.380 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:50.314+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:50.380 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:50.314+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:50.380 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:50.314+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:50.380 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:50.315+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:50.381 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:50.174+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:50.176211+0000) 2024-04-27T03:47:50.381 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:50.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:50.381 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:50.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:50.381 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:50.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:50.381 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:50.258+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:50.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:50.379+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:50.381399+0000) 2024-04-27T03:47:50.737 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:50.371+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:50.737 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:50.372+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:51.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:51.078+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:51.080272+0000) 2024-04-27T03:47:51.339 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:51.178+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:51.180317+0000) 2024-04-27T03:47:51.339 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:51.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:51.339 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:51.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:51.339 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:51.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:51.339 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:51.305+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:51.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:51.333+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:51.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:51.333+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:51.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:51.333+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:51.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:51.334+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:51.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:51.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:51.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:51.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:51.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:51.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:51.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:51.331+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:51.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:51.382+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:51.384387+0000) 2024-04-27T03:47:52.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:52.206+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:52.208766+0000) 2024-04-27T03:47:52.341 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:47:52.212+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72462 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:47:52.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:52.080+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:52.082505+0000) 2024-04-27T03:47:52.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:52.206+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:52.208440+0000) 2024-04-27T03:47:52.342 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:52.182+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:52.184725+0000) 2024-04-27T03:47:52.342 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:52.207+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:52.209675+0000) 2024-04-27T03:47:52.342 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:52.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:52.342 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:52.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:52.342 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:52.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:52.342 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:52.274+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:52.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:52.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:52.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:52.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:52.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:52.361+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:52.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:52.361+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:52.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:52.384+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:52.386361+0000) 2024-04-27T03:47:52.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:52.333+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:52.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:52.333+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:52.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:52.333+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:52.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:52.334+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:53.364 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:53.083+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:53.085045+0000) 2024-04-27T03:47:53.365 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:53.295+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:53.365 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:53.295+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:53.365 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:53.295+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:53.365 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:53.296+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:53.365 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:53.186+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:53.188671+0000) 2024-04-27T03:47:53.365 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:53.260+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:53.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:53.260+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:53.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:53.260+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:53.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:53.260+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:53.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:53.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:53.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:53.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:53.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:53.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:53.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:53.357+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:53.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:53.387+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:53.389182+0000) 2024-04-27T03:47:54.363 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:54.086+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:54.087437+0000) 2024-04-27T03:47:54.363 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:54.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:54.363 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:54.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:54.363 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:54.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:54.364 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:54.282+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:54.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:54.189+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:54.190912+0000) 2024-04-27T03:47:54.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:54.235+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:54.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:54.235+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:54.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:54.235+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:54.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:54.236+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:54.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:54.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:54.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:54.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:54.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:54.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:54.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:54.356+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:54.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:54.390+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:54.391288+0000) 2024-04-27T03:47:55.398 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:55.089+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:55.090072+0000) 2024-04-27T03:47:55.398 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:55.302+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:55.399 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:55.302+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:55.399 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:55.302+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:55.399 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:55.303+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:55.399 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:55.189+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:55.399 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:55.189+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:55.399 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:55.189+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:55.400 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:55.190+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:55.400 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:55.192+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:55.193189+0000) 2024-04-27T03:47:55.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:55.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:55.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:55.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:55.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:55.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:55.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:55.391+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:55.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:55.393+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:55.393824+0000) 2024-04-27T03:47:56.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:56.093+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:56.094278+0000) 2024-04-27T03:47:56.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:56.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:56.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:56.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:56.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:56.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:56.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:56.340+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:56.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:56.194+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:56.195367+0000) 2024-04-27T03:47:56.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:56.197+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:56.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:56.197+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:56.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:56.197+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:56.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:56.198+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:56.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:56.396+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:56.397068+0000) 2024-04-27T03:47:56.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:56.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:56.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:56.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:56.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:56.413+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:56.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:56.413+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:57.308 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:57.218+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:57.218953+0000) 2024-04-27T03:47:57.308 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:47:57.222+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72502 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:47:57.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:57.097+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:57.098452+0000) 2024-04-27T03:47:57.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:57.219+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:57.220077+0000) 2024-04-27T03:47:57.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:57.151+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:57.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:57.151+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:57.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:57.151+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:57.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:57.152+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:57.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:57.196+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:57.197723+0000) 2024-04-27T03:47:57.310 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:57.218+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:57.219604+0000) 2024-04-27T03:47:57.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:57.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:57.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:57.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:57.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:57.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:57.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:57.325+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:57.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:57.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:57.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:57.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:57.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:57.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:57.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:57.396+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:57.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:57.398+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:57.399266+0000) 2024-04-27T03:47:58.407 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:58.099+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:58.100596+0000) 2024-04-27T03:47:58.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:58.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:58.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:58.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:58.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:58.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:58.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:58.337+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:58.408 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:58.172+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:58.408 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:58.172+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:58.408 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:58.172+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:58.409 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:58.173+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:58.409 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:58.199+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:58.200530+0000) 2024-04-27T03:47:58.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:58.400+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:58.401369+0000) 2024-04-27T03:47:58.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:58.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:58.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:58.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:58.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:58.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:58.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:58.415+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:59.410 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:59.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:59.410 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:59.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:59.410 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:59.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:47:59.410 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:59.201+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:59.411 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:47:59.202+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:59.203375+0000) 2024-04-27T03:47:59.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:59.102+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:59.102897+0000) 2024-04-27T03:47:59.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:59.302+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:59.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:59.302+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:59.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:59.302+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:47:59.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:47:59.304+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:47:59.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:59.403+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:47:59.404117+0000) 2024-04-27T03:47:59.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:59.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:59.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:59.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:59.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:59.444+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:47:59.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:47:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:47:59.445+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:00.413 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:00.196+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:00.413 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:00.196+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:00.413 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:00.196+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:00.413 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:00.197+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:00.413 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:00.204+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:00.205511+0000) 2024-04-27T03:48:00.414 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:00.104+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:00.104990+0000) 2024-04-27T03:48:00.414 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:00.287+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:00.414 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:00.287+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:00.415 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:00.287+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:00.415 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:00.288+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:00.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:00.406+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:00.406934+0000) 2024-04-27T03:48:00.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:00.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:00.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:00.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:00.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:00.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:00.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:00.410+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:01.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:01.197+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:01.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:01.197+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:01.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:01.197+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:01.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:01.198+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:01.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:01.206+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:01.207738+0000) 2024-04-27T03:48:01.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:01.106+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:01.107723+0000) 2024-04-27T03:48:01.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:01.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:01.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:01.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:01.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:01.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:01.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:01.332+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:01.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:01.408+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:01.409716+0000) 2024-04-27T03:48:01.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:01.447+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:01.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:01.447+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:01.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:01.447+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:01.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:01.447+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:02.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:02.227+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:02.228585+0000) 2024-04-27T03:48:02.370 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:48:02.232+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72543 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:48:02.371 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:02.110+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:02.111004+0000) 2024-04-27T03:48:02.371 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:02.227+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:02.227970+0000) 2024-04-27T03:48:02.371 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:02.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:02.371 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:02.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:02.371 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:02.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:02.372 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:02.206+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:02.372 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:02.206+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:02.372 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:02.206+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:02.372 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:02.207+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:02.372 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:02.209+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:02.210029+0000) 2024-04-27T03:48:02.372 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:02.227+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:02.227966+0000) 2024-04-27T03:48:02.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:02.364+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:02.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:02.411+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:02.412543+0000) 2024-04-27T03:48:02.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:02.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:02.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:02.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:02.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:02.441+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:02.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:02.441+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:03.420 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:03.114+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:03.114949+0000) 2024-04-27T03:48:03.420 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:03.211+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:03.212118+0000) 2024-04-27T03:48:03.420 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:03.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:03.420 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:03.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:03.421 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:03.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:03.421 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:03.217+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:03.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:03.413+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:03.414653+0000) 2024-04-27T03:48:03.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:03.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:03.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:03.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:03.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:03.419+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:03.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:03.420+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:03.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:03.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:03.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:03.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:03.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:03.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:03.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:03.414+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:04.405 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:04.216+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:04.216912+0000) 2024-04-27T03:48:04.405 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:04.265+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:04.405 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:04.265+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:04.405 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:04.265+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:04.405 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:04.267+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:04.406 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:04.118+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:04.119098+0000) 2024-04-27T03:48:04.406 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:04.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:04.406 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:04.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:04.406 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:04.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:04.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:04.399+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:04.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:04.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:04.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:04.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:04.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:04.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:04.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:04.399+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:04.672 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:04.416+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:04.416888+0000) 2024-04-27T03:48:05.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:05.121+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:05.121842+0000) 2024-04-27T03:48:05.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:05.218+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:05.219006+0000) 2024-04-27T03:48:05.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:05.269+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:05.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:05.269+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:05.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:05.269+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:05.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:05.270+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:05.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:05.408+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:05.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:05.408+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:05.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:05.408+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:05.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:05.409+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:05.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:05.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:05.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:05.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:05.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:05.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:05.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:05.389+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:05.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:05.418+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:05.419006+0000) 2024-04-27T03:48:06.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:06.123+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:06.124120+0000) 2024-04-27T03:48:06.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:06.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:06.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:06.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:06.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:06.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:06.375 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:06.354+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:06.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:06.220+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:06.221103+0000) 2024-04-27T03:48:06.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:06.304+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:06.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:06.304+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:06.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:06.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:06.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:06.306+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:06.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:06.421+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:06.422172+0000) 2024-04-27T03:48:06.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:06.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:06.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:06.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:06.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:06.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:06.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:06.386+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:07.373 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:48:07.242+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72582 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:48:07.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:07.241+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:07.241999+0000) 2024-04-27T03:48:07.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:07.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:07.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:07.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:07.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:07.353+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:07.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:07.353+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:07.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:07.127+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:07.128372+0000) 2024-04-27T03:48:07.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:07.238+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:07.239448+0000) 2024-04-27T03:48:07.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:07.366+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:07.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:07.366+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:07.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:07.366+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:07.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:07.223+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:07.224218+0000) 2024-04-27T03:48:07.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:07.237+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:07.238738+0000) 2024-04-27T03:48:07.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:07.323+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:07.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:07.323+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:07.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:07.323+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:07.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:07.323+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:07.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:07.425+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:07.426559+0000) 2024-04-27T03:48:07.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:07.367+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:08.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:08.131+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:08.132313+0000) 2024-04-27T03:48:08.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:08.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:08.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:08.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:08.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:08.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:08.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:08.340+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:08.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:08.225+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:08.226217+0000) 2024-04-27T03:48:08.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:08.355+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:08.379 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:08.355+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:08.379 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:08.355+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:08.379 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:08.356+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:08.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:08.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:08.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:08.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:08.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:08.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:08.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:08.370+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:08.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:08.427+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:08.428633+0000) 2024-04-27T03:48:09.376 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:09.133+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:09.134385+0000) 2024-04-27T03:48:09.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:09.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:09.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:09.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:09.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:09.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:09.377 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:09.334+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:09.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:09.230+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:09.230999+0000) 2024-04-27T03:48:09.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:09.370+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:09.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:09.370+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:09.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:09.370+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:09.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:09.430+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:09.431396+0000) 2024-04-27T03:48:09.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:09.371+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:09.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:09.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:09.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:09.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:09.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:09.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:09.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:09.377+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:10.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:10.138+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:10.139533+0000) 2024-04-27T03:48:10.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:10.232+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:10.233111+0000) 2024-04-27T03:48:10.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:10.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:10.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:10.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:10.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:10.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:10.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:10.407+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:10.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:10.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:10.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:10.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:10.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:10.374+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:10.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:10.374+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:10.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:10.432+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:10.433501+0000) 2024-04-27T03:48:10.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:10.359+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:10.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:10.359+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:10.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:10.359+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:10.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:10.360+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:11.406 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:11.143+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:11.144584+0000) 2024-04-27T03:48:11.407 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:11.234+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:11.235558+0000) 2024-04-27T03:48:11.407 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:11.320+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:11.407 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:11.320+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:11.407 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:11.320+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:11.407 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:11.321+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:11.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:11.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:11.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:11.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:11.427+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:11.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:11.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:11.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:11.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:11.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:11.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:11.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:11.400+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:11.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:11.435+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:11.435849+0000) 2024-04-27T03:48:12.375 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:48:12.250+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72622 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:48:12.376 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:12.246+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:12.247649+0000) 2024-04-27T03:48:12.376 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:12.148+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:12.149613+0000) 2024-04-27T03:48:12.376 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:12.246+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:12.247392+0000) 2024-04-27T03:48:12.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:12.236+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:12.237714+0000) 2024-04-27T03:48:12.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:12.247+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:12.248177+0000) 2024-04-27T03:48:12.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:12.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:12.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:12.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:12.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:12.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:12.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:12.311+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:12.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:12.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:12.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:12.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:12.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:12.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:12.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:12.406+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:12.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:12.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:12.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:12.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:12.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:12.407+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:12.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:12.407+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:12.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:12.437+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:12.437960+0000) 2024-04-27T03:48:13.376 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:13.153+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:13.154231+0000) 2024-04-27T03:48:13.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:13.238+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:13.239759+0000) 2024-04-27T03:48:13.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:13.271+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:13.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:13.271+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:13.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:13.271+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:13.377 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:13.272+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:13.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:13.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:13.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:13.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:13.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:13.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:13.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:13.370+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:13.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:13.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:13.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:13.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:13.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:13.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:13.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:13.374+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:13.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:13.439+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:13.440106+0000) 2024-04-27T03:48:14.415 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:14.155+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:14.156465+0000) 2024-04-27T03:48:14.415 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:14.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:14.415 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:14.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:14.415 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:14.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:14.415 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:14.338+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:14.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:14.241+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:14.241840+0000) 2024-04-27T03:48:14.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:14.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:14.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:14.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:14.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:14.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:14.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:14.317+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:14.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:14.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:14.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:14.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:14.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:14.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:14.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:14.408+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:14.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:14.442+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:14.443157+0000) 2024-04-27T03:48:15.310 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:15.159+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:15.160446+0000) 2024-04-27T03:48:15.310 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:15.243+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:15.243917+0000) 2024-04-27T03:48:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:15.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:15.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:15.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:15.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:15.340+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:15.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:15.354+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:15.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:15.354+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:15.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:15.354+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:15.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:15.355+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:15.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:15.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:15.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:15.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:15.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:15.389+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:15.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:15.389+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:15.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:15.445+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:15.445818+0000) 2024-04-27T03:48:16.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:16.161+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:16.162750+0000) 2024-04-27T03:48:16.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:16.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:16.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:16.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:16.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:16.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:16.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:16.301+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:16.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:16.245+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:16.246075+0000) 2024-04-27T03:48:16.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:16.374+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:16.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:16.374+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:16.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:16.375+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:16.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:16.376+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:16.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:16.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:16.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:16.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:16.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:16.382+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:16.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:16.382+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:16.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:16.448+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:16.449032+0000) 2024-04-27T03:48:17.383 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:48:17.259+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72661 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:48:17.383 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:17.256+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:17.256933+0000) 2024-04-27T03:48:17.384 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:17.164+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:17.165029+0000) 2024-04-27T03:48:17.384 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:17.256+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:17.256832+0000) 2024-04-27T03:48:17.384 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:17.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:17.384 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:17.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:17.384 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:17.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:17.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:17.313+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:17.385 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:17.247+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:17.248400+0000) 2024-04-27T03:48:17.385 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:17.254+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:17.255629+0000) 2024-04-27T03:48:17.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:17.391+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:17.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:17.391+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:17.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:17.391+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:17.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:17.391+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:17.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:17.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:17.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:17.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:17.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:17.380+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:17.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:17.380+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:17.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:17.450+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:17.451602+0000) 2024-04-27T03:48:18.391 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:18.166+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:18.167550+0000) 2024-04-27T03:48:18.391 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:18.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:18.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:18.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:18.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:18.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:18.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:18.283+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:18.392 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:18.249+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:18.250352+0000) 2024-04-27T03:48:18.392 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:18.360+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:18.392 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:18.360+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:18.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:18.360+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:18.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:18.362+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:18.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:18.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:18.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:18.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:18.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:18.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:18.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:18.385+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:18.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:18.452+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:18.453670+0000) 2024-04-27T03:48:19.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:19.168+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:19.169694+0000) 2024-04-27T03:48:19.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:19.248+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:19.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:19.248+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:19.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:19.248+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:19.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:19.249+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:19.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:19.251+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:19.252366+0000) 2024-04-27T03:48:19.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:19.349+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:19.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:19.349+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:19.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:19.349+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:19.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:19.350+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:19.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:19.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:19.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:19.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:19.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:19.428+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:19.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:19.428+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:19.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:19.454+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:19.455785+0000) 2024-04-27T03:48:20.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:20.253+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:20.254656+0000) 2024-04-27T03:48:20.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:20.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:20.405 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:20.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:20.405 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:20.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:20.405 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:20.172+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:20.172900+0000) 2024-04-27T03:48:20.405 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:20.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:20.405 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:20.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:20.405 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:20.211+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:20.406 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:20.211+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:20.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:20.398+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:20.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:20.459+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:20.460058+0000) 2024-04-27T03:48:20.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:20.463+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:20.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:20.463+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:20.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:20.463+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:20.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:20.464+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:21.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:21.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:21.558 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:21.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:21.558 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:21.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:21.558 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:21.165+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:21.558 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:21.175+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:21.176318+0000) 2024-04-27T03:48:21.558 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:21.256+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:21.256924+0000) 2024-04-27T03:48:21.559 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:21.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:21.559 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:21.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:21.559 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:21.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:21.559 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:21.364+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:21.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:21.463+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:21.464280+0000) 2024-04-27T03:48:21.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:21.485+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:21.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:21.485+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:21.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:21.485+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:21.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:21.486+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:22.373 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:48:22.267+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72701 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:48:22.373 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:22.265+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:22.266087+0000) 2024-04-27T03:48:22.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:22.179+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:22.180619+0000) 2024-04-27T03:48:22.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:22.181+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:22.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:22.181+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:22.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:22.181+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:22.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:22.182+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:22.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:22.264+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:22.265326+0000) 2024-04-27T03:48:22.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:22.258+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:22.259175+0000) 2024-04-27T03:48:22.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:22.264+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:22.265509+0000) 2024-04-27T03:48:22.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:22.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:22.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:22.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:22.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:22.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:22.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:22.367+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:22.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:22.467+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:22.468137+0000) 2024-04-27T03:48:22.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:22.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:22.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:22.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:22.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:22.523+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:22.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:22.524+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:23.410 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:23.261+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:23.261826+0000) 2024-04-27T03:48:23.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:23.182+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:23.183427+0000) 2024-04-27T03:48:23.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:23.188+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:23.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:23.188+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:23.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:23.188+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:23.411 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:23.189+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:23.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:23.471+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:23.472391+0000) 2024-04-27T03:48:23.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:23.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:23.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:23.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:23.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:23.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:23.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:23.538+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:23.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:23.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:23.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:23.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:23.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:23.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:23.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:23.404+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:24.405 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:24.265+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:24.266163+0000) 2024-04-27T03:48:24.406 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:24.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:24.406 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:24.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:24.406 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:24.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:24.406 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:24.177+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:24.406 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:24.184+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:24.185569+0000) 2024-04-27T03:48:24.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:24.473+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:24.474647+0000) 2024-04-27T03:48:24.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:24.503+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:24.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:24.503+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:24.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:24.503+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:24.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:24.504+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:24.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:24.399+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:24.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:24.399+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:24.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:24.399+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:24.729 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:24.400+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:25.402 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:25.268+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:25.269210+0000) 2024-04-27T03:48:25.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:25.187+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:25.187820+0000) 2024-04-27T03:48:25.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:25.215+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:25.402 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:25.215+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:25.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:25.215+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:25.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:25.216+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:25.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:25.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:25.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:25.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:25.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:25.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:25.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:25.396+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:25.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:25.478+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:25.479442+0000) 2024-04-27T03:48:25.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:25.552+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:25.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:25.552+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:25.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:25.552+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:25.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:25.553+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:26.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:26.271+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:26.272228+0000) 2024-04-27T03:48:26.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:26.189+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:26.190085+0000) 2024-04-27T03:48:26.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:26.251+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:26.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:26.251+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:26.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:26.251+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:26.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:26.252+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:26.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:26.482+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:26.483701+0000) 2024-04-27T03:48:26.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:26.507+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:26.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:26.507+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:26.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:26.507+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:26.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:26.508+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:26.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:26.422+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:26.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:26.422+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:26.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:26.422+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:26.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:26.423+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:27.402 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:48:27.276+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72740 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:48:27.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:27.272+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:27.273141+0000) 2024-04-27T03:48:27.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:27.274+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:27.275216+0000) 2024-04-27T03:48:27.403 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:27.271+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:27.272770+0000) 2024-04-27T03:48:27.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:27.191+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:27.192204+0000) 2024-04-27T03:48:27.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:27.249+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:27.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:27.249+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:27.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:27.249+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:27.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:27.250+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:27.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:27.272+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:27.273540+0000) 2024-04-27T03:48:27.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:27.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:27.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:27.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:27.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:27.461+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:27.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:27.462+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:27.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:27.484+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:27.485753+0000) 2024-04-27T03:48:27.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:27.515+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:27.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:27.515+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:27.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:27.515+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:27.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:27.516+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:28.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:28.278+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:28.279591+0000) 2024-04-27T03:48:28.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:28.195+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:28.196188+0000) 2024-04-27T03:48:28.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:28.283+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:28.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:28.283+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:28.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:28.283+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:28.444 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:28.284+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:28.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:28.487+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:28.488445+0000) 2024-04-27T03:48:28.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:28.492+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:28.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:28.492+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:28.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:28.492+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:28.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:28.492+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:28.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:28.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:28.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:28.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:28.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:28.436+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:28.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:28.438+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:29.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:29.283+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:29.283892+0000) 2024-04-27T03:48:29.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:29.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:29.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:29.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:29.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:29.437+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:29.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:29.197+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:29.198327+0000) 2024-04-27T03:48:29.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:29.251+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:29.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:29.251+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:29.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:29.251+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:29.446 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:29.252+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:29.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:29.438+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:29.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:29.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:29.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:29.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:29.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:29.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:29.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:29.446+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:29.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:29.489+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:29.490655+0000) 2024-04-27T03:48:30.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:30.285+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:30.286213+0000) 2024-04-27T03:48:30.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:30.199+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:30.200467+0000) 2024-04-27T03:48:30.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:30.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:30.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:30.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:30.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:30.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:30.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:30.254+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:30.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:30.479+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:30.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:30.479+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:30.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:30.479+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:30.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:30.480+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:30.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:30.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:30.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:30.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:30.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:30.424+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:30.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:30.425+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:30.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:30.492+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:30.492918+0000) 2024-04-27T03:48:31.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:31.287+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:31.288424+0000) 2024-04-27T03:48:31.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:31.202+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:31.203430+0000) 2024-04-27T03:48:31.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:31.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:31.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:31.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:31.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:31.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:31.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:31.254+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:31.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:31.496+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:31.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:31.496+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:31.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:31.496+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:31.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:31.497+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:31.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:31.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:31.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:31.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:31.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:31.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:31.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:31.430+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:31.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:31.494+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:31.495013+0000) 2024-04-27T03:48:32.414 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:48:32.287+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72780 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:48:32.414 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:32.280+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:32.281766+0000) 2024-04-27T03:48:32.415 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:32.205+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:32.206446+0000) 2024-04-27T03:48:32.415 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:32.282+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:32.283155+0000) 2024-04-27T03:48:32.415 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:32.285+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:32.415 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:32.285+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:32.415 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:32.285+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:32.415 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:32.286+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:32.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:32.280+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:32.281663+0000) 2024-04-27T03:48:32.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:32.289+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:32.290441+0000) 2024-04-27T03:48:32.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:32.459+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:32.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:32.459+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:32.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:32.460+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:32.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:32.460+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:32.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:32.411+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:32.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:32.411+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:32.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:32.411+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:32.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:32.412+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:32.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:32.497+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:32.498354+0000) 2024-04-27T03:48:33.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:33.293+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:33.293831+0000) 2024-04-27T03:48:33.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:33.207+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:33.208534+0000) 2024-04-27T03:48:33.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:33.286+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:33.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:33.286+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:33.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:33.286+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:33.379 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:33.287+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:33.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:33.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:33.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:33.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:33.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:33.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:33.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:33.499+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:33.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:33.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:33.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:33.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:33.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:33.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:33.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:33.371+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:33.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:33.500+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:33.501349+0000) 2024-04-27T03:48:34.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:34.295+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:34.296012+0000) 2024-04-27T03:48:34.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:34.209+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:34.210679+0000) 2024-04-27T03:48:34.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:34.307+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:34.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:34.307+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:34.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:34.307+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:34.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:34.308+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:34.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:34.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:34.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:34.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:34.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:34.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:34.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:34.330+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:34.859 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:34.504+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:34.505458+0000) 2024-04-27T03:48:34.859 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:34.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:34.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:34.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:34.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:34.490+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:34.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:34.491+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:35.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:35.299+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:35.300786+0000) 2024-04-27T03:48:35.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:35.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:35.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:35.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:35.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:35.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:35.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:35.342+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:35.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:35.214+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:35.215234+0000) 2024-04-27T03:48:35.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:35.284+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:35.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:35.284+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:35.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:35.284+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:35.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:35.285+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:35.859 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:35.509+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:35.510169+0000) 2024-04-27T03:48:35.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:35.504+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:35.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:35.504+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:35.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:35.504+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:35.860 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:35.505+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:36.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:36.302+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:36.303430+0000) 2024-04-27T03:48:36.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:36.219+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:36.220012+0000) 2024-04-27T03:48:36.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:36.320+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:36.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:36.320+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:36.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:36.320+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:36.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:36.321+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:36.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:36.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:36.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:36.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:36.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:36.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:36.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:36.499+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:36.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:36.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:36.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:36.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:36.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:36.376+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:36.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:36.377+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:36.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:36.511+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:36.512412+0000) 2024-04-27T03:48:37.423 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:48:37.295+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72820 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:48:37.424 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:37.291+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:37.291966+0000) 2024-04-27T03:48:37.424 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:37.292+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:37.292877+0000) 2024-04-27T03:48:37.424 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:37.304+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:37.305492+0000) 2024-04-27T03:48:37.424 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:37.221+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:37.222291+0000) 2024-04-27T03:48:37.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:37.291+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:37.292114+0000) 2024-04-27T03:48:37.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:37.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:37.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:37.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:37.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:37.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:37.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:37.302+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:37.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:37.421+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:37.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:37.421+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:37.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:37.421+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:37.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:37.421+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:37.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:37.514+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:37.515473+0000) 2024-04-27T03:48:37.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:37.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:37.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:37.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:37.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:37.506+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:37.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:37.507+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:38.422 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:38.225+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:38.226353+0000) 2024-04-27T03:48:38.422 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:38.257+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:38.422 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:38.257+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:38.422 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:38.257+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:38.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:38.258+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:38.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:38.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:38.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:38.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:38.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:38.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:38.423 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:38.309+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:38.309812+0000) 2024-04-27T03:48:38.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:38.415+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:38.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:38.516+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:38.517565+0000) 2024-04-27T03:48:38.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:38.517+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:38.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:38.517+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:38.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:38.517+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:38.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:38.519+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:39.380 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:39.311+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:39.312493+0000) 2024-04-27T03:48:39.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:39.227+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:39.228510+0000) 2024-04-27T03:48:39.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:39.250+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:39.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:39.250+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:39.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:39.250+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:39.381 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:39.251+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:39.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:39.529+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:39.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:39.529+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:39.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:39.529+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:39.685 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:39.530+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:39.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:39.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:39.685 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:39.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:39.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:39.373+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:39.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:39.374+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:39.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:39.520+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:39.520809+0000) 2024-04-27T03:48:40.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:40.313+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:40.314753+0000) 2024-04-27T03:48:40.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:40.230+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:40.231300+0000) 2024-04-27T03:48:40.405 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:40.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:40.405 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:40.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:40.405 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:40.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:40.405 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:40.234+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:40.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:40.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:40.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:40.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:40.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:40.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:40.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:40.398+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:40.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:40.522+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:40.523502+0000) 2024-04-27T03:48:40.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:40.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:40.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:40.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:40.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:40.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:40.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:40.482+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:41.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:41.316+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:41.316931+0000) 2024-04-27T03:48:41.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:41.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:41.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:41.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:41.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:41.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:41.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:41.364+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:41.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:41.232+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:41.233411+0000) 2024-04-27T03:48:41.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:41.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:41.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:41.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:41.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:41.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:41.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:41.277+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:41.867 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:41.524+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:41.525733+0000) 2024-04-27T03:48:41.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:41.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:41.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:41.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:41.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:41.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:41.868 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:41.522+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:42.378 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:48:42.304+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72860 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:48:42.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:42.301+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:42.301874+0000) 2024-04-27T03:48:42.379 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:42.318+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:42.319491+0000) 2024-04-27T03:48:42.379 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:42.234+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:42.235517+0000) 2024-04-27T03:48:42.379 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:42.290+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:42.379 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:42.290+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:42.379 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:42.290+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:42.379 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:42.290+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:42.380 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:42.300+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:42.301267+0000) 2024-04-27T03:48:42.380 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:42.301+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:42.302204+0000) 2024-04-27T03:48:42.380 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:42.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:42.380 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:42.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:42.380 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:42.371+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:42.380 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:42.371+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:42.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:42.529+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:42.529841+0000) 2024-04-27T03:48:42.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:42.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:42.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:42.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:42.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:42.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:42.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:42.509+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:43.423 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:43.321+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:43.321983+0000) 2024-04-27T03:48:43.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:43.237+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:43.238376+0000) 2024-04-27T03:48:43.424 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:43.317+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:43.424 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:43.317+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:43.424 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:43.317+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:43.424 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:43.318+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:43.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:43.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:43.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:43.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:43.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:43.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:43.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:43.525+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:43.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:43.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:43.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:43.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:43.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:43.416+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:43.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:43.416+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:43.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:43.531+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:43.532525+0000) 2024-04-27T03:48:44.409 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:44.324+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:44.326325+0000) 2024-04-27T03:48:44.409 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:44.238+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:44.240521+0000) 2024-04-27T03:48:44.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:44.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:44.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:44.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:44.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:44.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:44.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:44.335+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:44.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:44.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:44.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:44.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:44.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:44.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:44.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:44.501+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:44.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:44.401+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:44.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:44.401+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:44.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:44.401+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:44.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:44.401+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:44.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:44.532+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:44.534673+0000) 2024-04-27T03:48:45.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:45.241+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:45.242797+0000) 2024-04-27T03:48:45.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:45.328+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:45.330688+0000) 2024-04-27T03:48:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:45.380+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:45.380+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:45.380+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:45.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:45.381+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:45.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:45.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:45.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:45.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:45.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:45.537+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:45.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:45.537+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:45.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:45.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:45.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:45.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:45.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:45.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:45.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:45.393+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:45.729 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:45.535+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:45.536818+0000) 2024-04-27T03:48:46.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:46.333+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:46.334989+0000) 2024-04-27T03:48:46.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:46.246+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:46.247887+0000) 2024-04-27T03:48:46.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:46.439+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:46.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:46.439+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:46.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:46.439+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:46.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:46.439+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:46.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:46.540+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:46.541825+0000) 2024-04-27T03:48:46.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:46.570+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:46.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:46.570+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:46.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:46.570+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:46.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:46.572+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:46.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:46.409+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:46.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:46.409+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:46.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:46.409+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:46.729 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:46.410+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:47.433 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:48:47.313+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72900 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:48:47.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:47.309+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:47.311152+0000) 2024-04-27T03:48:47.434 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:47.308+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:47.310438+0000) 2024-04-27T03:48:47.434 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:47.337+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:47.339097+0000) 2024-04-27T03:48:47.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:47.251+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:47.253040+0000) 2024-04-27T03:48:47.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:47.309+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:47.310909+0000) 2024-04-27T03:48:47.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:47.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:47.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:47.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:47.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:47.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:47.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:47.365+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:47.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:47.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:47.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:47.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:47.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:47.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:47.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:47.425+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:47.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:47.544+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:47.546736+0000) 2024-04-27T03:48:47.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:47.528+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:47.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:47.528+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:47.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:47.528+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:47.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:47.529+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:48.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:48.341+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:48.343415+0000) 2024-04-27T03:48:48.461 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:48.256+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:48.257971+0000) 2024-04-27T03:48:48.461 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:48.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:48.461 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:48.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:48.461 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:48.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:48.461 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:48.323+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:48.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:48.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:48.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:48.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:48.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:48.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:48.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:48.510+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:48.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:48.452+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:48.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:48.452+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:48.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:48.452+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:48.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:48.453+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:48.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:48.547+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:48.548818+0000) 2024-04-27T03:48:49.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:49.343+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:49.345520+0000) 2024-04-27T03:48:49.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:49.260+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:49.262212+0000) 2024-04-27T03:48:49.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:49.295+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:49.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:49.295+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:49.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:49.295+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:49.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:49.296+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:49.877 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:49.479+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:49.877 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:49.479+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:49.878 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:49.479+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:49.878 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:49.480+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:49.878 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:49.549+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:49.550955+0000) 2024-04-27T03:48:49.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:49.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:49.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:49.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:49.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:49.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:49.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:49.548+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:50.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:50.345+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:50.347664+0000) 2024-04-27T03:48:50.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:50.261+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:50.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:50.261+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:50.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:50.261+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:50.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:50.262+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:50.264306+0000) 2024-04-27T03:48:50.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:50.262+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:50.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:50.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:50.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:50.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:50.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:50.526+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:50.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:50.526+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:50.844 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:50.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:50.845 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:50.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:50.845 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:50.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:50.845 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:50.529+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:50.845 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:50.551+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:50.553292+0000) 2024-04-27T03:48:50.846 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-27T03:48:50.847 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-04-27T03:48:50.847 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-27T03:48:50.847 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-27T03:48:50.866 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-27T03:48:50.866 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-27T03:48:50.923 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T03:48:51.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:51.348+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:51.349820+0000) 2024-04-27T03:48:51.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:51.265+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:51.267245+0000) 2024-04-27T03:48:51.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:51.295+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:51.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:51.295+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:51.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:51.295+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:51.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:51.296+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:51.877 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:51.541+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:51.878 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:51.541+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:51.878 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:51.541+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:51.878 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:51.541+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:51.878 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:51.554+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:51.556023+0000) 2024-04-27T03:48:51.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:51.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:51.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:51.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:51.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:51.508+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:51.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:51.509+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:52.447 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:48:52.322+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72940 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:48:52.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:52.317+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:52.318934+0000) 2024-04-27T03:48:52.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:52.350+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:52.351981+0000) 2024-04-27T03:48:52.448 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:52.318+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:52.320136+0000) 2024-04-27T03:48:52.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:52.267+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:52.269354+0000) 2024-04-27T03:48:52.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:52.318+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:52.319933+0000) 2024-04-27T03:48:52.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:52.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:52.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:52.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:52.449 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:52.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:52.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:52.335+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:52.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:52.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:52.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:52.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:52.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:52.467+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:52.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:52.469+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:52.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:52.556+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:52.558115+0000) 2024-04-27T03:48:52.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:52.563+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:52.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:52.563+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:52.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:52.563+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:52.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:52.564+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:53.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:53.352+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:53.354072+0000) 2024-04-27T03:48:53.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:53.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:53.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:53.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:53.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:53.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:53.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:53.424+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:53.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:53.269+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:53.271426+0000) 2024-04-27T03:48:53.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:53.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:53.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:53.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:53.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:53.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:53.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:53.313+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:53.880 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:53.558+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:53.560344+0000) 2024-04-27T03:48:53.880 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:53.612+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:53.880 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:53.612+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:53.881 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:53.612+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:53.881 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:53.612+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:54.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:54.356+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:54.358415+0000) 2024-04-27T03:48:54.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:54.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:54.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:54.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:54.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:54.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:54.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:54.421+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:54.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:54.271+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:54.273664+0000) 2024-04-27T03:48:54.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:54.344+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:54.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:54.344+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:54.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:54.344+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:54.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:54.345+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:54.882 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:54.562+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:54.564563+0000) 2024-04-27T03:48:54.883 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:54.626+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:54.883 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:54.626+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:54.883 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:54.626+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:54.883 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:54.627+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:55.442 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:55.359+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:55.361360+0000) 2024-04-27T03:48:55.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:55.275+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:55.275868+0000) 2024-04-27T03:48:55.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:55.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:55.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:55.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:55.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:55.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:55.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:55.304+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:55.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:55.565+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:55.566882+0000) 2024-04-27T03:48:55.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:55.624+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:55.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:55.624+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:55.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:55.624+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:55.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:55.624+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:55.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:55.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:55.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:55.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:55.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:55.434+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:55.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:55.435+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:56.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:56.277+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:56.278106+0000) 2024-04-27T03:48:56.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:56.305+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:56.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:56.305+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:56.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:56.305+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:56.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:56.306+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:56.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:56.363+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:56.364590+0000) 2024-04-27T03:48:56.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:56.426+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:56.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:56.426+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:56.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:56.426+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:56.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:56.427+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:56.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:56.568+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:56.569137+0000) 2024-04-27T03:48:56.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:56.646+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:56.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:56.646+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:56.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:56.646+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:56.885 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:56.647+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:57.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:48:57.332+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 72981 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:48:57.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:57.280+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:57.280827+0000) 2024-04-27T03:48:57.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:57.324+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:57.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:57.324+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:57.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:57.324+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:57.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:57.325+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:57.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:57.329+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:57.330272+0000) 2024-04-27T03:48:57.460 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:57.327+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:57.328288+0000) 2024-04-27T03:48:57.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:57.327+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:57.328286+0000) 2024-04-27T03:48:57.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:57.366+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:57.367505+0000) 2024-04-27T03:48:57.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:57.391+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:57.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:57.391+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:57.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:57.391+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:57.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:57.392+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:57.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:57.571+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:57.572085+0000) 2024-04-27T03:48:57.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:57.677+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:57.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:57.677+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:57.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:57.677+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:57.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:57.678+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:58.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:58.284+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:58.284958+0000) 2024-04-27T03:48:58.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:58.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:58.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:58.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:58.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:58.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:58.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:58.374+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:58.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:58.353+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:58.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:58.353+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:58.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:58.353+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:58.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:58.353+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:58.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:58.371+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:58.372599+0000) 2024-04-27T03:48:58.886 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:58.573+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:58.574238+0000) 2024-04-27T03:48:58.886 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:58.659+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:58.886 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:58.659+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:58.886 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:58.659+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:58.887 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:58.659+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:59.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:59.286+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:59.286991+0000) 2024-04-27T03:48:59.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:59.370+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:59.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:59.370+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:59.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:59.370+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:48:59.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:48:59.371+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:59.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:59.361+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:59.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:59.361+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:59.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:59.361+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:48:59.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:59.362+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:48:59.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:48:59.374+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:59.375162+0000) 2024-04-27T03:48:59.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:59.576+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:48:59.577190+0000) 2024-04-27T03:48:59.889 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:59.667+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:59.889 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:59.667+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:59.889 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:59.667+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:48:59.889 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:48:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:48:59.667+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:00.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:00.288+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:00.289259+0000) 2024-04-27T03:49:00.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:00.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:00.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:00.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:00.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:00.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:00.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:00.342+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:00.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:00.378+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:00.379578+0000) 2024-04-27T03:49:00.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:00.390+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:00.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:00.390+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:00.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:00.390+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:00.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:00.391+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:00.889 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:00.578+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:00.579439+0000) 2024-04-27T03:49:00.889 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:00.705+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:00.889 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:00.705+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:00.889 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:00.705+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:00.890 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:00.705+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:01.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:01.381+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:01.381840+0000) 2024-04-27T03:49:01.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:01.413+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:01.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:01.413+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:01.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:01.413+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:01.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:01.414+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:01.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:01.292+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:01.293474+0000) 2024-04-27T03:49:01.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:01.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:01.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:01.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:01.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:01.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:01.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:01.359+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:01.889 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:01.580+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:01.581743+0000) 2024-04-27T03:49:01.889 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:01.676+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:01.889 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:01.676+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:01.890 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:01.676+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:01.890 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:01.677+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:02.468 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:49:02.341+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73021 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:49:02.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:02.336+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:02.337357+0000) 2024-04-27T03:49:02.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:02.337+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:02.338610+0000) 2024-04-27T03:49:02.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:02.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:02.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:02.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:02.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:02.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:02.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:02.367+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:02.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:02.383+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:02.384457+0000) 2024-04-27T03:49:02.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:02.294+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:02.295741+0000) 2024-04-27T03:49:02.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:02.318+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:02.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:02.318+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:02.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:02.318+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:02.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:02.319+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:02.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:02.337+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:02.338600+0000) 2024-04-27T03:49:02.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:02.585+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:02.585931+0000) 2024-04-27T03:49:03.026 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:02.721+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:03.026 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:02.721+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:03.026 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:02.721+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:03.027 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:02.721+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:03.596 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:03.297+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:03.298361+0000) 2024-04-27T03:49:03.596 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:03.305+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:03.596 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:03.305+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:03.596 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:03.305+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:03.597 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:03.307+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:03.597 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:03.386+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:03.387206+0000) 2024-04-27T03:49:03.597 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:03.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:03.597 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:03.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:03.597 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:03.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:03.597 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:03.407+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:03.892 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:03.589+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:03.590097+0000) 2024-04-27T03:49:03.892 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:03.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:03.893 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:03.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:03.893 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:03.762+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:03.893 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:03.762+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:04.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:04.299+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:04.300548+0000) 2024-04-27T03:49:04.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:04.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:04.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:04.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:04.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:04.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:04.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:04.355+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:04.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:04.388+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:04.389357+0000) 2024-04-27T03:49:04.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:04.406+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:04.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:04.406+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:04.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:04.406+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:04.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:04.406+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:04.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:04.592+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:04.592998+0000) 2024-04-27T03:49:04.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:04.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:04.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:04.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:04.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:04.751+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:04.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:04.752+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:05.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:05.301+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:05.302781+0000) 2024-04-27T03:49:05.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:05.372+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:05.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:05.372+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:05.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:05.372+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:05.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:05.374+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:05.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:05.381+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:05.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:05.381+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:05.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:05.381+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:05.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:05.381+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:05.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:05.392+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:05.393698+0000) 2024-04-27T03:49:05.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:05.595+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:05.596117+0000) 2024-04-27T03:49:05.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:05.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:05.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:05.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:05.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:05.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:05.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:05.730+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:06.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:06.304+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:06.304922+0000) 2024-04-27T03:49:06.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:06.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:06.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:06.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:06.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:06.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:06.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:06.359+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:06.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:06.355+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:06.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:06.355+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:06.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:06.355+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:06.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:06.356+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:06.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:06.395+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:06.395970+0000) 2024-04-27T03:49:06.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:06.597+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:06.598780+0000) 2024-04-27T03:49:06.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:06.683+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:06.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:06.683+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:06.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:06.683+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:06.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:06.684+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:07.477 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:49:07.352+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73061 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:49:07.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:07.345+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:07.346300+0000) 2024-04-27T03:49:07.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:07.308+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:07.309178+0000) 2024-04-27T03:49:07.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:07.343+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:07.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:07.343+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:07.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:07.343+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:07.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:07.345+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:07.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:07.347+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:07.348739+0000) 2024-04-27T03:49:07.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:07.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:07.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:07.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:07.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:07.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:07.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:07.345+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:07.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:07.347+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:07.348243+0000) 2024-04-27T03:49:07.480 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:07.399+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:07.400231+0000) 2024-04-27T03:49:07.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:07.600+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:07.600844+0000) 2024-04-27T03:49:07.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:07.671+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:07.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:07.671+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:07.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:07.671+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:07.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:07.671+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:08.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:08.312+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:08.313346+0000) 2024-04-27T03:49:08.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:08.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:08.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:08.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:08.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:08.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:08.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:08.378+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:08.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:08.315+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:08.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:08.315+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:08.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:08.315+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:08.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:08.316+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:08.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:08.401+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:08.402684+0000) 2024-04-27T03:49:08.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:08.602+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:08.602957+0000) 2024-04-27T03:49:08.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:08.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:08.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:08.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:08.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:08.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:08.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:08.713+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:09.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:09.317+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:09.317886+0000) 2024-04-27T03:49:09.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:09.381+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:09.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:09.381+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:09.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:09.381+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:09.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:09.381+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:09.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:09.302+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:09.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:09.302+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:09.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:09.302+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:09.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:09.303+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:09.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:09.405+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:09.405960+0000) 2024-04-27T03:49:09.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:09.604+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:09.605198+0000) 2024-04-27T03:49:09.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:09.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:09.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:09.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:09.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:09.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:09.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:09.730+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:10.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:10.322+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:10.322950+0000) 2024-04-27T03:49:10.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:10.341+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:10.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:10.341+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:10.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:10.341+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:10.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:10.343+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:10.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:10.274+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:10.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:10.274+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:10.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:10.274+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:10.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:10.275+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:10.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:10.409+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:10.410276+0000) 2024-04-27T03:49:10.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:10.606+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:10.607486+0000) 2024-04-27T03:49:10.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:10.694+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:10.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:10.694+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:10.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:10.694+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:10.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:10.694+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:11.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:11.326+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:11.327226+0000) 2024-04-27T03:49:11.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:11.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:11.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:11.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:11.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:11.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:11.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:11.379+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:11.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:11.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:11.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:11.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:11.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:11.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:11.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:11.259+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:11.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:11.411+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:11.412568+0000) 2024-04-27T03:49:11.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:11.611+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:11.612325+0000) 2024-04-27T03:49:11.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:11.688+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:11.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:11.688+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:11.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:11.688+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:11.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:11.689+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:12.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:49:12.361+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73101 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:49:12.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:12.358+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:12.358848+0000) 2024-04-27T03:49:12.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:12.330+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:12.331475+0000) 2024-04-27T03:49:12.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:12.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:12.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:12.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:12.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:12.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:12.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:12.358+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:12.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:12.359+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:12.359864+0000) 2024-04-27T03:49:12.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:12.246+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:12.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:12.246+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:12.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:12.246+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:12.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:12.247+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:12.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:12.358+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:12.358998+0000) 2024-04-27T03:49:12.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:12.414+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:12.414837+0000) 2024-04-27T03:49:12.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:12.613+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:12.614645+0000) 2024-04-27T03:49:12.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:12.683+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:12.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:12.683+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:12.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:12.683+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:12.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:12.684+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:13.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:13.332+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:13.333720+0000) 2024-04-27T03:49:13.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:13.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:13.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:13.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:13.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:13.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:13.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:13.356+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:13.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:13.269+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:13.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:13.269+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:13.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:13.269+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:13.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:13.271+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:13.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:13.416+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:13.416924+0000) 2024-04-27T03:49:13.909 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:13.617+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:13.617839+0000) 2024-04-27T03:49:13.909 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:13.706+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:13.909 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:13.706+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:13.909 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:13.706+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:13.909 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:13.706+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:14.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:14.240+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:14.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:14.240+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:14.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:14.240+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:14.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:14.241+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:14.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:14.418+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:14.419124+0000) 2024-04-27T03:49:14.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:14.318+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:14.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:14.318+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:14.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:14.318+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:14.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:14.320+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:14.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:14.335+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:14.336395+0000) 2024-04-27T03:49:14.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:14.619+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:14.620147+0000) 2024-04-27T03:49:14.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:14.687+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:14.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:14.687+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:14.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:14.687+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:14.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:14.688+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:15.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:15.295+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:15.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:15.295+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:15.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:15.295+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:15.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:15.297+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:15.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:15.337+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:15.338563+0000) 2024-04-27T03:49:15.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:15.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:15.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:15.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:15.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:15.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:15.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:15.251+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:15.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:15.421+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:15.422175+0000) 2024-04-27T03:49:15.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:15.623+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:15.624403+0000) 2024-04-27T03:49:15.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:15.726+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:15.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:15.726+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:15.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:15.726+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:15.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:15.726+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:16.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:16.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:16.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:16.335+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:16.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:16.335+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:16.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:16.336+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:16.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:16.340+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:16.341688+0000) 2024-04-27T03:49:16.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:16.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:16.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:16.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:16.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:16.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:16.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:16.256+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:16.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:16.425+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:16.426543+0000) 2024-04-27T03:49:16.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:16.625+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:16.626666+0000) 2024-04-27T03:49:16.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:16.775+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:16.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:16.775+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:16.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:16.775+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:16.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:16.775+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:17.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:49:17.370+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73141 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:49:17.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:17.366+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:17.367384+0000) 2024-04-27T03:49:17.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:17.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:17.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:17.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:17.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:17.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:17.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:17.216+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:17.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:17.365+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:17.366122+0000) 2024-04-27T03:49:17.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:17.427+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:17.428710+0000) 2024-04-27T03:49:17.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:17.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:17.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:17.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:17.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:17.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:17.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:17.337+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:17.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:17.343+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:17.343841+0000) 2024-04-27T03:49:17.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:17.366+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:17.367694+0000) 2024-04-27T03:49:17.828 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:17.629+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:17.629939+0000) 2024-04-27T03:49:17.828 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:17.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:18.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:17.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:18.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:17.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:18.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:17.821+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:18.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:18.219+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:18.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:18.219+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:18.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:18.219+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:18.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:18.219+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:18.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:18.347+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:18.348078+0000) 2024-04-27T03:49:18.437 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:18.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:18.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:18.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:18.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:18.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:18.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:18.368+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:18.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:18.430+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:18.430958+0000) 2024-04-27T03:49:18.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:18.631+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:18.632214+0000) 2024-04-27T03:49:19.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:18.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:19.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:18.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:19.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:18.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:19.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:18.866+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:19.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:19.349+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:19.350142+0000) 2024-04-27T03:49:19.410 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:19.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:19.410 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:19.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:19.410 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:19.215+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:19.411 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:19.216+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:19.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:19.432+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:19.433748+0000) 2024-04-27T03:49:19.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:19.633+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:19.634394+0000) 2024-04-27T03:49:19.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:19.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:19.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:19.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:19.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:19.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:19.728 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:19.404+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:20.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:19.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:20.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:19.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:20.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:19.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:20.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:19.879+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:20.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:20.244+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:20.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:20.245+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:20.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:20.245+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:20.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:20.246+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:20.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:20.436+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:20.436974+0000) 2024-04-27T03:49:20.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:20.352+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:20.353140+0000) 2024-04-27T03:49:20.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:20.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:20.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:20.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:20.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:20.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:20.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:20.418+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:20.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:20.635+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:20.636669+0000) 2024-04-27T03:49:20.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:20.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:20.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:20.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:20.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:20.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:20.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:20.871+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:21.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:21.354+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:21.355313+0000) 2024-04-27T03:49:21.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:21.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:21.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:21.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:21.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:21.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:21.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:21.386+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:21.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:21.244+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:21.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:21.244+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:21.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:21.244+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:21.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:21.244+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:21.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:21.438+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:21.439120+0000) 2024-04-27T03:49:21.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:21.638+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:21.638963+0000) 2024-04-27T03:49:22.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:21.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:22.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:21.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:22.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:21.910+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:22.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:21.911+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:22.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:49:22.378+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73181 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:49:22.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:22.374+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:22.375560+0000) 2024-04-27T03:49:22.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:22.357+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:22.358107+0000) 2024-04-27T03:49:22.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:22.370+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:22.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:22.370+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:22.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:22.370+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:22.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:22.371+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:22.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:22.375+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:22.376441+0000) 2024-04-27T03:49:22.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:22.257+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:22.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:22.257+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:22.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:22.257+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:22.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:22.258+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:22.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:22.375+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:22.376065+0000) 2024-04-27T03:49:22.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:22.441+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:22.441865+0000) 2024-04-27T03:49:22.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:22.640+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:22.641316+0000) 2024-04-27T03:49:23.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:22.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:23.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:22.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:23.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:22.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:23.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:22.955+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:23.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:23.210+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:23.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:23.210+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:23.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:23.210+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:23.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:23.211+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:23.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:23.359+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:23.360249+0000) 2024-04-27T03:49:23.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:23.408+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:23.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:23.408+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:23.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:23.408+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:23.451 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:23.409+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:23.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:23.642+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:23.643524+0000) 2024-04-27T03:49:23.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:23.443+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:23.443974+0000) 2024-04-27T03:49:24.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:23.984+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:24.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:23.984+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:24.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:23.984+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:24.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:23.985+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:24.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:24.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:24.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:24.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:24.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:24.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:24.200 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:24.163+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:24.453 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:24.362+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:24.363178+0000) 2024-04-27T03:49:24.453 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:24.380+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:24.453 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:24.380+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:24.453 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:24.380+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:24.453 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:24.380+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:24.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:24.644+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:24.645655+0000) 2024-04-27T03:49:24.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:24.446+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:24.447055+0000) 2024-04-27T03:49:25.187 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:25.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:25.187 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:25.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:25.187 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:25.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:25.187 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:25.022+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:25.455 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:25.180+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:25.455 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:25.180+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:25.456 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:25.180+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:25.456 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:25.181+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:25.456 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:25.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:25.456 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:25.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:25.456 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:25.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:25.456 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:25.340+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:25.456 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:25.364+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:25.365396+0000) 2024-04-27T03:49:25.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:25.448+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:25.449532+0000) 2024-04-27T03:49:25.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:25.647+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:25.647839+0000) 2024-04-27T03:49:26.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:26.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:26.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:26.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:26.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:26.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:26.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:26.070+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:26.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:26.159+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:26.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:26.159+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:26.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:26.159+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:26.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:26.160+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:26.655 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:26.452+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:26.453732+0000) 2024-04-27T03:49:26.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:26.296+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:26.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:26.296+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:26.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:26.296+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:26.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:26.297+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:26.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:26.367+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:26.368305+0000) 2024-04-27T03:49:26.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:26.649+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:26.649994+0000) 2024-04-27T03:49:27.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:27.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:27.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:27.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:27.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:27.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:27.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:27.129+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:27.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:27.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:27.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:27.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:27.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:27.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:27.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:27.030+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:27.462 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:49:27.387+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73221 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:49:27.462 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:27.383+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:27.384010+0000) 2024-04-27T03:49:27.463 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:27.383+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:27.384208+0000) 2024-04-27T03:49:27.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:27.263+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:27.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:27.263+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:27.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:27.263+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:27.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:27.264+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:27.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:27.370+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:27.370966+0000) 2024-04-27T03:49:27.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:27.383+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:27.384473+0000) 2024-04-27T03:49:27.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:27.455+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:27.456391+0000) 2024-04-27T03:49:27.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:27.651+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:27.652781+0000) 2024-04-27T03:49:28.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:28.149+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:28.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:28.149+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:28.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:28.149+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:28.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:28.150+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:28.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:28.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:28.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:28.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:28.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:28.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:28.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:28.009+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:28.465 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:28.222+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:28.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:28.222+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:28.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:28.222+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:28.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:28.223+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:28.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:28.373+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:28.374100+0000) 2024-04-27T03:49:28.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:28.458+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:28.459718+0000) 2024-04-27T03:49:28.727 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:28.655+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:28.656066+0000) 2024-04-27T03:49:29.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:29.133+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:29.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:29.133+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:29.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:29.133+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:29.194 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:29.134+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:29.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:29.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:29.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:29.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:29.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:29.004+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:29.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:29.004+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:29.468 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:29.187+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:29.468 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:29.187+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:29.468 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:29.187+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:29.468 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:29.188+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:29.468 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:29.375+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:29.376369+0000) 2024-04-27T03:49:29.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:29.658+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:29.658849+0000) 2024-04-27T03:49:29.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:29.461+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:29.462439+0000) 2024-04-27T03:49:30.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:30.142+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:30.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:30.142+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:30.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:30.142+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:30.199 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:30.143+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:30.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:29.970+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:30.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:29.970+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:30.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:29.970+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:30.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:29.971+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:30.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:30.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:30.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:30.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:30.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:30.210+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:30.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:30.211+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:30.463 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:30.377+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:30.378577+0000) 2024-04-27T03:49:30.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:30.660+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:30.661006+0000) 2024-04-27T03:49:30.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:30.464+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:30.464913+0000) 2024-04-27T03:49:31.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:30.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:31.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:30.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:31.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:30.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:31.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:30.981+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:31.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:31.185+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:31.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:31.185+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:31.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:31.185+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:31.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:31.186+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:31.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:31.246+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:31.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:31.246+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:31.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:31.246+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:31.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:31.247+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:31.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:31.380+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:31.380847+0000) 2024-04-27T03:49:31.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:31.662+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:31.663381+0000) 2024-04-27T03:49:31.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:31.468+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:31.469350+0000) 2024-04-27T03:49:32.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:31.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:32.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:31.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:32.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:31.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:32.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:31.985+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:32.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:49:32.397+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73261 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:49:32.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:32.392+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:32.392892+0000) 2024-04-27T03:49:32.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:32.187+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:32.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:32.187+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:32.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:32.187+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:32.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:32.188+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:32.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:32.392+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:32.393661+0000) 2024-04-27T03:49:32.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:32.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:32.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:32.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:32.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:32.233+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:32.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:32.234+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:32.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:32.382+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:32.382971+0000) 2024-04-27T03:49:32.479 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:32.392+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:32.393255+0000) 2024-04-27T03:49:32.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:32.666+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:32.667675+0000) 2024-04-27T03:49:32.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:32.471+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:32.471932+0000) 2024-04-27T03:49:33.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:33.002+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:33.190 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:33.002+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:33.190 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:33.002+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:33.190 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:33.002+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:33.190 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:33.182+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:33.190 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:33.182+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:33.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:33.182+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:33.191 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:33.183+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:33.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:33.214+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:33.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:33.214+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:33.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:33.214+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:33.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:33.215+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:33.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:33.385+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:33.386146+0000) 2024-04-27T03:49:33.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:33.473+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:33.474683+0000) 2024-04-27T03:49:33.927 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:33.669+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:33.669883+0000) 2024-04-27T03:49:34.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:33.979+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:34.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:33.979+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:34.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:33.979+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:34.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:33.980+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:34.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:34.225+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:34.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:34.225+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:34.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:34.225+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:34.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:34.226+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:34.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:34.263+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:34.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:34.263+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:34.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:34.263+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:34.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:34.264+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:34.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:34.387+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:34.388351+0000) 2024-04-27T03:49:34.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:34.478+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:34.479034+0000) 2024-04-27T03:49:34.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:34.671+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:34.671924+0000) 2024-04-27T03:49:35.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:34.996+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:35.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:34.996+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:35.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:34.996+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:35.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:34.996+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:35.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:35.226+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:35.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:35.226+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:35.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:35.226+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:35.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:35.227+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:35.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:35.389+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:35.390551+0000) 2024-04-27T03:49:35.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:35.272+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:35.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:35.272+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:35.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:35.272+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:35.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:35.273+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:35.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:35.480+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:35.481170+0000) 2024-04-27T03:49:35.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:35.673+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:35.674069+0000) 2024-04-27T03:49:36.199 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:35.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:36.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:35.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:36.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:35.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:36.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:35.948+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:36.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:36.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:36.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:36.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:36.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:36.250+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:36.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:36.251+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:36.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:36.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:36.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:36.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:36.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:36.256+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:36.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:36.258+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:36.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:36.392+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:36.392860+0000) 2024-04-27T03:49:36.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:36.482+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:36.483277+0000) 2024-04-27T03:49:36.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:36.675+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:36.676463+0000) 2024-04-27T03:49:37.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:36.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:37.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:36.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:37.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:36.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:37.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:36.915+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:37.476 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:49:37.405+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73301 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:49:37.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:37.401+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:37.402711+0000) 2024-04-27T03:49:37.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:37.229+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:37.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:37.229+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:37.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:37.229+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:37.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:37.229+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:37.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:37.394+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:37.395293+0000) 2024-04-27T03:49:37.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:37.401+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:37.402669+0000) 2024-04-27T03:49:37.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:37.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:37.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:37.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:37.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:37.291+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:37.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:37.292+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:37.479 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:37.401+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:37.402014+0000) 2024-04-27T03:49:37.880 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:37.677+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:37.678722+0000) 2024-04-27T03:49:37.880 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:37.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:37.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:37.484+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:37.485413+0000) 2024-04-27T03:49:38.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:37.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:38.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:37.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:38.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:37.873+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:38.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:38.183+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:38.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:38.183+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:38.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:38.183+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:38.404 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:38.185+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:38.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:38.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:38.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:38.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:38.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:38.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:38.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:38.262+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:38.686 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:38.486+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:38.487540+0000) 2024-04-27T03:49:38.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:38.396+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:38.397448+0000) 2024-04-27T03:49:38.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:38.680+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:38.681017+0000) 2024-04-27T03:49:38.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:38.888+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:38.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:38.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:38.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:38.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:38.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:38.889+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:39.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:39.224+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:39.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:39.224+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:39.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:39.224+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:39.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:39.225+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:39.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:39.215+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:39.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:39.215+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:39.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:39.215+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:39.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:39.216+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:39.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:39.399+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:39.400204+0000) 2024-04-27T03:49:39.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:39.682+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:39.683526+0000) 2024-04-27T03:49:39.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:39.489+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:39.490409+0000) 2024-04-27T03:49:40.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:39.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:40.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:39.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:40.200 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:39.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:40.201 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:39.925+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:40.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:40.195+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:40.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:40.195+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:40.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:40.195+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:40.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:40.196+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:40.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:40.198+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:40.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:40.198+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:40.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:40.198+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:40.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:40.199+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:40.478 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:40.401+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:40.402356+0000) 2024-04-27T03:49:40.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:40.684+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:40.685723+0000) 2024-04-27T03:49:40.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:40.491+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:40.492576+0000) 2024-04-27T03:49:41.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:40.904+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:41.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:40.904+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:41.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:40.904+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:41.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:40.905+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:41.197 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:41.153+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:41.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:41.153+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:41.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:41.153+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:41.198 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:41.154+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:41.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:41.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:41.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:41.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:41.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:41.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:41.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:41.192+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:41.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:41.405+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:41.406591+0000) 2024-04-27T03:49:41.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:41.688+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:41.689074+0000) 2024-04-27T03:49:41.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:41.493+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:41.494764+0000) 2024-04-27T03:49:42.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:41.890+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:42.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:41.890+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:42.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:41.890+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:42.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:41.891+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:42.166 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:42.117+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:42.166 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:42.117+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:42.166 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:42.117+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:42.166 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:42.117+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:42.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:42.158+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:42.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:42.158+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:42.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:42.158+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:42.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:42.159+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:42.696 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:49:42.415+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73341 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:49:42.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:42.411+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:42.412062+0000) 2024-04-27T03:49:42.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:42.411+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:42.412089+0000) 2024-04-27T03:49:42.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:42.411+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:42.411996+0000) 2024-04-27T03:49:42.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:42.410+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:42.411050+0000) 2024-04-27T03:49:42.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:42.498+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:42.499586+0000) 2024-04-27T03:49:42.957 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:42.690+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:42.691422+0000) 2024-04-27T03:49:42.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:42.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:42.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:42.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:42.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:42.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:42.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:42.850+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:43.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:43.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:43.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:43.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:43.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:43.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:43.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:43.136+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:43.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:43.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:43.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:43.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:43.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:43.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:43.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:43.165+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:43.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:43.414+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:43.414955+0000) 2024-04-27T03:49:43.700 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:43.500+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:43.501739+0000) 2024-04-27T03:49:43.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:43.693+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:43.694719+0000) 2024-04-27T03:49:43.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:43.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:43.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:43.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:43.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:43.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:43.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:43.821+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:44.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:44.144+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:44.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:44.144+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:44.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:44.144+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:44.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:44.145+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:44.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:44.169+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:44.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:44.169+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:44.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:44.169+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:44.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:44.169+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:44.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:44.416+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:44.417003+0000) 2024-04-27T03:49:44.704 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:44.503+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:44.504188+0000) 2024-04-27T03:49:44.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:44.697+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:44.698033+0000) 2024-04-27T03:49:44.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:44.830+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:44.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:44.830+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:44.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:44.830+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:44.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:44.830+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:45.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:45.115+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:45.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:45.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:45.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:45.116+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:45.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:45.116+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:45.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:45.199+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:45.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:45.199+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:45.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:45.199+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:45.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:45.199+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:45.602 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:45.420+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:45.421277+0000) 2024-04-27T03:49:45.602 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:45.504+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:45.506270+0000) 2024-04-27T03:49:45.853 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:45.698+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:45.700349+0000) 2024-04-27T03:49:46.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:45.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:46.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:45.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:46.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:45.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:46.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:45.847+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:46.132 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:46.094+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:46.132 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:46.094+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:46.132 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:46.094+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:46.132 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:46.095+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:46.429 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:46.174+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:46.429 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:46.174+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:46.429 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:46.174+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:46.429 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:46.175+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:46.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:46.421+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:46.423508+0000) 2024-04-27T03:49:46.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:46.508+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:46.510634+0000) 2024-04-27T03:49:46.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:46.700+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:46.702674+0000) 2024-04-27T03:49:46.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:46.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:46.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:46.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:46.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:46.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:46.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:46.820+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:47.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:47.094+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:47.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:47.094+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:47.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:47.094+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:47.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:47.095+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:47.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:47.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:47.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:47.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:47.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:47.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:47.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:47.161+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:47.520 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:49:47.422+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73381 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:49:47.521 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:47.420+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:47.421850+0000) 2024-04-27T03:49:47.521 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:47.419+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:47.421145+0000) 2024-04-27T03:49:47.521 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:47.424+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:47.425929+0000) 2024-04-27T03:49:47.521 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:47.419+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:47.421452+0000) 2024-04-27T03:49:47.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:47.513+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:47.515002+0000) 2024-04-27T03:49:47.844 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:47.704+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:47.706226+0000) 2024-04-27T03:49:48.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:47.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:48.131 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:47.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:48.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:47.867+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:48.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:47.868+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:48.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:48.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:48.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:48.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:48.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:48.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:48.434 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:48.129+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:48.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:48.199+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:48.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:48.199+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:48.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:48.199+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:48.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:48.199+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:48.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:48.426+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:48.428752+0000) 2024-04-27T03:49:48.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:48.515+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:48.517659+0000) 2024-04-27T03:49:48.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:48.706+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:48.708320+0000) 2024-04-27T03:49:48.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:48.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:48.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:48.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:48.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:48.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:48.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:48.851+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:49.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:49.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:49.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:49.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:49.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:49.088+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:49.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:49.089+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:49.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:49.171+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:49.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:49.171+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:49.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:49.171+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:49.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:49.172+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:49.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:49.429+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:49.430957+0000) 2024-04-27T03:49:49.718 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:49.517+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:49.519764+0000) 2024-04-27T03:49:49.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:49.710+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:49.712273+0000) 2024-04-27T03:49:49.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:49.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:49.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:49.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:49.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:49.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:49.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:49.827+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:50.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:50.137+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:50.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:50.137+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:50.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:50.137+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:50.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:50.138+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:50.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:50.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:50.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:50.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:50.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:50.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:50.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:50.141+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:50.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:50.432+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:50.434126+0000) 2024-04-27T03:49:50.647 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:50.521+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:50.523032+0000) 2024-04-27T03:49:50.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:50.714+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:50.716555+0000) 2024-04-27T03:49:50.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:50.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:50.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:50.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:50.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:50.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:50.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:50.795+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:51.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:51.110+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:51.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:51.110+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:51.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:51.110+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:51.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:51.111+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:51.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:51.174+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:51.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:51.174+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:51.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:51.174+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:51.228 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:51.175+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:51.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:51.434+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:51.436467+0000) 2024-04-27T03:49:51.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:51.523+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:51.525345+0000) 2024-04-27T03:49:51.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:51.717+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:51.718887+0000) 2024-04-27T03:49:51.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:51.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:51.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:51.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:51.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:51.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:51.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:51.834+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:52.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:52.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:52.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:52.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:52.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:52.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:52.320 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:52.063+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:52.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:52.201+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:52.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:52.201+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:52.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:52.201+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:52.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:52.202+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:52.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:49:52.430+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73421 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:49:52.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:52.427+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:52.428898+0000) 2024-04-27T03:49:52.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:52.438+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:52.439822+0000) 2024-04-27T03:49:52.728 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:52.427+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:52.429317+0000) 2024-04-27T03:49:52.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:52.426+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:52.428466+0000) 2024-04-27T03:49:52.728 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:52.525+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:52.527534+0000) 2024-04-27T03:49:53.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:52.720+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:52.722093+0000) 2024-04-27T03:49:53.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:52.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:53.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:52.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:53.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:52.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:53.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:52.845+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:53.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:53.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:53.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:53.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:53.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:53.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:53.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:53.227+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:53.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:53.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:53.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:53.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:53.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:53.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:53.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:53.055+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:53.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:53.440+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:53.442515+0000) 2024-04-27T03:49:53.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:53.527+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:53.529793+0000) 2024-04-27T03:49:54.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:53.722+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:53.724777+0000) 2024-04-27T03:49:54.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:53.854+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:54.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:53.854+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:54.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:53.854+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:54.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:53.855+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:54.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:54.187+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:54.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:54.187+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:54.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:54.187+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:54.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:54.188+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:54.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:54.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:54.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:54.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:54.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:54.033+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:54.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:54.034+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:54.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:54.443+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:54.445683+0000) 2024-04-27T03:49:54.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:54.532+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:54.534113+0000) 2024-04-27T03:49:55.023 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:54.727+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:54.729057+0000) 2024-04-27T03:49:55.023 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:54.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:55.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:54.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:55.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:54.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:55.024 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:54.887+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:55.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:55.191+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:55.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:55.191+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:55.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:55.191+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:55.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:55.191+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:55.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:55.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:55.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:55.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:55.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:55.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:55.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:55.017+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:55.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:55.534+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:55.536312+0000) 2024-04-27T03:49:55.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:55.446+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:55.448403+0000) 2024-04-27T03:49:55.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:55.730+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:55.731154+0000) 2024-04-27T03:49:55.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:55.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:55.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:55.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:55.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:55.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:55.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:55.927+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:56.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:56.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:56.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:56.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:56.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:56.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:56.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:56.211+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:56.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:56.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:56.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:56.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:56.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:56.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:56.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:56.054+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:56.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:56.537+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:56.538580+0000) 2024-04-27T03:49:56.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:56.449+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:56.451620+0000) 2024-04-27T03:49:57.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:56.732+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:56.733268+0000) 2024-04-27T03:49:57.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:56.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:57.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:56.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:57.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:56.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:57.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:56.941+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:57.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:57.236+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:57.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:57.236+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:57.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:57.236+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:57.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:57.237+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:57.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:57.037+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:57.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:57.037+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:57.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:57.037+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:57.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:57.038+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:57.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:57.435+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:57.437200+0000) 2024-04-27T03:49:57.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:49:57.439+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73461 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:49:57.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:57.435+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:57.437657+0000) 2024-04-27T03:49:57.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:57.455+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:57.456036+0000) 2024-04-27T03:49:57.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:57.435+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:57.437707+0000) 2024-04-27T03:49:57.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:57.540+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:57.541197+0000) 2024-04-27T03:49:58.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:57.734+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:57.735514+0000) 2024-04-27T03:49:58.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:57.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:58.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:57.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:58.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:57.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:58.032 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:57.949+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:58.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:58.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:58.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:58.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:58.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:58.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:58.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:58.189+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:58.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:58.083+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:58.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:58.083+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:58.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:58.083+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:58.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:58.084+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:58.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:58.458+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:58.459074+0000) 2024-04-27T03:49:58.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:58.542+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:58.543276+0000) 2024-04-27T03:49:59.030 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:58.737+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:58.737860+0000) 2024-04-27T03:49:59.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:58.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:59.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:58.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:59.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:58.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:49:59.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:58.902+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:59.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:59.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:59.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:59.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:59.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:59.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:49:59.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:59.090+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:59.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:59.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:59.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:59.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:59.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:59.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:49:59.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:59.146+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:49:59.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:49:59.462+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:59.463366+0000) 2024-04-27T03:49:59.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:49:59.545+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:59.545923+0000) 2024-04-27T03:50:00.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:59.739+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:49:59.740099+0000) 2024-04-27T03:50:00.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:59.926+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:00.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:59.926+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:00.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:59.926+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:00.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:49:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:49:59.926+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:00.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:00.119+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:00.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:00.119+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:00.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:00.119+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:00.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:00.120+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:00.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:00.151+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:00.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:00.151+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:00.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:00.151+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:00.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:00.152+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:00.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:00.549+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:00.550549+0000) 2024-04-27T03:50:00.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:00.464+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:00.465579+0000) 2024-04-27T03:50:00.987 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:00.741+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:00.742402+0000) 2024-04-27T03:50:00.987 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:00.967+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:00.987 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:00.967+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:00.987 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:00.967+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:00.987 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:00.968+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:01.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:01.112+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:01.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:01.112+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:01.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:01.112+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:01.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:01.112+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:01.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:01.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:01.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:01.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:01.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:01.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:01.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:01.167+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:01.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:01.467+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:01.467842+0000) 2024-04-27T03:50:01.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:01.552+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:01.552841+0000) 2024-04-27T03:50:02.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:01.744+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:01.745780+0000) 2024-04-27T03:50:02.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:01.989+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:02.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:01.989+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:02.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:01.989+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:02.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:01.990+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:02.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:02.078+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:02.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:02.078+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:02.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:02.078+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:02.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:02.078+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:02.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:02.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:02.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:02.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:02.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:02.159+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:02.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:02.160+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:02.574 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:50:02.449+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73501 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:50:02.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:02.445+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:02.446657+0000) 2024-04-27T03:50:02.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:02.470+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:02.471044+0000) 2024-04-27T03:50:02.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:02.444+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:02.445094+0000) 2024-04-27T03:50:02.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:02.444+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:02.444836+0000) 2024-04-27T03:50:02.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:02.554+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:02.554986+0000) 2024-04-27T03:50:02.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:02.748+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:02.749065+0000) 2024-04-27T03:50:02.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:02.959+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:02.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:02.959+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:02.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:02.959+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:03.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:02.960+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:03.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:03.106+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:03.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:03.106+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:03.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:03.106+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:03.227 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:03.107+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:03.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:03.110+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:03.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:03.110+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:03.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:03.110+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:03.227 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:03.112+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:03.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:03.474+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:03.474953+0000) 2024-04-27T03:50:03.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:03.558+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:03.559633+0000) 2024-04-27T03:50:04.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:03.750+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:03.751192+0000) 2024-04-27T03:50:04.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:03.925+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:04.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:03.925+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:04.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:03.925+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:04.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:03.926+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:04.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:04.160+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:04.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:04.160+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:04.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:04.160+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:04.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:04.161+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:04.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:04.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:04.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:04.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:04.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:04.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:04.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:04.082+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:04.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:04.476+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:04.477255+0000) 2024-04-27T03:50:04.726 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:04.560+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:04.561774+0000) 2024-04-27T03:50:05.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:04.752+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:04.753556+0000) 2024-04-27T03:50:05.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:04.967+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:05.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:04.967+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:05.031 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:04.967+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:05.032 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:04.967+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:05.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:05.173+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:05.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:05.173+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:05.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:05.173+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:05.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:05.173+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:05.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:05.101+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:05.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:05.101+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:05.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:05.101+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:05.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:05.102+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:05.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:05.564+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:05.565066+0000) 2024-04-27T03:50:05.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:05.478+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:05.479548+0000) 2024-04-27T03:50:05.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:05.755+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:05.756682+0000) 2024-04-27T03:50:06.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:06.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:06.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:06.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:06.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:06.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:06.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:06.217+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:06.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:05.977+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:06.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:05.977+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:06.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:05.977+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:06.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:05.978+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:06.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:06.132+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:06.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:06.132+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:06.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:06.132+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:06.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:06.132+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:06.572 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:06.480+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:06.481730+0000) 2024-04-27T03:50:06.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:06.566+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:06.567149+0000) 2024-04-27T03:50:06.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:06.760+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:06.761512+0000) 2024-04-27T03:50:07.247 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:07.179+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:07.247 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:07.180+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:07.247 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:07.180+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:07.247 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:07.181+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:07.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:07.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:07.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:07.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:07.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:07.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:07.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:07.013+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:07.574 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:50:07.458+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73541 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:50:07.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:07.453+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:07.454690+0000) 2024-04-27T03:50:07.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:07.454+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:07.455456+0000) 2024-04-27T03:50:07.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:07.240+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:07.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:07.240+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:07.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:07.240+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:07.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:07.242+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:07.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:07.453+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:07.454514+0000) 2024-04-27T03:50:07.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:07.483+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:07.483826+0000) 2024-04-27T03:50:07.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:07.763+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:07.764745+0000) 2024-04-27T03:50:07.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:07.568+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:07.569594+0000) 2024-04-27T03:50:08.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:08.208+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:08.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:08.208+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:08.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:08.208+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:08.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:08.208+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:08.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:08.019+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:08.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:08.019+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:08.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:08.019+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:08.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:08.020+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:08.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:08.151+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:08.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:08.151+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:08.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:08.151+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:08.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:08.151+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:08.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:08.485+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:08.485973+0000) 2024-04-27T03:50:08.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:08.572+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:08.572842+0000) 2024-04-27T03:50:08.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:08.768+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:08.769016+0000) 2024-04-27T03:50:09.260 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:09.153+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:09.260 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:09.153+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:09.260 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:09.153+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:09.260 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:09.153+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:09.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:09.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:09.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:09.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:09.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:09.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:09.261 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:09.039+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:09.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:09.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:09.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:09.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:09.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:09.253+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:09.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:09.254+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:09.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:09.489+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:09.490210+0000) 2024-04-27T03:50:09.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:09.574+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:09.575645+0000) 2024-04-27T03:50:09.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:09.770+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:09.771678+0000) 2024-04-27T03:50:10.255 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:10.140+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:10.255 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:10.140+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:10.255 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:10.140+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:10.256 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:10.141+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:10.256 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:10.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:10.256 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:10.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:10.256 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:10.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:10.256 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:10.024+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:10.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:10.248+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:10.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:10.248+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:10.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:10.248+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:10.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:10.249+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:10.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:10.491+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:10.492461+0000) 2024-04-27T03:50:10.839 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:10.773+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:10.773849+0000) 2024-04-27T03:50:10.839 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:10.579+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:10.579839+0000) 2024-04-27T03:50:11.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:10.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:11.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:10.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:11.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:10.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:11.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:10.980+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:11.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:11.247+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:11.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:11.247+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:11.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:11.247+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:11.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:11.248+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:11.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:11.172+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:11.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:11.172+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:11.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:11.172+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:11.478 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:11.173+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:11.782 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:11.493+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:11.494600+0000) 2024-04-27T03:50:11.782 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:11.581+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:11.582196+0000) 2024-04-27T03:50:12.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:11.776+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:11.777071+0000) 2024-04-27T03:50:12.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:11.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:12.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:11.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:12.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:11.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:12.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:11.982+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:12.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:12.248+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:12.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:12.248+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:12.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:12.248+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:12.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:12.249+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:12.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:12.174+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:12.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:12.174+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:12.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:12.174+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:12.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:12.175+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:12.726 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:50:12.466+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73581 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:50:12.726 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:12.462+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:12.463281+0000) 2024-04-27T03:50:12.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:12.464+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:12.465438+0000) 2024-04-27T03:50:12.727 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:12.495+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:12.496779+0000) 2024-04-27T03:50:12.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:12.463+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:12.464545+0000) 2024-04-27T03:50:12.727 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:12.586+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:12.587326+0000) 2024-04-27T03:50:13.035 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:12.778+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:12.779456+0000) 2024-04-27T03:50:13.035 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:13.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:13.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:13.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:13.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:13.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:13.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:13.030+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:13.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:13.172+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:13.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:13.172+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:13.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:13.172+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:13.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:13.173+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:13.326 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:13.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:13.326 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:13.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:13.326 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:13.252+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:13.326 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:13.253+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:13.595 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:13.498+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:13.499455+0000) 2024-04-27T03:50:13.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:13.781+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:13.782345+0000) 2024-04-27T03:50:13.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:13.589+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:13.590286+0000) 2024-04-27T03:50:14.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:14.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:14.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:14.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:14.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:14.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:14.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:14.151+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:14.204 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:13.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:14.204 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:13.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:14.204 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:13.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:14.205 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:13.996+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:14.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:14.244+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:14.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:14.244+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:14.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:14.244+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:14.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:14.244+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:14.790 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:14.502+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:14.503693+0000) 2024-04-27T03:50:14.790 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:14.591+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:14.592564+0000) 2024-04-27T03:50:15.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:14.784+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:14.785281+0000) 2024-04-27T03:50:15.052 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:15.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:15.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:15.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:15.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:15.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:15.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:15.014+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:15.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:15.245+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:15.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:15.245+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:15.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:15.245+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:15.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:15.246+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:15.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:15.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:15.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:15.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:15.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:15.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:15.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:15.189+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:15.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:15.506+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:15.506941+0000) 2024-04-27T03:50:15.877 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:15.786+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:15.787489+0000) 2024-04-27T03:50:15.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:15.594+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:15.594825+0000) 2024-04-27T03:50:16.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:15.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:16.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:15.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:16.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:15.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:16.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:15.999+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:16.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:16.207+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:16.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:16.207+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:16.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:16.207+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:16.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:16.207+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:16.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:16.221+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:16.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:16.221+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:16.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:16.221+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:16.477 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:16.221+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:16.795 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:16.597+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:16.598117+0000) 2024-04-27T03:50:16.796 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:16.510+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:16.511202+0000) 2024-04-27T03:50:17.049 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:16.789+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:16.790372+0000) 2024-04-27T03:50:17.049 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:17.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:17.049 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:17.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:17.049 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:17.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:17.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:17.043+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:17.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:17.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:17.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:17.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:17.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:17.242+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:17.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:17.243+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:17.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:17.182+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:17.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:17.182+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:17.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:17.182+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:17.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:17.183+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:17.599 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:17.471+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:17.471952+0000) 2024-04-27T03:50:17.599 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:50:17.475+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73621 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:50:17.599 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:17.471+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:17.472708+0000) 2024-04-27T03:50:17.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:17.472+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:17.473212+0000) 2024-04-27T03:50:17.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:17.512+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:17.513466+0000) 2024-04-27T03:50:17.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:17.599+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:17.600230+0000) 2024-04-27T03:50:17.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:17.793+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:17.793898+0000) 2024-04-27T03:50:18.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:18.208+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:18.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:18.208+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:18.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:18.208+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:18.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:18.209+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:18.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:18.144+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:18.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:18.144+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:18.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:18.144+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:18.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:18.145+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:18.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:18.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:18.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:18.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:18.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:18.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:18.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:18.046+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:18.607 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:18.516+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:18.516951+0000) 2024-04-27T03:50:18.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:18.601+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:18.602401+0000) 2024-04-27T03:50:18.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:18.795+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:18.796048+0000) 2024-04-27T03:50:19.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:19.073+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:19.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:19.073+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:19.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:19.073+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:19.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:19.074+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:19.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:19.202+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:19.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:19.202+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:19.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:19.202+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:19.324 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:19.203+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:19.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:19.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:19.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:19.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:19.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:19.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:19.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:19.097+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:19.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:19.518+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:19.519022+0000) 2024-04-27T03:50:19.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:19.603+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:19.604531+0000) 2024-04-27T03:50:19.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:19.800+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:19.800889+0000) 2024-04-27T03:50:20.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:20.177+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:20.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:20.177+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:20.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:20.177+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:20.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:20.178+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:20.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:20.086+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:20.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:20.086+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:20.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:20.086+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:20.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:20.086+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:20.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:20.110+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:20.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:20.110+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:20.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:20.110+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:20.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:20.111+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:20.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:20.520+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:20.521662+0000) 2024-04-27T03:50:20.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:20.608+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:20.609257+0000) 2024-04-27T03:50:20.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:20.802+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:20.803756+0000) 2024-04-27T03:50:21.204 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:21.115+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:21.204 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:21.115+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:21.204 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:21.115+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:21.204 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:21.115+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:21.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:21.073+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:21.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:21.073+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:21.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:21.073+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:21.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:21.074+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:21.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:21.209+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:21.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:21.209+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:21.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:21.209+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:21.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:21.210+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:21.811 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:21.525+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:21.525903+0000) 2024-04-27T03:50:21.811 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:21.611+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:21.612293+0000) 2024-04-27T03:50:22.093 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:22.027+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:22.093 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:22.027+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:22.093 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:22.027+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:22.093 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:22.028+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:22.094 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:21.805+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:21.805933+0000) 2024-04-27T03:50:22.094 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:22.087+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:22.094 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:22.087+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:22.094 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:22.087+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:22.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:22.088+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:22.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:22.221+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:22.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:22.221+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:22.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:22.221+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:22.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:22.221+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:22.814 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:50:22.484+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73661 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:50:22.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:22.482+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:22.483739+0000) 2024-04-27T03:50:22.814 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:22.528+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:22.529104+0000) 2024-04-27T03:50:22.815 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:22.480+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:22.480969+0000) 2024-04-27T03:50:22.815 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:22.480+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:22.481419+0000) 2024-04-27T03:50:22.815 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:22.613+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:22.614583+0000) 2024-04-27T03:50:23.089 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:22.808+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:22.808992+0000) 2024-04-27T03:50:23.089 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:23.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:23.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:22.993+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:23.090 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:22.993+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:23.090 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:22.993+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:23.090 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:22.994+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:23.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:23.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:23.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:23.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:23.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:23.084+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:23.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:23.232+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:23.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:23.232+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:23.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:23.232+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:23.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:23.233+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:23.818 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:23.531+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:23.532110+0000) 2024-04-27T03:50:23.819 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:23.618+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:23.619191+0000) 2024-04-27T03:50:24.110 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:23.813+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:23.813848+0000) 2024-04-27T03:50:24.111 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:23.997+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:24.111 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:23.997+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:24.111 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:23.997+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:24.111 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:23.999+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:24.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:24.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:24.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:24.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:24.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:24.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:24.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:24.105+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:24.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:24.276+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:24.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:24.276+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:24.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:24.276+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:24.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:24.277+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:24.821 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:24.533+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:24.534332+0000) 2024-04-27T03:50:24.821 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:24.620+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:24.621437+0000) 2024-04-27T03:50:25.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:24.815+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:24.816099+0000) 2024-04-27T03:50:25.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:24.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:25.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:24.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:25.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:24.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:25.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:24.951+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:25.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:25.297+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:25.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:25.297+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:25.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:25.297+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:25.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:25.298+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:25.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:25.122+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:25.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:25.122+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:25.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:25.122+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:25.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:25.123+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:25.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:25.535+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:25.536458+0000) 2024-04-27T03:50:25.823 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:25.622+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:25.623728+0000) 2024-04-27T03:50:26.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:25.817+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:25.818559+0000) 2024-04-27T03:50:26.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:25.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:26.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:25.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:26.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:25.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:26.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:25.941+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:26.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:26.130+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:26.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:26.130+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:26.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:26.130+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:26.476 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:26.131+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:26.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:26.288+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:26.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:26.288+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:26.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:26.288+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:26.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:26.289+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:26.826 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:26.627+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:26.628260+0000) 2024-04-27T03:50:26.827 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:26.537+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:26.538763+0000) 2024-04-27T03:50:27.095 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:26.820+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:26.821481+0000) 2024-04-27T03:50:27.095 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:26.991+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:27.095 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:26.991+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:27.095 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:26.991+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:27.095 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:26.992+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:27.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:27.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:27.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:27.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:27.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:27.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:27.477 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:27.306+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:27.477 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:27.089+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:27.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:27.089+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:27.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:27.089+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:27.478 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:27.090+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:27.828 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:50:27.493+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73701 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:50:27.829 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:27.489+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:27.490639+0000) 2024-04-27T03:50:27.829 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:27.540+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:27.541490+0000) 2024-04-27T03:50:27.829 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:27.489+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:27.490704+0000) 2024-04-27T03:50:27.830 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:27.629+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:27.630532+0000) 2024-04-27T03:50:27.830 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:27.488+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:27.489541+0000) 2024-04-27T03:50:28.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:27.823+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:27.823943+0000) 2024-04-27T03:50:28.136 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:28.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:28.136 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:28.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:28.136 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:28.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:28.136 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:28.058+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:28.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:27.987+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:28.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:27.987+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:28.137 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:27.987+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:28.137 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:27.988+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:28.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:28.286+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:28.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:28.286+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:28.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:28.286+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:28.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:28.287+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:28.834 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:28.544+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:28.545694+0000) 2024-04-27T03:50:28.834 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:28.631+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:28.632579+0000) 2024-04-27T03:50:29.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:28.828+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:28.828951+0000) 2024-04-27T03:50:29.088 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:28.965+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:29.088 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:28.965+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:29.088 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:28.965+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:29.088 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:28.966+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:29.339 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:29.250+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:29.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:29.250+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:29.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:29.250+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:29.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:29.251+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:29.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:29.081+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:29.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:29.081+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:29.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:29.081+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:29.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:29.082+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:29.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:29.548+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:29.548894+0000) 2024-04-27T03:50:29.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:29.833+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:29.834027+0000) 2024-04-27T03:50:29.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:29.633+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:29.634789+0000) 2024-04-27T03:50:30.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:29.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:30.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:29.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:30.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:29.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:30.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:29.968+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:30.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:30.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:30.226 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:30.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:30.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:30.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:30.227 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:30.053+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:30.555 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:30.262+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:30.556 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:30.262+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:30.556 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:30.262+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:30.556 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:30.263+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:30.844 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:30.550+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:30.551068+0000) 2024-04-27T03:50:30.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:30.639+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:30.639941+0000) 2024-04-27T03:50:31.097 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:30.983+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:31.098 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:30.983+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:31.098 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:30.983+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:31.098 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:30.984+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:31.098 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:30.838+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:30.839114+0000) 2024-04-27T03:50:31.098 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:31.091+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:31.099 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:31.092+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:31.099 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:31.092+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:31.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:31.092+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:31.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:31.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:31.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:31.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:31.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:31.289+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:31.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:31.290+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:31.849 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:31.552+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:31.553444+0000) 2024-04-27T03:50:31.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:31.644+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:31.645012+0000) 2024-04-27T03:50:32.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:31.954+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:32.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:31.954+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:32.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:31.954+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:32.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:31.955+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:32.136 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:31.843+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:31.844182+0000) 2024-04-27T03:50:32.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:32.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:32.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:32.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:32.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:32.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:32.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:32.046+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:32.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:32.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:32.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:32.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:32.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:32.332+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:32.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:32.332+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:32.854 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:32.498+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:32.498997+0000) 2024-04-27T03:50:32.854 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:50:32.501+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73741 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:50:32.855 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:32.497+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:32.498455+0000) 2024-04-27T03:50:32.855 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:32.556+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:32.557722+0000) 2024-04-27T03:50:32.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:32.498+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:32.498849+0000) 2024-04-27T03:50:32.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:32.649+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:32.650095+0000) 2024-04-27T03:50:33.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:32.987+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:33.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:32.987+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:33.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:32.987+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:33.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:32.988+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:33.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:32.848+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:32.849219+0000) 2024-04-27T03:50:33.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:33.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:33.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:33.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:33.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:33.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:33.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:33.033+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:33.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:33.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:33.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:33.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:33.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:33.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:33.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:33.369+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:33.857 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:33.653+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:33.654426+0000) 2024-04-27T03:50:33.857 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:33.559+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:33.559797+0000) 2024-04-27T03:50:34.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:34.025+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:34.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:34.025+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:34.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:34.025+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:34.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:34.026+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:34.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:33.851+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:33.852263+0000) 2024-04-27T03:50:34.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:34.067+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:34.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:34.067+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:34.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:34.067+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:34.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:34.067+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:34.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:34.360+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:34.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:34.360+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:34.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:34.360+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:34.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:34.361+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:34.861 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:34.561+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:34.561942+0000) 2024-04-27T03:50:34.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:34.655+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:34.656590+0000) 2024-04-27T03:50:35.116 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:34.855+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:34.856531+0000) 2024-04-27T03:50:35.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:35.044+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:35.117 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:35.044+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:35.117 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:35.044+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:35.117 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:35.045+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:35.386 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:35.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:35.386 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:35.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:35.386 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:35.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:35.386 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:35.111+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:35.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:35.380+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:35.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:35.380+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:35.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:35.380+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:35.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:35.381+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:35.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:35.563+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:35.564276+0000) 2024-04-27T03:50:35.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:35.858+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:35.859788+0000) 2024-04-27T03:50:35.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:35.658+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:35.658843+0000) 2024-04-27T03:50:36.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:36.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:36.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:36.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:36.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:36.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:36.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:36.092+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:36.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:36.093+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:36.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:36.093+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:36.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:36.093+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:36.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:36.094+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:36.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:36.408+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:36.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:36.408+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:36.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:36.408+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:36.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:36.409+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:36.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:36.567+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:36.568554+0000) 2024-04-27T03:50:36.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:36.861+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:36.861909+0000) 2024-04-27T03:50:36.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:36.662+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:36.663506+0000) 2024-04-27T03:50:37.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:37.076+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:37.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:37.076+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:37.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:37.076+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:37.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:37.077+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:37.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:37.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:37.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:37.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:37.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:37.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:37.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:37.077+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:37.576 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:50:37.510+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73781 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:50:37.577 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:37.506+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:37.507505+0000) 2024-04-27T03:50:37.577 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:37.507+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:37.508222+0000) 2024-04-27T03:50:37.577 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:37.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:37.577 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:37.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:37.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:37.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:37.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:37.401+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:37.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:37.507+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:37.508105+0000) 2024-04-27T03:50:37.871 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:37.665+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:37.666342+0000) 2024-04-27T03:50:37.871 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:37.570+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:37.571653+0000) 2024-04-27T03:50:38.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:38.083+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:38.137 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:38.083+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:38.137 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:38.083+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:38.137 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:38.084+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:38.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:37.865+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:37.866195+0000) 2024-04-27T03:50:38.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:38.086+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:38.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:38.086+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:38.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:38.086+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:38.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:38.086+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:38.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:38.365+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:38.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:38.365+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:38.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:38.365+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:38.476 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:38.365+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:38.873 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:38.573+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:38.574508+0000) 2024-04-27T03:50:38.873 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:38.668+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:38.669527+0000) 2024-04-27T03:50:39.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:39.057+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:39.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:39.057+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:39.137 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:39.057+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:39.137 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:39.058+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:39.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:38.867+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:38.868450+0000) 2024-04-27T03:50:39.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:39.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:39.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:39.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:39.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:39.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:39.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:39.062+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:39.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:39.391+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:39.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:39.391+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:39.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:39.391+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:39.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:39.391+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:39.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:39.576+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:39.577495+0000) 2024-04-27T03:50:39.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:39.670+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:39.671551+0000) 2024-04-27T03:50:39.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:39.871+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:39.872700+0000) 2024-04-27T03:50:40.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:40.054+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:40.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:40.054+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:40.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:40.054+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:40.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:40.055+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:40.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:40.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:40.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:40.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:40.326 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:40.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:40.326 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:40.086+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:40.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:40.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:40.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:40.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:40.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:40.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:40.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:40.435+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:40.881 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:40.578+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:40.579655+0000) 2024-04-27T03:50:40.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:40.675+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:40.675946+0000) 2024-04-27T03:50:41.136 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:41.101+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:41.137 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:41.101+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:41.137 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:41.101+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:41.137 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:41.103+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:41.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:40.876+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:40.876950+0000) 2024-04-27T03:50:41.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:41.082+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:41.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:41.082+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:41.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:41.082+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:41.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:41.083+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:41.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:41.423+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:41.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:41.423+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:41.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:41.423+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:41.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:41.425+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:41.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:41.581+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:41.582483+0000) 2024-04-27T03:50:41.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:41.878+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:41.879350+0000) 2024-04-27T03:50:41.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:41.677+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:41.678604+0000) 2024-04-27T03:50:42.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:42.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:42.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:42.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:42.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:42.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:42.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:42.099+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:42.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:42.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:42.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:42.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:42.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:42.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:42.326 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:42.078+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:42.591 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:50:42.519+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73821 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:50:42.592 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:42.515+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:42.516321+0000) 2024-04-27T03:50:42.592 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:42.514+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:42.515448+0000) 2024-04-27T03:50:42.592 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:42.438+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:42.592 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:42.438+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:42.593 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:42.438+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:42.593 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:42.439+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:42.593 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:42.519+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:42.519840+0000) 2024-04-27T03:50:42.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:42.585+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:42.586738+0000) 2024-04-27T03:50:42.886 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:42.680+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:42.681282+0000) 2024-04-27T03:50:43.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:42.880+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:42.881698+0000) 2024-04-27T03:50:43.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:43.095+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:43.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:43.095+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:43.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:43.095+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:43.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:43.096+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:43.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:43.071+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:43.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:43.071+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:43.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:43.071+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:43.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:43.071+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:43.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:43.682+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:43.683352+0000) 2024-04-27T03:50:43.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:43.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:43.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:43.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:43.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:43.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:43.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:43.464+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:43.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:43.588+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:43.589696+0000) 2024-04-27T03:50:44.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:43.883+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:43.884564+0000) 2024-04-27T03:50:44.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:44.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:44.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:44.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:44.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:44.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:44.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:44.062+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:44.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:44.040+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:44.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:44.040+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:44.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:44.040+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:44.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:44.041+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:44.892 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:44.684+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:44.685728+0000) 2024-04-27T03:50:44.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:44.508+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:44.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:44.508+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:44.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:44.508+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:44.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:44.509+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:44.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:44.593+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:44.594422+0000) 2024-04-27T03:50:45.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:44.886+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:44.887548+0000) 2024-04-27T03:50:45.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:45.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:45.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:45.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:45.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:45.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:45.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:45.054+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:45.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:45.046+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:45.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:45.046+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:45.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:45.046+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:45.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:45.047+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:45.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:45.687+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:45.687910+0000) 2024-04-27T03:50:45.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:45.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:45.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:45.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:45.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:45.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:45.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:45.555+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:45.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:45.595+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:45.596719+0000) 2024-04-27T03:50:46.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:46.011+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:46.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:46.011+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:46.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:46.011+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:46.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:46.012+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:46.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:45.889+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:45.889952+0000) 2024-04-27T03:50:46.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:46.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:46.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:46.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:46.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:46.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:46.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:46.058+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:46.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:46.690+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:46.691046+0000) 2024-04-27T03:50:46.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:46.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:46.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:46.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:46.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:46.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:46.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:46.536+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:46.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:46.598+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:46.598841+0000) 2024-04-27T03:50:47.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:47.039+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:47.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:47.039+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:47.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:47.039+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:47.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:47.039+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:47.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:46.890+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:46.892320+0000) 2024-04-27T03:50:47.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:47.019+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:47.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:47.019+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:47.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:47.019+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:47.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:47.019+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:47.899 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:50:47.528+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73861 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:50:47.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:47.523+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:47.525083+0000) 2024-04-27T03:50:47.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:47.523+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:47.525274+0000) 2024-04-27T03:50:47.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:47.691+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:47.693170+0000) 2024-04-27T03:50:47.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:47.523+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:47.525466+0000) 2024-04-27T03:50:47.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:47.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:47.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:47.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:47.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:47.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:47.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:47.526+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:47.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:47.599+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:47.600945+0000) 2024-04-27T03:50:48.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:47.893+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:47.895130+0000) 2024-04-27T03:50:48.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:48.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:48.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:48.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:48.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:48.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:48.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:48.049+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:48.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:47.999+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:48.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:47.999+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:48.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:47.999+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:48.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:48.001+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:48.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:48.694+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:48.695812+0000) 2024-04-27T03:50:48.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:48.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:48.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:48.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:48.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:48.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:48.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:48.558+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:48.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:48.601+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:48.603041+0000) 2024-04-27T03:50:49.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:49.025+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:49.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:49.025+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:49.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:49.025+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:49.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:49.026+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:49.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:48.896+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:48.898203+0000) 2024-04-27T03:50:49.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:49.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:49.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:49.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:49.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:49.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:49.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:49.052+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:49.795 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:49.696+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:49.698506+0000) 2024-04-27T03:50:49.795 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:49.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:49.796 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:49.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:49.796 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:49.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:49.796 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:49.520+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:49.796 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:49.603+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:49.605163+0000) 2024-04-27T03:50:50.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:49.900+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:49.902509+0000) 2024-04-27T03:50:50.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:50.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:50.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:50.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:50.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:50.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:50.056 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:50.024+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:50.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:49.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:50.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:49.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:50.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:49.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:50.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:49.978+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:50.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:50.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:50.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:50.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:50.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:50.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:50.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:50.488+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:50.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:50.607+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:50.609411+0000) 2024-04-27T03:50:50.909 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:50.700+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:50.701878+0000) 2024-04-27T03:50:50.992 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-27T03:50:50.993 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-04-27T03:50:50.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T03:50:50.994 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-04-27T03:50:51.025 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip: Cannot mkdir: Input/output error 2024-04-27T03:50:51.025 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip6-dstaddr: Cannot open: Input/output error 2024-04-27T03:50:51.025 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip6-nexthdr: Cannot open: Input/output error 2024-04-27T03:50:51.025 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip4-protcol: Cannot open: Input/output error 2024-04-27T03:50:51.025 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip4-checksum: Cannot open: Input/output error 2024-04-27T03:50:51.026 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip6-version: Cannot open: Input/output error 2024-04-27T03:50:51.026 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip4-ihl: Cannot open: Input/output error 2024-04-27T03:50:51.026 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.026 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip4-fragment: Cannot open: Input/output error 2024-04-27T03:50:51.026 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip4-totlen: Cannot open: Input/output error 2024-04-27T03:50:51.026 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.026 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip4-version: Cannot open: Input/output error 2024-04-27T03:50:51.026 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip6-plen: Cannot open: Input/output error 2024-04-27T03:50:51.033 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip4-dstaddr: Cannot open: Input/output error 2024-04-27T03:50:51.034 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast: Cannot mkdir: Input/output error 2024-04-27T03:50:51.034 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation: Cannot mkdir: Input/output error 2024-04-27T03:50:51.034 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast6-grpope01: Cannot open: Input/output error 2024-04-27T03:50:51.034 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast6-grpope02: Cannot open: Input/output error 2024-04-27T03:50:51.034 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast4-grpope03: Cannot open: Input/output error 2024-04-27T03:50:51.034 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.034 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.035 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast4-grpope04: Cannot open: Input/output error 2024-04-27T03:50:51.035 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast6-grpope03: Cannot open: Input/output error 2024-04-27T03:50:51.035 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast4-grpope01: Cannot open: Input/output error 2024-04-27T03:50:51.035 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast4-grpope02: Cannot open: Input/output error 2024-04-27T03:50:51.035 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast6-grpope04: Cannot open: Input/output error 2024-04-27T03:50:51.035 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood: Cannot mkdir: Input/output error 2024-04-27T03:50:51.035 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast4-queryfld01: Cannot open: Input/output error 2024-04-27T03:50:51.036 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast6-queryfld03: Cannot open: Input/output error 2024-04-27T03:50:51.036 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast4-queryfld05: Cannot open: Input/output error 2024-04-27T03:50:51.036 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast6-queryfld02: Cannot open: Input/output error 2024-04-27T03:50:51.036 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast6-queryfld05: Cannot open: Input/output error 2024-04-27T03:50:51.036 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast6-queryfld01: Cannot open: Input/output error 2024-04-27T03:50:51.036 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast4-queryfld02: Cannot open: Input/output error 2024-04-27T03:50:51.037 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.037 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.037 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast6-queryfld06: Cannot open: Input/output error 2024-04-27T03:50:51.037 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast4-queryfld03: Cannot open: Input/output error 2024-04-27T03:50:51.037 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast4-queryfld04: Cannot open: Input/output error 2024-04-27T03:50:51.037 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast4-queryfld06: Cannot open: Input/output error 2024-04-27T03:50:51.037 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast6-queryfld04: Cannot open: Input/output error 2024-04-27T03:50:51.037 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/packet-flood: Cannot mkdir: Input/output error 2024-04-27T03:50:51.038 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/packet-flood/mcast6-pktfld02: Cannot open: Input/output error 2024-04-27T03:50:51.038 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/packet-flood/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.038 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/packet-flood/mcast6-pktfld01: Cannot open: Input/output error 2024-04-27T03:50:51.038 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/packet-flood/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.038 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/packet-flood/mcast4-pktfld02: Cannot open: Input/output error 2024-04-27T03:50:51.038 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/packet-flood/mcast4-pktfld01: Cannot open: Input/output error 2024-04-27T03:50:51.038 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.038 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/route: Cannot mkdir: Input/output error 2024-04-27T03:50:51.038 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route4-redirect: Cannot open: Input/output error 2024-04-27T03:50:51.039 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route6-change-if: Cannot open: Input/output error 2024-04-27T03:50:51.039 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/route/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.039 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/route/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.039 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route4-change-gw: Cannot open: Input/output error 2024-04-27T03:50:51.039 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route6-ifdown: Cannot open: Input/output error 2024-04-27T03:50:51.039 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route4-change-if: Cannot open: Input/output error 2024-04-27T03:50:51.039 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route4-change-dst: Cannot open: Input/output error 2024-04-27T03:50:51.039 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route6-change-dst: Cannot open: Input/output error 2024-04-27T03:50:51.040 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route6-change-gw: Cannot open: Input/output error 2024-04-27T03:50:51.040 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route4-ifdown: Cannot open: Input/output error 2024-04-27T03:50:51.040 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route4-rmmod: Cannot open: Input/output error 2024-04-27T03:50:51.040 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route6-redirect: Cannot open: Input/output error 2024-04-27T03:50:51.040 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route6-rmmod: Cannot open: Input/output error 2024-04-27T03:50:51.040 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.040 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp: Cannot mkdir: Input/output error 2024-04-27T03:50:51.040 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff: Cannot mkdir: Input/output error 2024-04-27T03:50:51.040 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff14: Cannot open: Input/output error 2024-04-27T03:50:51.041 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff06: Cannot open: Input/output error 2024-04-27T03:50:51.041 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff09: Cannot open: Input/output error 2024-04-27T03:50:51.041 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff02: Cannot open: Input/output error 2024-04-27T03:50:51.041 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff13: Cannot open: Input/output error 2024-04-27T03:50:51.041 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff01: Cannot open: Input/output error 2024-04-27T03:50:51.041 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff09: Cannot open: Input/output error 2024-04-27T03:50:51.041 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff03: Cannot open: Input/output error 2024-04-27T03:50:51.041 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff04: Cannot open: Input/output error 2024-04-27T03:50:51.042 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff10: Cannot open: Input/output error 2024-04-27T03:50:51.042 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff05: Cannot open: Input/output error 2024-04-27T03:50:51.042 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff04: Cannot open: Input/output error 2024-04-27T03:50:51.042 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.042 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.042 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff07: Cannot open: Input/output error 2024-04-27T03:50:51.042 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff11: Cannot open: Input/output error 2024-04-27T03:50:51.042 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff12: Cannot open: Input/output error 2024-04-27T03:50:51.043 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff05: Cannot open: Input/output error 2024-04-27T03:50:51.043 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff10: Cannot open: Input/output error 2024-04-27T03:50:51.043 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff07: Cannot open: Input/output error 2024-04-27T03:50:51.043 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff11: Cannot open: Input/output error 2024-04-27T03:50:51.043 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff12: Cannot open: Input/output error 2024-04-27T03:50:51.043 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff13: Cannot open: Input/output error 2024-04-27T03:50:51.043 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff08: Cannot open: Input/output error 2024-04-27T03:50:51.043 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff14: Cannot open: Input/output error 2024-04-27T03:50:51.043 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff03: Cannot open: Input/output error 2024-04-27T03:50:51.044 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff02: Cannot open: Input/output error 2024-04-27T03:50:51.044 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff08: Cannot open: Input/output error 2024-04-27T03:50:51.044 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff06: Cannot open: Input/output error 2024-04-27T03:50:51.044 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff01: Cannot open: Input/output error 2024-04-27T03:50:51.044 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport: Cannot mkdir: Input/output error 2024-04-27T03:50:51.044 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport04: Cannot open: Input/output error 2024-04-27T03:50:51.044 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport03: Cannot open: Input/output error 2024-04-27T03:50:51.045 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport12: Cannot open: Input/output error 2024-04-27T03:50:51.087 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport07: Cannot open: Input/output error 2024-04-27T03:50:51.087 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport05: Cannot open: Input/output error 2024-04-27T03:50:51.087 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport04: Cannot open: Input/output error 2024-04-27T03:50:51.087 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport14: Cannot open: Input/output error 2024-04-27T03:50:51.088 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport10: Cannot open: Input/output error 2024-04-27T03:50:51.088 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport02: Cannot open: Input/output error 2024-04-27T03:50:51.088 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport05: Cannot open: Input/output error 2024-04-27T03:50:51.088 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.088 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.088 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport06: Cannot open: Input/output error 2024-04-27T03:50:51.088 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport01: Cannot open: Input/output error 2024-04-27T03:50:51.088 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport09: Cannot open: Input/output error 2024-04-27T03:50:51.089 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport13: Cannot open: Input/output error 2024-04-27T03:50:51.089 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport10: Cannot open: Input/output error 2024-04-27T03:50:51.089 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport07: Cannot open: Input/output error 2024-04-27T03:50:51.089 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport02: Cannot open: Input/output error 2024-04-27T03:50:51.089 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport08: Cannot open: Input/output error 2024-04-27T03:50:51.089 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport12: Cannot open: Input/output error 2024-04-27T03:50:51.090 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport01: Cannot open: Input/output error 2024-04-27T03:50:51.090 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport08: Cannot open: Input/output error 2024-04-27T03:50:51.091 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2024-04-27T03:50:51.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T03:50:51.091 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-27T03:50:51.132 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-27T03:50:51.132 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport03: Cannot open: Input/output error 2024-04-27T03:50:51.132 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport11: Cannot open: Input/output error 2024-04-27T03:50:51.133 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport13: Cannot open: Input/output error 2024-04-27T03:50:51.133 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport11: Cannot open: Input/output error 2024-04-27T03:50:51.133 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport14: Cannot open: Input/output error 2024-04-27T03:50:51.133 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport06: Cannot open: Input/output error 2024-04-27T03:50:51.133 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport09: Cannot open: Input/output error 2024-04-27T03:50:51.133 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic: Cannot mkdir: Input/output error 2024-04-27T03:50:51.133 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic04: Cannot open: Input/output error 2024-04-27T03:50:51.133 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic06: Cannot open: Input/output error 2024-04-27T03:50:51.134 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic13: Cannot open: Input/output error 2024-04-27T03:50:51.134 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic14: Cannot open: Input/output error 2024-04-27T03:50:51.134 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic06: Cannot open: Input/output error 2024-04-27T03:50:51.134 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic11: Cannot open: Input/output error 2024-04-27T03:50:51.134 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic05: Cannot open: Input/output error 2024-04-27T03:50:51.134 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.134 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic01: Cannot open: Input/output error 2024-04-27T03:50:51.135 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic12: Cannot open: Input/output error 2024-04-27T03:50:51.135 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic11: Cannot open: Input/output error 2024-04-27T03:50:51.135 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.135 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic10: Cannot open: Input/output error 2024-04-27T03:50:51.135 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic07: Cannot open: Input/output error 2024-04-27T03:50:51.135 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic10: Cannot open: Input/output error 2024-04-27T03:50:51.135 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic08: Cannot open: Input/output error 2024-04-27T03:50:51.135 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic13: Cannot open: Input/output error 2024-04-27T03:50:51.135 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic03: Cannot open: Input/output error 2024-04-27T03:50:51.136 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic08: Cannot open: Input/output error 2024-04-27T03:50:51.136 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic02: Cannot open: Input/output error 2024-04-27T03:50:51.136 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic09: Cannot open: Input/output error 2024-04-27T03:50:51.136 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic03: Cannot open: Input/output error 2024-04-27T03:50:51.136 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic01: Cannot open: Input/output error 2024-04-27T03:50:51.136 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic04: Cannot open: Input/output error 2024-04-27T03:50:51.136 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic07: Cannot open: Input/output error 2024-04-27T03:50:51.136 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic09: Cannot open: Input/output error 2024-04-27T03:50:51.137 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic02: Cannot open: Input/output error 2024-04-27T03:50:51.137 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic12: Cannot open: Input/output error 2024-04-27T03:50:51.137 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic14: Cannot open: Input/output error 2024-04-27T03:50:51.137 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic05: Cannot open: Input/output error 2024-04-27T03:50:51.137 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport: Cannot mkdir: Input/output error 2024-04-27T03:50:51.137 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport07: Cannot open: Input/output error 2024-04-27T03:50:51.137 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport14: Cannot open: Input/output error 2024-04-27T03:50:51.137 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport11: Cannot open: Input/output error 2024-04-27T03:50:51.138 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport06: Cannot open: Input/output error 2024-04-27T03:50:51.138 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport09: Cannot open: Input/output error 2024-04-27T03:50:51.138 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport02: Cannot open: Input/output error 2024-04-27T03:50:51.138 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport04: Cannot open: Input/output error 2024-04-27T03:50:51.138 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport01: Cannot open: Input/output error 2024-04-27T03:50:51.138 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.138 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport08: Cannot open: Input/output error 2024-04-27T03:50:51.138 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport12: Cannot open: Input/output error 2024-04-27T03:50:51.139 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.139 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport09: Cannot open: Input/output error 2024-04-27T03:50:51.139 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport05: Cannot open: Input/output error 2024-04-27T03:50:51.139 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport13: Cannot open: Input/output error 2024-04-27T03:50:51.139 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport05: Cannot open: Input/output error 2024-04-27T03:50:51.139 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport08: Cannot open: Input/output error 2024-04-27T03:50:51.139 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport02: Cannot open: Input/output error 2024-04-27T03:50:51.139 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport14: Cannot open: Input/output error 2024-04-27T03:50:51.140 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport12: Cannot open: Input/output error 2024-04-27T03:50:51.140 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport11: Cannot open: Input/output error 2024-04-27T03:50:51.140 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport10: Cannot open: Input/output error 2024-04-27T03:50:51.140 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport03: Cannot open: Input/output error 2024-04-27T03:50:51.140 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport04: Cannot open: Input/output error 2024-04-27T03:50:51.140 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport13: Cannot open: Input/output error 2024-04-27T03:50:51.140 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport06: Cannot open: Input/output error 2024-04-27T03:50:51.140 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport07: Cannot open: Input/output error 2024-04-27T03:50:51.140 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport10: Cannot open: Input/output error 2024-04-27T03:50:51.141 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport01: Cannot open: Input/output error 2024-04-27T03:50:51.141 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport03: Cannot open: Input/output error 2024-04-27T03:50:51.141 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.141 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend: Cannot mkdir: Input/output error 2024-04-27T03:50:51.141 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend12: Cannot open: Input/output error 2024-04-27T03:50:51.141 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend09: Cannot open: Input/output error 2024-04-27T03:50:51.141 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend03: Cannot open: Input/output error 2024-04-27T03:50:51.141 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend06: Cannot open: Input/output error 2024-04-27T03:50:51.142 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend11: Cannot open: Input/output error 2024-04-27T03:50:51.142 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend06: Cannot open: Input/output error 2024-04-27T03:50:51.142 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend07: Cannot open: Input/output error 2024-04-27T03:50:51.142 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend04: Cannot open: Input/output error 2024-04-27T03:50:51.142 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend11: Cannot open: Input/output error 2024-04-27T03:50:51.142 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend01: Cannot open: Input/output error 2024-04-27T03:50:51.142 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.142 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend10: Cannot open: Input/output error 2024-04-27T03:50:51.143 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend02: Cannot open: Input/output error 2024-04-27T03:50:51.143 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.143 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend04: Cannot open: Input/output error 2024-04-27T03:50:51.143 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend14: Cannot open: Input/output error 2024-04-27T03:50:51.143 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend13: Cannot open: Input/output error 2024-04-27T03:50:51.143 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend12: Cannot open: Input/output error 2024-04-27T03:50:51.143 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend01: Cannot open: Input/output error 2024-04-27T03:50:51.143 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend05: Cannot open: Input/output error 2024-04-27T03:50:51.144 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend14: Cannot open: Input/output error 2024-04-27T03:50:51.144 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend02: Cannot open: Input/output error 2024-04-27T03:50:51.144 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend07: Cannot open: Input/output error 2024-04-27T03:50:51.144 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend08: Cannot open: Input/output error 2024-04-27T03:50:51.144 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend03: Cannot open: Input/output error 2024-04-27T03:50:51.144 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend05: Cannot open: Input/output error 2024-04-27T03:50:51.144 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend10: Cannot open: Input/output error 2024-04-27T03:50:51.144 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend08: Cannot open: Input/output error 2024-04-27T03:50:51.145 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend13: Cannot open: Input/output error 2024-04-27T03:50:51.145 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend09: Cannot open: Input/output error 2024-04-27T03:50:51.145 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic: Cannot mkdir: Input/output error 2024-04-27T03:50:51.145 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic04: Cannot open: Input/output error 2024-04-27T03:50:51.145 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic05: Cannot open: Input/output error 2024-04-27T03:50:51.145 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic09: Cannot open: Input/output error 2024-04-27T03:50:51.145 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic01: Cannot open: Input/output error 2024-04-27T03:50:51.145 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic02: Cannot open: Input/output error 2024-04-27T03:50:51.146 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic07: Cannot open: Input/output error 2024-04-27T03:50:51.146 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic03: Cannot open: Input/output error 2024-04-27T03:50:51.146 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic03: Cannot open: Input/output error 2024-04-27T03:50:51.146 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic04: Cannot open: Input/output error 2024-04-27T03:50:51.146 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.146 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic01: Cannot open: Input/output error 2024-04-27T03:50:51.146 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic05: Cannot open: Input/output error 2024-04-27T03:50:51.146 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic14: Cannot open: Input/output error 2024-04-27T03:50:51.146 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.147 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic11: Cannot open: Input/output error 2024-04-27T03:50:51.147 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic12: Cannot open: Input/output error 2024-04-27T03:50:51.147 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic06: Cannot open: Input/output error 2024-04-27T03:50:51.147 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic13: Cannot open: Input/output error 2024-04-27T03:50:51.147 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic08: Cannot open: Input/output error 2024-04-27T03:50:51.147 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic02: Cannot open: Input/output error 2024-04-27T03:50:51.147 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic14: Cannot open: Input/output error 2024-04-27T03:50:51.147 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic06: Cannot open: Input/output error 2024-04-27T03:50:51.148 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic11: Cannot open: Input/output error 2024-04-27T03:50:51.148 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic09: Cannot open: Input/output error 2024-04-27T03:50:51.148 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic07: Cannot open: Input/output error 2024-04-27T03:50:51.148 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic10: Cannot open: Input/output error 2024-04-27T03:50:51.148 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic13: Cannot open: Input/output error 2024-04-27T03:50:51.148 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic08: Cannot open: Input/output error 2024-04-27T03:50:51.148 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic12: Cannot open: Input/output error 2024-04-27T03:50:51.148 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic10: Cannot open: Input/output error 2024-04-27T03:50:51.148 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup: Cannot mkdir: Input/output error 2024-04-27T03:50:51.149 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup09: Cannot open: Input/output error 2024-04-27T03:50:51.149 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup06: Cannot open: Input/output error 2024-04-27T03:50:51.149 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup13: Cannot open: Input/output error 2024-04-27T03:50:51.149 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup10: Cannot open: Input/output error 2024-04-27T03:50:51.149 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup09: Cannot open: Input/output error 2024-04-27T03:50:51.149 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup03: Cannot open: Input/output error 2024-04-27T03:50:51.149 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup01: Cannot open: Input/output error 2024-04-27T03:50:51.150 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup03: Cannot open: Input/output error 2024-04-27T03:50:51.150 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup02: Cannot open: Input/output error 2024-04-27T03:50:51.150 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup11: Cannot open: Input/output error 2024-04-27T03:50:51.150 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup01: Cannot open: Input/output error 2024-04-27T03:50:51.150 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup10: Cannot open: Input/output error 2024-04-27T03:50:51.150 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup14: Cannot open: Input/output error 2024-04-27T03:50:51.150 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.150 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup08: Cannot open: Input/output error 2024-04-27T03:50:51.150 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.151 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup05: Cannot open: Input/output error 2024-04-27T03:50:51.151 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup04: Cannot open: Input/output error 2024-04-27T03:50:51.151 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup13: Cannot open: Input/output error 2024-04-27T03:50:51.151 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup07: Cannot open: Input/output error 2024-04-27T03:50:51.151 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup06: Cannot open: Input/output error 2024-04-27T03:50:51.151 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup12: Cannot open: Input/output error 2024-04-27T03:50:51.151 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup04: Cannot open: Input/output error 2024-04-27T03:50:51.151 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup05: Cannot open: Input/output error 2024-04-27T03:50:51.152 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup12: Cannot open: Input/output error 2024-04-27T03:50:51.152 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup11: Cannot open: Input/output error 2024-04-27T03:50:51.152 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup02: Cannot open: Input/output error 2024-04-27T03:50:51.152 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup14: Cannot open: Input/output error 2024-04-27T03:50:51.152 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup08: Cannot open: Input/output error 2024-04-27T03:50:51.152 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup07: Cannot open: Input/output error 2024-04-27T03:50:51.152 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff: Cannot mkdir: Input/output error 2024-04-27T03:50:51.152 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff14: Cannot open: Input/output error 2024-04-27T03:50:51.153 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff02: Cannot open: Input/output error 2024-04-27T03:50:51.153 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff06: Cannot open: Input/output error 2024-04-27T03:50:51.153 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff02: Cannot open: Input/output error 2024-04-27T03:50:51.153 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff06: Cannot open: Input/output error 2024-04-27T03:50:51.153 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff09: Cannot open: Input/output error 2024-04-27T03:50:51.153 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff08: Cannot open: Input/output error 2024-04-27T03:50:51.153 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff11: Cannot open: Input/output error 2024-04-27T03:50:51.153 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff10: Cannot open: Input/output error 2024-04-27T03:50:51.153 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff11: Cannot open: Input/output error 2024-04-27T03:50:51.154 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff09: Cannot open: Input/output error 2024-04-27T03:50:51.154 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff10: Cannot open: Input/output error 2024-04-27T03:50:51.154 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.154 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.154 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff13: Cannot open: Input/output error 2024-04-27T03:50:51.154 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff05: Cannot open: Input/output error 2024-04-27T03:50:51.154 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff07: Cannot open: Input/output error 2024-04-27T03:50:51.154 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff13: Cannot open: Input/output error 2024-04-27T03:50:51.155 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff01: Cannot open: Input/output error 2024-04-27T03:50:51.155 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff14: Cannot open: Input/output error 2024-04-27T03:50:51.155 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff01: Cannot open: Input/output error 2024-04-27T03:50:51.155 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff12: Cannot open: Input/output error 2024-04-27T03:50:51.155 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff03: Cannot open: Input/output error 2024-04-27T03:50:51.155 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff04: Cannot open: Input/output error 2024-04-27T03:50:51.155 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff05: Cannot open: Input/output error 2024-04-27T03:50:51.155 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff03: Cannot open: Input/output error 2024-04-27T03:50:51.156 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff07: Cannot open: Input/output error 2024-04-27T03:50:51.156 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff08: Cannot open: Input/output error 2024-04-27T03:50:51.156 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff12: Cannot open: Input/output error 2024-04-27T03:50:51.156 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff04: Cannot open: Input/output error 2024-04-27T03:50:51.156 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip: Cannot mkdir: Input/output error 2024-04-27T03:50:51.156 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip09: Cannot open: Input/output error 2024-04-27T03:50:51.156 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip08: Cannot open: Input/output error 2024-04-27T03:50:51.156 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip06: Cannot open: Input/output error 2024-04-27T03:50:51.156 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip02: Cannot open: Input/output error 2024-04-27T03:50:51.157 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip01: Cannot open: Input/output error 2024-04-27T03:50:51.157 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip04: Cannot open: Input/output error 2024-04-27T03:50:51.157 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip12: Cannot open: Input/output error 2024-04-27T03:50:51.157 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip03: Cannot open: Input/output error 2024-04-27T03:50:51.157 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip03: Cannot open: Input/output error 2024-04-27T03:50:51.157 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip13: Cannot open: Input/output error 2024-04-27T03:50:51.157 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip14: Cannot open: Input/output error 2024-04-27T03:50:51.157 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.157 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip02: Cannot open: Input/output error 2024-04-27T03:50:51.158 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip11: Cannot open: Input/output error 2024-04-27T03:50:51.158 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip04: Cannot open: Input/output error 2024-04-27T03:50:51.158 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.158 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip06: Cannot open: Input/output error 2024-04-27T03:50:51.158 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip10: Cannot open: Input/output error 2024-04-27T03:50:51.158 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip12: Cannot open: Input/output error 2024-04-27T03:50:51.158 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip09: Cannot open: Input/output error 2024-04-27T03:50:51.158 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip14: Cannot open: Input/output error 2024-04-27T03:50:51.159 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip07: Cannot open: Input/output error 2024-04-27T03:50:51.159 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip08: Cannot open: Input/output error 2024-04-27T03:50:51.159 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip13: Cannot open: Input/output error 2024-04-27T03:50:51.159 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip05: Cannot open: Input/output error 2024-04-27T03:50:51.159 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip10: Cannot open: Input/output error 2024-04-27T03:50:51.159 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip07: Cannot open: Input/output error 2024-04-27T03:50:51.159 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip01: Cannot open: Input/output error 2024-04-27T03:50:51.159 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip11: Cannot open: Input/output error 2024-04-27T03:50:51.159 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip05: Cannot open: Input/output error 2024-04-27T03:50:51.160 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso: Cannot mkdir: Input/output error 2024-04-27T03:50:51.160 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso11: Cannot open: Input/output error 2024-04-27T03:50:51.160 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso07: Cannot open: Input/output error 2024-04-27T03:50:51.160 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso14: Cannot open: Input/output error 2024-04-27T03:50:51.160 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso05: Cannot open: Input/output error 2024-04-27T03:50:51.160 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso08: Cannot open: Input/output error 2024-04-27T03:50:51.160 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso04: Cannot open: Input/output error 2024-04-27T03:50:51.160 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso10: Cannot open: Input/output error 2024-04-27T03:50:51.160 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso06: Cannot open: Input/output error 2024-04-27T03:50:51.161 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso08: Cannot open: Input/output error 2024-04-27T03:50:51.161 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso12: Cannot open: Input/output error 2024-04-27T03:50:51.161 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.161 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso03: Cannot open: Input/output error 2024-04-27T03:50:51.161 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso01: Cannot open: Input/output error 2024-04-27T03:50:51.161 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.161 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso06: Cannot open: Input/output error 2024-04-27T03:50:51.161 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso13: Cannot open: Input/output error 2024-04-27T03:50:51.161 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso07: Cannot open: Input/output error 2024-04-27T03:50:51.162 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso02: Cannot open: Input/output error 2024-04-27T03:50:51.162 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso04: Cannot open: Input/output error 2024-04-27T03:50:51.162 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso12: Cannot open: Input/output error 2024-04-27T03:50:51.162 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso01: Cannot open: Input/output error 2024-04-27T03:50:51.162 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso10: Cannot open: Input/output error 2024-04-27T03:50:51.162 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso09: Cannot open: Input/output error 2024-04-27T03:50:51.162 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso14: Cannot open: Input/output error 2024-04-27T03:50:51.162 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso02: Cannot open: Input/output error 2024-04-27T03:50:51.163 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso09: Cannot open: Input/output error 2024-04-27T03:50:51.163 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso11: Cannot open: Input/output error 2024-04-27T03:50:51.163 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso03: Cannot open: Input/output error 2024-04-27T03:50:51.163 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso05: Cannot open: Input/output error 2024-04-27T03:50:51.163 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso13: Cannot open: Input/output error 2024-04-27T03:50:51.163 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale: Cannot mkdir: Input/output error 2024-04-27T03:50:51.163 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale07: Cannot open: Input/output error 2024-04-27T03:50:51.163 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale01: Cannot open: Input/output error 2024-04-27T03:50:51.163 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale13: Cannot open: Input/output error 2024-04-27T03:50:51.164 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale06: Cannot open: Input/output error 2024-04-27T03:50:51.164 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale12: Cannot open: Input/output error 2024-04-27T03:50:51.164 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale05: Cannot open: Input/output error 2024-04-27T03:50:51.164 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale11: Cannot open: Input/output error 2024-04-27T03:50:51.164 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale02: Cannot open: Input/output error 2024-04-27T03:50:51.164 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale02: Cannot open: Input/output error 2024-04-27T03:50:51.164 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale07: Cannot open: Input/output error 2024-04-27T03:50:51.164 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale01: Cannot open: Input/output error 2024-04-27T03:50:51.164 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale11: Cannot open: Input/output error 2024-04-27T03:50:51.165 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale14: Cannot open: Input/output error 2024-04-27T03:50:51.165 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.165 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.165 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale13: Cannot open: Input/output error 2024-04-27T03:50:51.165 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale08: Cannot open: Input/output error 2024-04-27T03:50:51.165 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale06: Cannot open: Input/output error 2024-04-27T03:50:51.165 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale10: Cannot open: Input/output error 2024-04-27T03:50:51.165 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale05: Cannot open: Input/output error 2024-04-27T03:50:51.165 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale04: Cannot open: Input/output error 2024-04-27T03:50:51.166 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale10: Cannot open: Input/output error 2024-04-27T03:50:51.166 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale09: Cannot open: Input/output error 2024-04-27T03:50:51.166 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale09: Cannot open: Input/output error 2024-04-27T03:50:51.166 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale03: Cannot open: Input/output error 2024-04-27T03:50:51.166 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale03: Cannot open: Input/output error 2024-04-27T03:50:51.166 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale04: Cannot open: Input/output error 2024-04-27T03:50:51.166 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale08: Cannot open: Input/output error 2024-04-27T03:50:51.166 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale12: Cannot open: Input/output error 2024-04-27T03:50:51.167 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale14: Cannot open: Input/output error 2024-04-27T03:50:51.167 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/dns: Cannot mkdir: Input/output error 2024-04-27T03:50:51.167 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/dns/dns6-stress: Cannot open: Input/output error 2024-04-27T03:50:51.167 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/dns/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.167 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/dns/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.167 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/dns/dns4-stress: Cannot open: Input/output error 2024-04-27T03:50:51.167 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/dns/dns-stress01-rmt: Cannot open: Input/output error 2024-04-27T03:50:51.167 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/dns/dns-stress02-rmt: Cannot open: Input/output error 2024-04-27T03:50:51.167 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp: Cannot mkdir: Input/output error 2024-04-27T03:50:51.168 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp6-download-stress: Cannot open: Input/output error 2024-04-27T03:50:51.168 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp-upload-stress01-rmt: Cannot open: Input/output error 2024-04-27T03:50:51.168 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp-download-stress02-rmt: Cannot open: Input/output error 2024-04-27T03:50:51.168 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.168 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp4-upload-stress: Cannot open: Input/output error 2024-04-27T03:50:51.168 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.168 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp-upload-stress02-rmt: Cannot open: Input/output error 2024-04-27T03:50:51.168 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp6-upload-stress: Cannot open: Input/output error 2024-04-27T03:50:51.169 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp-download-stress01-rmt: Cannot open: Input/output error 2024-04-27T03:50:51.169 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp4-download-stress: Cannot open: Input/output error 2024-04-27T03:50:51.169 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/README: Cannot open: Input/output error 2024-04-27T03:50:51.169 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/interface: Cannot mkdir: Input/output error 2024-04-27T03:50:51.169 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if4-updown: Cannot open: Input/output error 2024-04-27T03:50:51.169 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.169 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.169 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if6-updown: Cannot open: Input/output error 2024-04-27T03:50:51.169 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if4-route-addlarge: Cannot open: Input/output error 2024-04-27T03:50:51.170 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if4-alias-adddel: Cannot open: Input/output error 2024-04-27T03:50:51.170 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if4-addr-change: Cannot open: Input/output error 2024-04-27T03:50:51.170 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if4-alias-addlarge: Cannot open: Input/output error 2024-04-27T03:50:51.170 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if6-addr-addlarge: Cannot open: Input/output error 2024-04-27T03:50:51.170 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if4-mtu-change: Cannot open: Input/output error 2024-04-27T03:50:51.170 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if6-route-addlarge: Cannot open: Input/output error 2024-04-27T03:50:51.170 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if6-mtu-change: Cannot open: Input/output error 2024-04-27T03:50:51.171 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if6-addr-adddel: Cannot open: Input/output error 2024-04-27T03:50:51.171 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if6-route-adddel: Cannot open: Input/output error 2024-04-27T03:50:51.171 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if4-route-adddel: Cannot open: Input/output error 2024-04-27T03:50:51.171 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp: Cannot mkdir: Input/output error 2024-04-27T03:50:51.171 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport: Cannot mkdir: Input/output error 2024-04-27T03:50:51.171 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp6-multi-diffport03: Cannot open: Input/output error 2024-04-27T03:50:51.171 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp6-multi-diffport05: Cannot open: Input/output error 2024-04-27T03:50:51.171 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp4-multi-diffport02: Cannot open: Input/output error 2024-04-27T03:50:51.172 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp4-multi-diffport05: Cannot open: Input/output error 2024-04-27T03:50:51.172 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.172 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp4-multi-diffport01: Cannot open: Input/output error 2024-04-27T03:50:51.172 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp6-multi-diffport07: Cannot open: Input/output error 2024-04-27T03:50:51.172 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.172 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp6-multi-diffport06: Cannot open: Input/output error 2024-04-27T03:50:51.172 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp6-multi-diffport02: Cannot open: Input/output error 2024-04-27T03:50:51.172 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp4-multi-diffport06: Cannot open: Input/output error 2024-04-27T03:50:51.172 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp4-multi-diffport03: Cannot open: Input/output error 2024-04-27T03:50:51.173 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp6-multi-diffport04: Cannot open: Input/output error 2024-04-27T03:50:51.173 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp4-multi-diffport04: Cannot open: Input/output error 2024-04-27T03:50:51.173 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp6-multi-diffport01: Cannot open: Input/output error 2024-04-27T03:50:51.173 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp4-multi-diffport07: Cannot open: Input/output error 2024-04-27T03:50:51.173 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic: Cannot mkdir: Input/output error 2024-04-27T03:50:51.173 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp4-uni-basic05: Cannot open: Input/output error 2024-04-27T03:50:51.173 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp4-uni-basic04: Cannot open: Input/output error 2024-04-27T03:50:51.173 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp6-uni-basic06: Cannot open: Input/output error 2024-04-27T03:50:51.174 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp6-uni-basic03: Cannot open: Input/output error 2024-04-27T03:50:51.174 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.174 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.174 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp6-uni-basic02: Cannot open: Input/output error 2024-04-27T03:50:51.174 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp4-uni-basic06: Cannot open: Input/output error 2024-04-27T03:50:51.174 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp6-uni-basic01: Cannot open: Input/output error 2024-04-27T03:50:51.174 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp4-uni-basic07: Cannot open: Input/output error 2024-04-27T03:50:51.174 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp6-uni-basic07: Cannot open: Input/output error 2024-04-27T03:50:51.174 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp6-uni-basic05: Cannot open: Input/output error 2024-04-27T03:50:51.175 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp4-uni-basic02: Cannot open: Input/output error 2024-04-27T03:50:51.175 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp4-uni-basic03: Cannot open: Input/output error 2024-04-27T03:50:51.175 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp4-uni-basic01: Cannot open: Input/output error 2024-04-27T03:50:51.175 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp6-uni-basic04: Cannot open: Input/output error 2024-04-27T03:50:51.175 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.175 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic: Cannot mkdir: Input/output error 2024-04-27T03:50:51.175 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp4-multi-diffnic01: Cannot open: Input/output error 2024-04-27T03:50:51.175 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp4-multi-diffnic02: Cannot open: Input/output error 2024-04-27T03:50:51.176 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp4-multi-diffnic06: Cannot open: Input/output error 2024-04-27T03:50:51.176 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp4-multi-diffnic04: Cannot open: Input/output error 2024-04-27T03:50:51.176 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.176 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp4-multi-diffnic07: Cannot open: Input/output error 2024-04-27T03:50:51.176 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp6-multi-diffnic01: Cannot open: Input/output error 2024-04-27T03:50:51.176 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.176 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp4-multi-diffnic03: Cannot open: Input/output error 2024-04-27T03:50:51.176 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp6-multi-diffnic05: Cannot open: Input/output error 2024-04-27T03:50:51.176 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp6-multi-diffnic02: Cannot open: Input/output error 2024-04-27T03:50:51.177 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp6-multi-diffnic04: Cannot open: Input/output error 2024-04-27T03:50:51.177 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp6-multi-diffnic03: Cannot open: Input/output error 2024-04-27T03:50:51.177 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp6-multi-diffnic06: Cannot open: Input/output error 2024-04-27T03:50:51.177 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp4-multi-diffnic05: Cannot open: Input/output error 2024-04-27T03:50:51.177 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp6-multi-diffnic07: Cannot open: Input/output error 2024-04-27T03:50:51.177 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip: Cannot mkdir: Input/output error 2024-04-27T03:50:51.177 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp6-multi-diffip02: Cannot open: Input/output error 2024-04-27T03:50:51.177 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.177 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.178 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp6-multi-diffip06: Cannot open: Input/output error 2024-04-27T03:50:51.178 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp6-multi-diffip04: Cannot open: Input/output error 2024-04-27T03:50:51.178 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp6-multi-diffip05: Cannot open: Input/output error 2024-04-27T03:50:51.178 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp4-multi-diffip06: Cannot open: Input/output error 2024-04-27T03:50:51.178 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp4-multi-diffip07: Cannot open: Input/output error 2024-04-27T03:50:51.178 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp6-multi-diffip01: Cannot open: Input/output error 2024-04-27T03:50:51.178 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp4-multi-diffip03: Cannot open: Input/output error 2024-04-27T03:50:51.178 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp4-multi-diffip02: Cannot open: Input/output error 2024-04-27T03:50:51.179 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp4-multi-diffip01: Cannot open: Input/output error 2024-04-27T03:50:51.179 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp4-multi-diffip05: Cannot open: Input/output error 2024-04-27T03:50:51.179 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp6-multi-diffip03: Cannot open: Input/output error 2024-04-27T03:50:51.179 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp6-multi-diffip07: Cannot open: Input/output error 2024-04-27T03:50:51.179 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp4-multi-diffip04: Cannot open: Input/output error 2024-04-27T03:50:51.179 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/http: Cannot mkdir: Input/output error 2024-04-27T03:50:51.179 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/http/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.180 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/http/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.180 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/http/http-stress01-rmt: Cannot open: Input/output error 2024-04-27T03:50:51.180 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/http/http4-stress: Cannot open: Input/output error 2024-04-27T03:50:51.180 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/http/http-stress02-rmt: Cannot open: Input/output error 2024-04-27T03:50:51.180 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/http/http6-stress: Cannot open: Input/output error 2024-04-27T03:50:51.180 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh: Cannot mkdir: Input/output error 2024-04-27T03:50:51.180 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.180 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh/ssh-stress02-rmt: Cannot open: Input/output error 2024-04-27T03:50:51.181 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh/ssh6-stress: Cannot open: Input/output error 2024-04-27T03:50:51.181 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh/ssh4-stress: Cannot open: Input/output error 2024-04-27T03:50:51.181 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh/00_Descriptions.txt: Cannot open: Input/output error 2024-04-27T03:50:51.181 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh/ssh-stress03-rmt: Cannot open: Input/output error 2024-04-27T03:50:51.181 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh/ssh-stress01-rmt: Cannot open: Input/output error 2024-04-27T03:50:51.181 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/stress: Cannot utime: Input/output error 2024-04-27T03:50:51.181 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4: Cannot mkdir: Input/output error 2024-04-27T03:50:51.181 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl: Cannot mkdir: Input/output error 2024-04-27T03:50:51.181 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/random_gen.py: Cannot open: Input/output error 2024-04-27T03:50:51.182 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/runtest: Cannot open: Input/output error 2024-04-27T03:50:51.182 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/setacl_stress.py: Cannot open: Input/output error 2024-04-27T03:50:51.182 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/test_long_acl.py: Cannot open: Input/output error 2024-04-27T03:50:51.182 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.182 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/cleanusers.py: Cannot open: Input/output error 2024-04-27T03:50:51.182 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/README: Cannot open: Input/output error 2024-04-27T03:50:51.182 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/create_users.py: Cannot open: Input/output error 2024-04-27T03:50:51.182 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/test_acl.py: Cannot open: Input/output error 2024-04-27T03:50:51.183 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/acl1.c: Cannot open: Input/output error 2024-04-27T03:50:51.183 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/cleangroups.py: Cannot open: Input/output error 2024-04-27T03:50:51.183 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.183 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks: Cannot mkdir: Input/output error 2024-04-27T03:50:51.183 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/netsync.c: Cannot open: Input/output error 2024-04-27T03:50:51.183 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/deploy_info: Cannot open: Input/output error 2024-04-27T03:50:51.183 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/locktests.h: Cannot open: Input/output error 2024-04-27T03:50:51.183 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.184 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/locktests.py: Cannot open: Input/output error 2024-04-27T03:50:51.184 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/locktests.c: Cannot open: Input/output error 2024-04-27T03:50:51.184 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/README: Cannot open: Input/output error 2024-04-27T03:50:51.184 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/deploy: Cannot mkdir: Input/output error 2024-04-27T03:50:51.184 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/deploy/locktests.tar.gz: Cannot open: Input/output error 2024-04-27T03:50:51.184 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/VERSION: Cannot open: Input/output error 2024-04-27T03:50:51.184 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/iproute: Cannot mkdir: Input/output error 2024-04-27T03:50:51.184 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/iproute/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.185 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/iproute/ip_tests.sh: Cannot open: Input/output error 2024-04-27T03:50:51.185 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/iptables: Cannot mkdir: Input/output error 2024-04-27T03:50:51.185 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/iptables/iptables_tests.sh: Cannot open: Input/output error 2024-04-27T03:50:51.185 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/iptables/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.185 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp: Cannot mkdir: Input/output error 2024-04-27T03:50:51.185 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.185 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib: Cannot mkdir: Input/output error 2024-04-27T03:50:51.185 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/addrs.c: Cannot open: Input/output error 2024-04-27T03:50:51.186 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/connectx.c: Cannot open: Input/output error 2024-04-27T03:50:51.186 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/bindx.c: Cannot open: Input/output error 2024-04-27T03:50:51.186 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/opt_info.c: Cannot open: Input/output error 2024-04-27T03:50:51.186 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.186 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/peeloff.c: Cannot open: Input/output error 2024-04-27T03:50:51.186 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/sendmsg.c: Cannot open: Input/output error 2024-04-27T03:50:51.186 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/recvmsg.c: Cannot open: Input/output error 2024-04-27T03:50:51.186 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests: Cannot mkdir: Input/output error 2024-04-27T03:50:51.186 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_inaddr_any.c: Cannot open: Input/output error 2024-04-27T03:50:51.187 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_sctp_sendrecvmsg.c: Cannot open: Input/output error 2024-04-27T03:50:51.187 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_autoclose.c: Cannot open: Input/output error 2024-04-27T03:50:51.187 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_basic.c: Cannot open: Input/output error 2024-04-27T03:50:51.187 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_send.c: Cannot open: Input/output error 2024-04-27T03:50:51.187 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_threads.c: Cannot open: Input/output error 2024-04-27T03:50:51.187 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_getname.c: Cannot open: Input/output error 2024-04-27T03:50:51.187 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_peeloff.c: Cannot open: Input/output error 2024-04-27T03:50:51.187 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_connectx.c: Cannot open: Input/output error 2024-04-27T03:50:51.188 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_socket_bind_listen.c: Cannot open: Input/output error 2024-04-27T03:50:51.188 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_shutdown.c: Cannot open: Input/output error 2024-04-27T03:50:51.188 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_recvmsg.c: Cannot open: Input/output error 2024-04-27T03:50:51.188 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.188 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_accept_close.c: Cannot open: Input/output error 2024-04-27T03:50:51.188 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_nonblock.c: Cannot open: Input/output error 2024-04-27T03:50:51.188 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_recvfrom.c: Cannot open: Input/output error 2024-04-27T03:50:51.189 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_connect.c: Cannot open: Input/output error 2024-04-27T03:50:51.189 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_initmsg_connect.c: Cannot open: Input/output error 2024-04-27T03:50:51.189 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_sendmsg.c: Cannot open: Input/output error 2024-04-27T03:50:51.189 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_sendto.c: Cannot open: Input/output error 2024-04-27T03:50:51.189 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_connect.c: Cannot open: Input/output error 2024-04-27T03:50:51.189 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_tcp_style.c: Cannot open: Input/output error 2024-04-27T03:50:51.189 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_sockopt.c: Cannot open: Input/output error 2024-04-27T03:50:51.189 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_connectx.c: Cannot open: Input/output error 2024-04-27T03:50:51.190 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_fragments.c: Cannot open: Input/output error 2024-04-27T03:50:51.190 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_timetolive.c: Cannot open: Input/output error 2024-04-27T03:50:51.190 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_recvmsg.c: Cannot open: Input/output error 2024-04-27T03:50:51.190 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_assoc_shutdown.c: Cannot open: Input/output error 2024-04-27T03:50:51.190 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_events.c: Cannot open: Input/output error 2024-04-27T03:50:51.190 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_sockopt.c: Cannot open: Input/output error 2024-04-27T03:50:51.190 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_assoc_abort.c: Cannot open: Input/output error 2024-04-27T03:50:51.190 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_rtoinfo.c: Cannot open: Input/output error 2024-04-27T03:50:51.191 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_addrs.c: Cannot open: Input/output error 2024-04-27T03:50:51.191 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/testlib: Cannot mkdir: Input/output error 2024-04-27T03:50:51.191 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/testlib/sctputil.h: Cannot open: Input/output error 2024-04-27T03:50:51.191 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/testlib/sctputil.c: Cannot open: Input/output error 2024-04-27T03:50:51.191 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/testlib/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.191 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/include: Cannot mkdir: Input/output error 2024-04-27T03:50:51.191 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/include/netinet: Cannot mkdir: Input/output error 2024-04-27T03:50:51.191 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/sctp/include/netinet/sctp.h: Cannot open: Input/output error 2024-04-27T03:50:51.192 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs: Cannot mkdir: Input/output error 2024-04-27T03:50:51.192 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/fsx-linux: Cannot mkdir: Input/output error 2024-04-27T03:50:51.192 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/fsx-linux/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.192 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/fsx-linux/fsx.sh: Cannot open: Input/output error 2024-04-27T03:50:51.192 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfsstress: Cannot mkdir: Input/output error 2024-04-27T03:50:51.192 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfsstress/make_tree.c: Cannot open: Input/output error 2024-04-27T03:50:51.192 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfsstress/nfsstress: Cannot open: Input/output error 2024-04-27T03:50:51.192 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfsstress/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.193 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs04: Cannot mkdir: Input/output error 2024-04-27T03:50:51.193 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs04/nfs04: Cannot open: Input/output error 2024-04-27T03:50:51.193 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs04/nfs04_create_file.c: Cannot open: Input/output error 2024-04-27T03:50:51.193 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs04/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.193 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01: Cannot mkdir: Input/output error 2024-04-27T03:50:51.193 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01/nfs_flock.c: Cannot open: Input/output error 2024-04-27T03:50:51.193 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01/nfs_flock.h: Cannot open: Input/output error 2024-04-27T03:50:51.193 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01/nfslock01: Cannot open: Input/output error 2024-04-27T03:50:51.193 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.194 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01/nfs_flock_dgen.c: Cannot open: Input/output error 2024-04-27T03:50:51.194 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01/nfs_flock_frk.c: Cannot open: Input/output error 2024-04-27T03:50:51.194 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01/nfs_flock_func.c: Cannot open: Input/output error 2024-04-27T03:50:51.194 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.194 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfsstat01: Cannot mkdir: Input/output error 2024-04-27T03:50:51.194 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfsstat01/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.194 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfsstat01/nfsstat01: Cannot open: Input/output error 2024-04-27T03:50:51.195 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs02: Cannot mkdir: Input/output error 2024-04-27T03:50:51.195 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs02/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.195 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs02/nfs02: Cannot open: Input/output error 2024-04-27T03:50:51.195 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs01: Cannot mkdir: Input/output error 2024-04-27T03:50:51.195 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs01/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.195 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs01/nfs01: Cannot open: Input/output error 2024-04-27T03:50:51.195 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs01/open_files.c: Cannot open: Input/output error 2024-04-27T03:50:51.195 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs03: Cannot mkdir: Input/output error 2024-04-27T03:50:51.196 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs03/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.196 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs03/nfs03: Cannot open: Input/output error 2024-04-27T03:50:51.196 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/README: Cannot open: Input/output error 2024-04-27T03:50:51.196 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/CTHON04_README: Cannot open: Input/output error 2024-04-27T03:50:51.196 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs_fsstress: Cannot mkdir: Input/output error 2024-04-27T03:50:51.196 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs_fsstress/global.h: Cannot open: Input/output error 2024-04-27T03:50:51.196 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs_fsstress/nfs_fsstress.sh: Cannot open: Input/output error 2024-04-27T03:50:51.196 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs_fsstress/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.197 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs_fsstress/xfscompat.h: Cannot open: Input/output error 2024-04-27T03:50:51.197 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs_fsstress/fsstress.c: Cannot open: Input/output error 2024-04-27T03:50:51.197 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/network: Cannot utime: Input/output error 2024-04-27T03:50:51.197 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21: Cannot mkdir: Input/output error 2024-04-27T03:50:51.197 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/COPYING: Cannot open: Input/output error 2024-04-27T03:50:51.197 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/debug.c: Cannot open: Input/output error 2024-04-27T03:50:51.197 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts: Cannot mkdir: Input/output error 2024-04-27T03:50:51.198 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/00check_cdrom_presence: Cannot open: Input/output error 2024-04-27T03:50:51.198 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/mem_alloc: Cannot open: Input/output error 2024-04-27T03:50:51.198 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/ramsnake: Cannot open: Input/output error 2024-04-27T03:50:51.198 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/ltp: Cannot open: Input/output error 2024-04-27T03:50:51.198 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/memxfer5b: Cannot open: Input/output error 2024-04-27T03:50:51.198 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/memtest: Cannot open: Input/output error 2024-04-27T03:50:51.198 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/cpufreq: Cannot open: Input/output error 2024-04-27T03:50:51.198 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/00xbonkers: Cannot open: Input/output error 2024-04-27T03:50:51.198 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/coords: Cannot open: Input/output error 2024-04-27T03:50:51.199 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/00checklatest: Cannot open: Input/output error 2024-04-27T03:50:51.199 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/build_kernel: Cannot open: Input/output error 2024-04-27T03:50:51.199 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/schedutils: Cannot open: Input/output error 2024-04-27T03:50:51.199 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/time_test: Cannot open: Input/output error 2024-04-27T03:50:51.199 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/lame: Cannot open: Input/output error 2024-04-27T03:50:51.199 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/ipmitool: Cannot open: Input/output error 2024-04-27T03:50:51.199 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-04-27T03:50:51.199 INFO:tasks.workunit.client.0.smithi029.stderr:gzip: stdin: Stale file handle 2024-04-27T03:50:51.200 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/bonnie++: Cannot open: Input/output error 2024-04-27T03:50:51.200 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/random_syscall: Cannot open: Input/output error 2024-04-27T03:50:51.200 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/nasm: Cannot open: Input/output error 2024-04-27T03:50:51.200 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/QUICKSTART: Cannot open: Input/output error 2024-04-27T03:50:51.200 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/CHANGELOG: Cannot open: Input/output error 2024-04-27T03:50:51.200 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/nfs_logging: Cannot open: Input/output error 2024-04-27T03:50:51.200 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/Makefile: Cannot open: Input/output error 2024-04-27T03:50:51.200 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/test-tests.tar.gz: Cannot open: Input/output error 2024-04-27T03:50:51.200 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/fast-tests.tar.gz: Cannot open: Input/output error 2024-04-27T03:50:51.201 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/run.c: Cannot open: Input/output error 2024-04-27T03:50:51.201 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/README: Cannot open: Input/output error 2024-04-27T03:50:51.201 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/Install: Cannot open: Input/output error 2024-04-27T03:50:51.201 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases/pounder21/fancy_timed_loop.c: Cannot open: Input/output error 2024-04-27T03:50:51.201 INFO:tasks.workunit.client.0.smithi029.stderr:tar: Child returned status 1 2024-04-27T03:50:51.201 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231/testcases: Cannot utime: Input/output error 2024-04-27T03:50:51.201 INFO:tasks.workunit.client.0.smithi029.stderr:tar: ltp-full-20091231: Cannot utime: Input/output error 2024-04-27T03:50:51.201 INFO:tasks.workunit.client.0.smithi029.stderr:tar: Error is not recoverable: exiting now 2024-04-27T03:50:51.202 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-04-27T03:50:51.202 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-27T03:50:51.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:50.973+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:51.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:50.973+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:51.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:50.973+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:51.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:50.974+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:51.204 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:50.902+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:50.904682+0000) 2024-04-27T03:50:51.204 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:50.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:51.204 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:50.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:51.204 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:50.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:51.204 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:50.976+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:51.245 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-27T03:50:51.245 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-27T03:50:51.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:51.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:51.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:51.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:51.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:51.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:51.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:51.443+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:51.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:51.611+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:51.613731+0000) 2024-04-27T03:50:51.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:51.702+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:51.703987+0000) 2024-04-27T03:50:51.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:51.906+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:51.907986+0000) 2024-04-27T03:50:51.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:51.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:51.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:51.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:51.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:51.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:51.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:51.940+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:52.326 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:51.975+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:52.326 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:51.975+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:52.326 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:51.975+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:52.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:51.976+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:52.620 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:50:52.537+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73902 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:50:52.621 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:52.532+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:52.534422+0000) 2024-04-27T03:50:52.621 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:52.533+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:52.534933+0000) 2024-04-27T03:50:52.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:52.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:52.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:52.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:52.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:52.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:52.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:52.489+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:52.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:52.533+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:52.535192+0000) 2024-04-27T03:50:52.914 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:52.614+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:52.616105+0000) 2024-04-27T03:50:52.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:52.706+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:52.708517+0000) 2024-04-27T03:50:53.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:53.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:53.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:53.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:53.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:53.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:53.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:53.011+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:53.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:52.908+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:52.910337+0000) 2024-04-27T03:50:53.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:52.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:53.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:52.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:53.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:52.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:53.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:52.971+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:53.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:53.708+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:53.710683+0000) 2024-04-27T03:50:53.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:53.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:53.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:53.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:53.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:53.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:53.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:53.515+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:53.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:53.617+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:53.618913+0000) 2024-04-27T03:50:54.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:53.983+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:54.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:53.983+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:54.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:53.983+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:54.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:53.984+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:54.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:53.910+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:53.912571+0000) 2024-04-27T03:50:54.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:53.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:54.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:53.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:54.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:53.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:54.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:53.944+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:54.850 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:54.711+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:54.713343+0000) 2024-04-27T03:50:54.851 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:54.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:54.851 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:54.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:54.851 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:54.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:54.851 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:54.525+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:54.851 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:54.619+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:54.621028+0000) 2024-04-27T03:50:55.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:54.993+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:55.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:54.993+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:55.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:54.993+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:55.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:54.994+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:55.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:54.914+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:54.916709+0000) 2024-04-27T03:50:55.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:54.922+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:55.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:54.922+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:55.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:54.922+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:55.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:54.922+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:55.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:55.713+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:55.715487+0000) 2024-04-27T03:50:55.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:55.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:55.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:55.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:55.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:55.571+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:55.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:55.571+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:55.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:55.624+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:55.625853+0000) 2024-04-27T03:50:56.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:55.918+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:55.920023+0000) 2024-04-27T03:50:56.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:55.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:56.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:55.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:56.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:55.924+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:56.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:55.925+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:56.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:55.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:56.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:55.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:56.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:55.956+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:56.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:55.957+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:56.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:56.716+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:56.718489+0000) 2024-04-27T03:50:56.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:56.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:56.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:56.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:56.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:56.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:56.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:56.591+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:56.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:56.626+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:56.628042+0000) 2024-04-27T03:50:57.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:56.994+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:57.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:56.994+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:57.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:56.994+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:57.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:56.995+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:57.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:56.922+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:56.924249+0000) 2024-04-27T03:50:57.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:56.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:57.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:56.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:57.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:56.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:57.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:56.937+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:57.931 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:50:57.546+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73942 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:50:57.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:57.543+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:57.544826+0000) 2024-04-27T03:50:57.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:57.543+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:57.545115+0000) 2024-04-27T03:50:57.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:57.718+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:57.720733+0000) 2024-04-27T03:50:57.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:57.542+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:57.544613+0000) 2024-04-27T03:50:57.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:57.562+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:57.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:57.562+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:57.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:57.562+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:57.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:57.563+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:57.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:57.628+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:57.630227+0000) 2024-04-27T03:50:58.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:58.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:58.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:58.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:58.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:58.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:58.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:58.016+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:58.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:57.925+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:57.927263+0000) 2024-04-27T03:50:58.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:57.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:58.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:57.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:58.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:57.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:58.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:57.968+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:58.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:58.720+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:58.722789+0000) 2024-04-27T03:50:58.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:58.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:58.937 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:58.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:58.937 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:58.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:58.937 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:58.554+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:58.937 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:58.630+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:58.632418+0000) 2024-04-27T03:50:59.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:58.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:59.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:58.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:59.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:58.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:50:59.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:58.990+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:59.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:58.931+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:58.931988+0000) 2024-04-27T03:50:59.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:58.962+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:59.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:58.962+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:59.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:58.962+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:50:59.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:58.962+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:59.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:50:59.724+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:59.724939+0000) 2024-04-27T03:50:59.939 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:59.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:59.939 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:59.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:59.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:59.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:50:59.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:59.567+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:50:59.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:50:59.634+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:59.635259+0000) 2024-04-27T03:51:00.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:59.933+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:50:59.934657+0000) 2024-04-27T03:51:00.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:59.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:00.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:59.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:00.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:59.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:00.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:50:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:50:59.965+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:00.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:00.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:00.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:00.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:00.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:00.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:00.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:00.014+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:00.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:00.727+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:00.728215+0000) 2024-04-27T03:51:00.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:00.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:00.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:00.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:00.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:00.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:00.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:00.582+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:00.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:00.638+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:00.639358+0000) 2024-04-27T03:51:01.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:00.982+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:01.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:00.982+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:01.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:00.982+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:01.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:00.983+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:01.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:00.938+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:00.938933+0000) 2024-04-27T03:51:01.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:00.950+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:01.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:00.950+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:01.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:00.950+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:01.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:00.950+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:01.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:01.732+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:01.732894+0000) 2024-04-27T03:51:01.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:01.613+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:01.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:01.613+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:01.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:01.613+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:01.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:01.613+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:01.947 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:01.640+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:01.641705+0000) 2024-04-27T03:51:02.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:02.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:02.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:02.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:02.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:02.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:02.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:02.017+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:02.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:01.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:02.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:01.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:02.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:01.941+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:02.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:01.941+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:01.942099+0000) 2024-04-27T03:51:02.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:01.941+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:02.902 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:51:02.555+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 73986 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:51:02.903 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:02.553+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:02.553957+0000) 2024-04-27T03:51:02.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:02.552+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:02.552964+0000) 2024-04-27T03:51:02.903 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:02.736+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:02.737217+0000) 2024-04-27T03:51:02.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:02.552+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:02.553392+0000) 2024-04-27T03:51:02.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:02.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:02.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:02.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:02.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:02.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:02.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:02.584+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:02.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:02.643+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:02.643821+0000) 2024-04-27T03:51:03.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:02.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:03.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:02.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:03.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:02.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:03.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:02.895+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:03.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:02.943+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:02.944586+0000) 2024-04-27T03:51:03.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:03.021+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:03.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:03.021+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:03.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:03.021+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:03.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:03.022+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:03.861 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:03.739+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:03.740214+0000) 2024-04-27T03:51:03.862 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:03.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:03.862 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:03.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:03.862 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:03.567+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:03.862 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:03.568+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:03.862 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:03.645+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:03.646029+0000) 2024-04-27T03:51:04.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:03.993+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:04.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:03.993+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:04.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:03.993+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:04.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:03.994+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:04.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:03.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:04.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:03.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:04.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:03.856+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:04.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:03.856+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:04.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:03.945+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:03.946626+0000) 2024-04-27T03:51:04.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:04.744+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:04.744886+0000) 2024-04-27T03:51:04.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:04.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:04.902 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:04.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:04.902 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:04.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:04.902 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:04.556+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:04.902 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:04.647+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:04.648117+0000) 2024-04-27T03:51:05.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:04.949+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:05.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:04.949+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:05.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:04.949+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:05.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:04.950+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:05.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:04.896+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:05.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:04.896+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:05.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:04.896+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:05.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:04.896+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:05.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:04.949+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:04.949840+0000) 2024-04-27T03:51:05.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:05.746+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:05.746990+0000) 2024-04-27T03:51:05.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:05.543+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:05.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:05.543+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:05.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:05.543+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:05.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:05.545+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:05.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:05.649+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:05.650359+0000) 2024-04-27T03:51:06.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:05.969+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:06.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:05.969+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:06.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:05.969+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:06.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:05.970+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:06.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:05.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:06.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:05.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:06.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:05.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:06.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:05.893+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:06.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:05.951+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:05.952788+0000) 2024-04-27T03:51:06.858 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:06.748+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:06.749131+0000) 2024-04-27T03:51:06.859 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:06.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:06.859 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:06.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:06.859 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:06.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:06.859 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:06.517+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:06.859 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:06.651+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:06.652605+0000) 2024-04-27T03:51:07.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:06.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:07.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:06.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:07.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:06.938+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:07.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:06.939+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:07.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:06.853+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:07.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:06.853+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:07.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:06.853+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:07.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:06.853+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:07.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:06.955+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:06.956040+0000) 2024-04-27T03:51:07.868 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:51:07.564+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74026 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:51:07.869 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:07.560+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:07.561760+0000) 2024-04-27T03:51:07.869 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:07.532+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:07.869 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:07.532+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:07.869 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:07.532+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:07.869 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:07.533+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:07.870 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:07.561+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:07.562179+0000) 2024-04-27T03:51:07.870 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:07.654+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:07.655719+0000) 2024-04-27T03:51:07.870 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:07.560+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:07.561258+0000) 2024-04-27T03:51:07.870 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:07.751+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:07.752169+0000) 2024-04-27T03:51:08.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:07.904+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:08.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:07.904+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:08.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:07.904+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:08.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:07.905+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:08.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:07.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:08.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:07.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:08.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:07.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:08.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:07.863+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:08.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:07.957+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:07.958733+0000) 2024-04-27T03:51:08.758 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:08.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:08.759 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:08.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:08.759 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:08.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:08.759 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:08.487+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:08.759 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:08.657+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:08.657920+0000) 2024-04-27T03:51:09.037 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:08.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:09.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:08.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:09.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:08.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:09.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:08.865+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:09.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:08.962+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:08.963485+0000) 2024-04-27T03:51:09.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:08.753+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:08.754322+0000) 2024-04-27T03:51:09.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:08.922+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:09.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:08.922+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:09.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:08.922+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:09.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:08.923+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:09.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:09.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:09.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:09.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:09.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:09.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:09.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:09.505+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:09.937 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:09.661+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:09.662159+0000) 2024-04-27T03:51:09.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:09.839+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:09.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:09.839+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:09.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:09.839+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:09.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:09.839+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:09.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:09.756+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:09.757252+0000) 2024-04-27T03:51:09.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:09.932+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:09.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:09.932+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:09.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:09.932+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:10.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:09.933+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:10.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:09.965+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:09.966196+0000) 2024-04-27T03:51:10.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:10.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:10.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:10.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:10.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:10.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:10.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:10.469+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:10.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:10.665+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:10.666423+0000) 2024-04-27T03:51:11.037 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:10.760+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:10.760991+0000) 2024-04-27T03:51:11.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:10.903+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:11.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:10.903+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:11.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:10.903+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:11.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:10.904+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:11.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:10.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:11.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:10.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:11.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:10.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:11.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:10.879+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:11.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:10.967+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:10.968449+0000) 2024-04-27T03:51:11.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:11.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:11.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:11.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:11.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:11.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:11.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:11.448+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:11.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:11.667+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:11.668696+0000) 2024-04-27T03:51:11.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:11.763+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:11.764192+0000) 2024-04-27T03:51:11.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:11.923+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:11.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:11.923+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:11.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:11.923+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:11.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:11.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:11.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:11.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:11.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:11.865+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:11.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:11.865+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:12.209 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:11.924+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:12.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:11.972+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:11.973255+0000) 2024-04-27T03:51:12.770 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:51:12.575+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74066 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:51:12.771 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:12.569+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:12.569955+0000) 2024-04-27T03:51:12.771 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:12.568+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:12.569779+0000) 2024-04-27T03:51:12.771 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:12.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:12.772 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:12.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:12.772 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:12.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:12.772 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:12.488+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:12.772 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:12.569+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:12.570652+0000) 2024-04-27T03:51:12.772 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:12.671+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:12.672211+0000) 2024-04-27T03:51:13.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:12.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:13.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:12.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:13.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:12.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:13.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:12.835+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:13.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:12.974+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:12.975626+0000) 2024-04-27T03:51:13.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:12.765+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:12.766392+0000) 2024-04-27T03:51:13.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:12.880+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:13.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:12.881+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:13.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:12.881+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:13.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:12.881+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:13.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:13.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:13.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:13.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:13.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:13.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:13.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:13.478+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:13.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:13.673+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:13.674275+0000) 2024-04-27T03:51:13.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:13.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:13.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:13.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:13.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:13.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:13.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:13.851+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:13.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:13.770+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:13.770964+0000) 2024-04-27T03:51:13.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:13.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:13.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:13.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:13.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:13.873+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:13.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:13.874+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:14.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:13.977+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:13.978036+0000) 2024-04-27T03:51:14.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:14.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:14.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:14.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:14.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:14.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:14.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:14.466+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:14.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:14.677+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:14.678359+0000) 2024-04-27T03:51:15.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:14.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:15.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:14.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:15.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:14.885+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:15.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:14.886+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:15.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:14.979+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:14.980779+0000) 2024-04-27T03:51:15.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:14.773+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:14.774113+0000) 2024-04-27T03:51:15.040 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:14.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:15.040 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:14.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:15.040 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:14.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:15.040 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:14.845+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:15.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:15.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:15.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:15.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:15.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:15.453+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:15.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:15.454+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:15.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:15.680+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:15.681249+0000) 2024-04-27T03:51:15.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:15.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:15.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:15.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:15.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:15.878+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:15.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:15.878+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:15.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:15.775+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:15.776771+0000) 2024-04-27T03:51:15.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:15.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:15.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:15.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:15.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:15.850+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:15.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:15.851+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:16.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:15.984+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:15.985592+0000) 2024-04-27T03:51:16.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:16.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:16.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:16.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:16.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:16.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:16.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:16.447+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:16.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:16.683+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:16.684189+0000) 2024-04-27T03:51:16.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:16.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:16.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:16.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:16.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:16.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:16.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:16.864+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:16.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:16.779+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:16.780005+0000) 2024-04-27T03:51:16.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:16.863+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:16.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:16.863+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:16.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:16.863+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:16.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:16.864+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:17.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:16.987+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:16.988404+0000) 2024-04-27T03:51:17.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:17.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:17.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:17.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:17.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:17.467+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:17.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:17.468+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:17.904 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:51:17.584+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74106 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:51:17.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:17.581+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:17.581817+0000) 2024-04-27T03:51:17.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:17.581+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:17.581979+0000) 2024-04-27T03:51:17.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:17.688+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:17.688915+0000) 2024-04-27T03:51:17.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:17.580+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:17.581708+0000) 2024-04-27T03:51:17.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:17.781+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:17.782728+0000) 2024-04-27T03:51:17.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:17.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:17.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:17.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:17.906 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:17.899+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:18.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:17.900+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:18.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:17.902+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:18.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:17.902+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:18.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:17.902+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:18.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:17.903+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:18.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:17.989+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:17.990674+0000) 2024-04-27T03:51:18.789 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:18.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:18.789 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:18.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:18.789 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:18.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:18.789 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:18.504+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:18.790 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:18.690+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:18.690948+0000) 2024-04-27T03:51:19.095 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:18.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:19.095 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:18.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:19.095 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:18.900+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:19.095 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:18.900+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:19.096 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:18.992+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:18.992958+0000) 2024-04-27T03:51:19.096 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:18.784+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:18.784929+0000) 2024-04-27T03:51:19.096 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:18.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:19.096 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:18.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:19.096 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:18.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:19.097 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:18.856+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:19.791 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:19.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:19.792 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:19.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:19.792 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:19.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:19.792 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:19.504+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:19.792 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:19.692+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:19.693618+0000) 2024-04-27T03:51:20.099 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:19.786+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:19.787189+0000) 2024-04-27T03:51:20.100 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:19.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:20.100 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:19.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:20.100 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:19.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:20.100 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:19.887+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:20.100 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:19.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:20.100 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:19.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:20.101 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:19.916+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:20.101 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:19.916+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:20.101 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:19.996+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:19.997750+0000) 2024-04-27T03:51:20.794 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:20.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:20.794 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:20.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:20.794 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:20.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:20.794 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:20.478+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:20.794 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:20.695+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:20.695847+0000) 2024-04-27T03:51:21.100 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:20.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:21.100 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:20.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:21.100 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:20.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:21.100 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:20.876+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:21.101 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:21.001+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:21.002791+0000) 2024-04-27T03:51:21.101 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:20.788+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:20.789559+0000) 2024-04-27T03:51:21.101 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:20.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:21.101 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:20.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:21.101 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:20.891+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:21.102 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:20.892+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:21.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:21.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:21.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:21.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:21.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:21.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:21.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:21.430+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:21.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:21.697+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:21.698084+0000) 2024-04-27T03:51:21.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:21.791+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:21.791854+0000) 2024-04-27T03:51:21.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:21.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:21.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:21.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:21.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:21.889+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:21.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:21.890+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:21.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:21.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:21.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:21.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:21.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:21.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:21.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:21.917+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:22.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:22.006+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:22.007511+0000) 2024-04-27T03:51:22.590 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:22.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:22.590 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:22.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:22.591 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:22.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:22.591 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:22.451+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:22.893 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:51:22.593+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74146 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:51:22.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:22.588+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:22.589308+0000) 2024-04-27T03:51:22.894 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:22.589+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:22.589894+0000) 2024-04-27T03:51:22.894 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:22.702+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:22.702903+0000) 2024-04-27T03:51:22.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:22.590+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:22.591515+0000) 2024-04-27T03:51:22.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:22.796+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:22.796967+0000) 2024-04-27T03:51:22.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:22.888+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:22.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:22.888+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:22.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:22.888+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:23.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:22.889+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:23.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:22.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:23.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:22.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:23.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:22.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:23.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:22.918+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:23.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:23.011+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:23.012631+0000) 2024-04-27T03:51:23.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:23.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:23.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:23.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:23.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:23.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:23.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:23.407+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:23.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:23.707+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:23.708038+0000) 2024-04-27T03:51:23.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:23.906+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:23.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:23.906+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:23.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:23.906+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:23.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:23.907+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:23.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:23.799+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:23.800048+0000) 2024-04-27T03:51:23.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:23.890+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:23.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:23.890+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:23.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:23.890+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:23.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:23.891+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:24.329 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:24.014+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:24.015624+0000) 2024-04-27T03:51:24.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:24.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:24.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:24.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:24.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:24.386+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:24.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:24.387+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:24.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:24.712+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:24.713153+0000) 2024-04-27T03:51:24.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:24.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:24.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:24.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:24.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:24.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:24.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:24.864+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:24.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:24.803+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:24.804419+0000) 2024-04-27T03:51:24.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:24.923+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:24.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:24.923+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:24.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:24.923+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:24.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:24.924+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:25.329 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:25.016+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:25.017772+0000) 2024-04-27T03:51:25.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:25.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:25.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:25.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:25.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:25.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:25.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:25.407+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:25.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:25.714+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:25.715438+0000) 2024-04-27T03:51:25.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:25.882+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:25.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:25.882+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:25.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:25.882+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:25.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:25.882+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:25.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:25.805+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:25.806561+0000) 2024-04-27T03:51:25.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:25.949+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:25.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:25.949+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:25.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:25.949+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:25.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:25.950+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:26.329 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:26.019+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:26.020612+0000) 2024-04-27T03:51:26.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:26.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:26.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:26.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:26.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:26.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:26.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:26.359+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:26.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:26.718+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:26.719688+0000) 2024-04-27T03:51:26.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:26.808+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:26.808862+0000) 2024-04-27T03:51:26.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:26.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:26.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:26.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:26.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:26.871+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:26.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:26.872+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:27.329 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:27.022+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:27.022909+0000) 2024-04-27T03:51:27.329 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:26.976+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:27.329 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:26.976+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:27.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:26.976+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:27.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:26.977+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:27.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:27.324+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:27.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:27.324+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:27.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:27.324+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:27.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:27.325+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:27.975 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:51:27.602+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74187 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:51:27.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:27.597+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:27.597982+0000) 2024-04-27T03:51:27.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:27.722+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:27.722902+0000) 2024-04-27T03:51:27.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:27.599+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:27.599930+0000) 2024-04-27T03:51:27.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:27.810+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:27.811159+0000) 2024-04-27T03:51:27.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:27.597+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:27.598669+0000) 2024-04-27T03:51:27.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:27.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:27.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:27.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:27.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:27.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:27.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:27.825+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:28.329 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:28.024+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:28.025128+0000) 2024-04-27T03:51:28.329 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:28.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:28.329 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:28.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:28.329 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:28.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:28.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:28.014+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:28.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:28.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:28.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:28.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:28.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:28.363+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:28.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:28.364+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:29.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:28.724+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:28.725135+0000) 2024-04-27T03:51:29.032 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:28.812+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:28.813415+0000) 2024-04-27T03:51:29.032 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:28.853+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:29.032 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:28.853+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:29.033 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:28.853+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:29.033 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:28.854+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:29.033 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:29.027+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:29.027840+0000) 2024-04-27T03:51:29.329 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:29.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:29.329 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:29.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:29.329 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:29.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:29.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:29.064+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:29.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:29.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:29.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:29.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:29.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:29.410+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:29.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:29.411+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:30.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:29.728+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:29.729362+0000) 2024-04-27T03:51:30.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:29.815+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:29.816297+0000) 2024-04-27T03:51:30.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:29.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:30.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:29.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:30.037 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:29.873+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:30.037 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:29.873+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:30.329 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:30.031+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:30.031841+0000) 2024-04-27T03:51:30.329 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:30.094+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:30.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:30.094+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:30.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:30.094+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:30.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:30.096+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:30.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:30.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:30.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:30.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:30.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:30.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:30.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:30.450+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:31.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:30.817+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:30.818601+0000) 2024-04-27T03:51:31.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:30.731+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:30.732346+0000) 2024-04-27T03:51:31.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:30.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:31.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:30.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:31.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:30.863+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:31.039 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:30.863+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:31.329 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:31.033+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:31.034228+0000) 2024-04-27T03:51:31.329 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:31.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:31.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:31.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:31.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:31.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:31.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:31.060+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:31.590 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:31.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:31.591 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:31.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:31.591 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:31.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:31.591 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:31.447+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:31.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:31.820+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:31.821762+0000) 2024-04-27T03:51:31.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:31.735+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:31.736602+0000) 2024-04-27T03:51:31.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:31.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:31.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:31.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:31.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:31.817+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:31.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:31.818+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:32.329 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:32.035+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:32.036759+0000) 2024-04-27T03:51:32.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:32.068+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:32.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:32.068+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:32.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:32.068+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:32.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:32.069+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:32.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:32.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:32.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:32.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:32.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:32.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:32.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:32.466+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:32.975 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:51:32.611+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74226 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:51:32.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:32.607+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:32.607915+0000) 2024-04-27T03:51:32.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:32.823+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:32.824424+0000) 2024-04-27T03:51:32.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:32.609+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:32.609937+0000) 2024-04-27T03:51:32.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:32.738+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:32.738843+0000) 2024-04-27T03:51:32.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:32.607+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:32.608133+0000) 2024-04-27T03:51:32.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:32.775+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:32.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:32.775+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:32.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:32.775+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:32.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:32.776+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:33.307 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:33.039+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:33.040054+0000) 2024-04-27T03:51:33.308 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:33.071+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:33.308 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:33.072+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:33.308 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:33.072+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:33.308 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:33.072+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:33.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:33.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:33.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:33.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:33.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:33.476+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:33.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:33.477+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:34.039 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:33.740+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:33.740906+0000) 2024-04-27T03:51:34.040 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:33.826+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:33.827333+0000) 2024-04-27T03:51:34.040 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:33.753+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:34.040 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:33.753+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:34.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:33.753+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:34.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:33.753+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:34.329 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:34.041+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:34.042229+0000) 2024-04-27T03:51:34.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:34.087+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:34.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:34.087+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:34.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:34.087+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:34.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:34.088+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:34.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:34.492+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:34.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:34.492+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:34.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:34.492+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:34.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:34.492+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:35.040 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:34.744+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:34.745137+0000) 2024-04-27T03:51:35.040 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:34.828+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:34.829434+0000) 2024-04-27T03:51:35.040 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:34.758+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:35.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:34.758+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:35.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:34.758+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:35.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:34.759+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:35.329 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:35.044+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:35.044892+0000) 2024-04-27T03:51:35.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:35.138+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:35.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:35.138+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:35.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:35.138+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:35.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:35.139+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:35.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:35.498+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:35.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:35.498+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:35.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:35.498+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:35.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:35.499+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:36.040 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:35.747+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:35.747850+0000) 2024-04-27T03:51:36.040 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:35.831+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:35.832296+0000) 2024-04-27T03:51:36.040 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:35.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:36.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:35.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:36.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:35.744+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:36.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:35.744+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:36.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:36.047+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:36.048071+0000) 2024-04-27T03:51:36.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:36.163+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:36.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:36.163+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:36.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:36.163+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:36.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:36.164+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:36.629 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:36.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:36.629 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:36.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:36.629 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:36.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:36.629 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:36.530+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:36.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:36.751+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:36.752646+0000) 2024-04-27T03:51:36.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:36.835+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:36.836686+0000) 2024-04-27T03:51:36.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:36.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:36.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:36.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:36.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:36.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:36.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:36.733+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:37.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:37.049+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:37.050265+0000) 2024-04-27T03:51:37.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:37.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:37.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:37.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:37.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:37.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:37.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:37.131+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:37.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:37.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:37.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:37.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:37.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:37.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:37.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:37.545+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:37.975 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:51:37.621+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74267 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:51:37.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:37.616+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:37.617507+0000) 2024-04-27T03:51:37.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:37.754+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:37.754962+0000) 2024-04-27T03:51:37.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:37.616+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:37.617597+0000) 2024-04-27T03:51:37.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:37.770+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:37.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:37.770+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:37.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:37.770+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:37.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:37.771+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:37.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:37.616+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:37.617244+0000) 2024-04-27T03:51:37.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:37.838+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:37.838930+0000) 2024-04-27T03:51:38.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:38.052+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:38.053045+0000) 2024-04-27T03:51:38.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:38.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:38.330 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:38.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:38.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:38.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:38.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:38.098+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:38.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:38.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:38.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:38.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:38.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:38.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:38.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:38.521+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:39.040 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:38.757+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:38.758125+0000) 2024-04-27T03:51:39.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:38.841+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:38.842200+0000) 2024-04-27T03:51:39.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:38.728+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:39.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:38.728+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:39.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:38.728+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:39.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:38.729+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:39.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:39.054+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:39.055226+0000) 2024-04-27T03:51:39.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:39.088+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:39.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:39.088+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:39.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:39.088+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:39.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:39.089+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:39.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:39.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:39.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:39.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:39.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:39.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:39.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:39.542+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:40.040 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:39.760+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:39.760891+0000) 2024-04-27T03:51:40.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:39.844+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:39.844883+0000) 2024-04-27T03:51:40.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:39.750+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:40.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:39.750+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:40.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:39.750+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:40.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:39.751+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:40.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:40.057+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:40.058213+0000) 2024-04-27T03:51:40.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:40.043+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:40.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:40.043+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:40.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:40.043+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:40.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:40.044+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:40.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:40.555+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:40.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:40.555+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:40.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:40.555+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:40.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:40.556+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:41.040 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:40.762+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:40.763466+0000) 2024-04-27T03:51:41.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:40.847+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:40.848125+0000) 2024-04-27T03:51:41.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:40.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:41.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:40.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:41.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:40.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:41.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:40.799+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:41.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:41.060+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:41.060939+0000) 2024-04-27T03:51:41.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:41.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:41.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:41.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:41.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:41.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:41.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:41.093+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:41.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:41.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:41.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:41.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:41.674 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:41.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:41.674 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:41.529+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:41.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:41.765+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:41.766333+0000) 2024-04-27T03:51:41.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:41.849+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:41.850511+0000) 2024-04-27T03:51:41.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:41.792+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:41.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:41.792+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:41.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:41.792+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:41.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:41.793+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:42.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:42.062+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:42.063362+0000) 2024-04-27T03:51:42.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:42.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:42.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:42.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:42.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:42.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:42.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:42.123+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:42.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:42.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:42.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:42.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:42.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:42.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:42.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:42.569+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:42.974 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:51:42.631+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74306 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:51:42.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:42.625+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:42.626476+0000) 2024-04-27T03:51:42.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:42.768+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:42.769200+0000) 2024-04-27T03:51:42.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:42.626+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:42.627639+0000) 2024-04-27T03:51:42.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:42.851+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:42.852734+0000) 2024-04-27T03:51:42.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:42.626+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:42.627595+0000) 2024-04-27T03:51:42.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:42.761+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:42.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:42.761+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:42.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:42.761+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:42.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:42.761+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:43.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:43.065+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:43.066282+0000) 2024-04-27T03:51:43.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:43.144+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:43.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:43.144+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:43.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:43.144+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:43.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:43.145+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:43.860 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:43.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:43.860 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:43.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:43.860 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:43.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:43.860 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:43.583+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:43.860 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:43.771+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:43.772187+0000) 2024-04-27T03:51:43.861 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:43.761+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:43.861 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:43.761+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:43.861 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:43.761+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:43.861 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:43.762+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:44.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:44.068+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:44.068895+0000) 2024-04-27T03:51:44.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:43.855+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:43.856003+0000) 2024-04-27T03:51:44.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:44.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:44.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:44.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:44.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:44.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:44.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:44.119+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:44.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:44.857+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:44.858328+0000) 2024-04-27T03:51:44.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:44.803+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:44.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:44.803+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:44.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:44.803+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:44.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:44.803+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:44.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:44.631+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:44.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:44.631+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:44.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:44.631+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:44.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:44.632+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:44.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:44.774+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:44.774834+0000) 2024-04-27T03:51:45.331 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:45.070+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:45.071192+0000) 2024-04-27T03:51:45.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:45.145+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:45.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:45.145+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:45.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:45.145+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:45.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:45.146+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:45.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:45.859+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:45.860626+0000) 2024-04-27T03:51:45.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:45.801+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:45.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:45.801+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:45.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:45.801+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:45.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:45.801+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:45.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:45.639+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:45.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:45.639+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:45.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:45.639+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:45.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:45.640+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:45.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:45.778+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:45.779078+0000) 2024-04-27T03:51:46.331 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:46.073+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:46.074303+0000) 2024-04-27T03:51:46.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:46.138+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:46.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:46.138+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:46.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:46.138+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:46.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:46.139+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:46.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:46.864+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:46.864978+0000) 2024-04-27T03:51:46.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:46.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:46.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:46.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:46.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:46.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:46.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:46.850+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:46.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:46.636+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:46.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:46.636+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:46.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:46.636+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:46.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:46.637+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:46.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:46.782+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:46.783342+0000) 2024-04-27T03:51:47.331 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:47.078+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:47.079418+0000) 2024-04-27T03:51:47.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:47.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:47.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:47.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:47.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:47.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:47.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:47.123+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:47.975 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:51:47.640+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74347 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:51:47.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:47.638+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:47.639005+0000) 2024-04-27T03:51:47.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:47.868+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:47.869342+0000) 2024-04-27T03:51:47.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:47.636+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:47.637210+0000) 2024-04-27T03:51:47.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:47.661+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:47.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:47.661+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:47.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:47.661+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:47.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:47.661+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:47.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:47.785+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:47.786258+0000) 2024-04-27T03:51:47.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:47.636+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:47.637153+0000) 2024-04-27T03:51:47.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:47.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:47.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:47.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:47.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:47.835+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:47.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:47.836+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:48.331 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:48.079+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:48.081678+0000) 2024-04-27T03:51:48.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:48.101+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:48.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:48.101+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:48.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:48.101+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:48.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:48.102+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:48.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:48.869+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:48.871691+0000) 2024-04-27T03:51:48.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:48.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:48.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:48.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:48.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:48.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:48.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:48.828+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:48.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:48.704+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:48.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:48.704+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:48.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:48.704+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:48.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:48.705+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:48.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:48.788+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:48.790203+0000) 2024-04-27T03:51:49.331 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:49.082+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:49.084465+0000) 2024-04-27T03:51:49.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:49.082+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:49.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:49.082+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:49.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:49.082+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:49.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:49.083+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:49.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:49.873+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:49.874968+0000) 2024-04-27T03:51:49.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:49.823+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:49.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:49.823+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:49.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:49.823+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:49.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:49.823+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:49.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:49.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:49.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:49.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:49.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:49.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:49.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:49.674+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:49.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:49.791+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:49.793252+0000) 2024-04-27T03:51:50.331 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:50.086+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:50.088694+0000) 2024-04-27T03:51:50.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:50.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:50.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:50.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:50.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:50.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:50.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:50.125+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:50.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:50.877+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:50.879445+0000) 2024-04-27T03:51:50.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:50.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:50.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:50.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:50.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:50.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:50.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:50.683+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:50.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:50.794+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:50.795904+0000) 2024-04-27T03:51:50.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:50.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:50.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:50.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:50.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:50.832+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:50.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:50.833+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:51.331 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:51.089+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:51.091465+0000) 2024-04-27T03:51:51.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:51.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:51.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:51.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:51.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:51.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:51.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:51.176+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:51.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:51.881+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:51.882800+0000) 2024-04-27T03:51:51.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:51.670+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:51.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:51.670+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:51.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:51.670+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:51.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:51.671+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:51.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:51.796+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:51.798047+0000) 2024-04-27T03:51:51.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:51.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:51.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:51.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:51.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:51.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:51.976 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:51.784+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:52.331 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:52.094+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:52.095830+0000) 2024-04-27T03:51:52.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:52.182+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:52.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:52.182+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:52.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:52.182+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:52.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:52.183+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:52.974 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:51:52.649+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74386 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:51:52.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:52.644+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:52.646525+0000) 2024-04-27T03:51:52.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:52.883+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:52.885129+0000) 2024-04-27T03:51:52.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:52.644+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:52.646333+0000) 2024-04-27T03:51:52.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:52.705+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:52.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:52.705+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:52.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:52.705+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:52.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:52.705+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:52.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:52.800+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:52.802312+0000) 2024-04-27T03:51:52.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:52.644+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:52.646716+0000) 2024-04-27T03:51:52.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:52.796+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:52.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:52.796+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:52.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:52.796+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:52.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:52.797+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:53.331 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:53.097+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:53.099025+0000) 2024-04-27T03:51:53.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:53.216+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:53.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:53.216+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:53.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:53.216+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:53.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:53.216+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:53.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:53.888+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:53.889893+0000) 2024-04-27T03:51:53.935 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:53.792+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:53.935 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:53.792+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:53.935 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:53.792+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:53.935 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:53.793+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:53.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:53.656+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:53.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:53.656+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:53.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:53.656+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:53.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:53.658+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:53.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:53.805+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:53.807001+0000) 2024-04-27T03:51:54.205 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:54.099+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:54.101085+0000) 2024-04-27T03:51:54.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:54.199+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:54.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:54.199+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:54.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:54.199+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:54.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:54.200+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:54.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:54.810+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:54.898 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:54.810+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:54.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:54.810+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:54.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:54.810+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:54.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:54.610+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:54.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:54.610+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:54.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:54.610+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:54.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:54.612+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:54.900 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:54.809+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:54.811728+0000) 2024-04-27T03:51:55.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:55.102+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:55.103795+0000) 2024-04-27T03:51:55.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:54.892+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:54.894468+0000) 2024-04-27T03:51:55.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:55.198+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:55.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:55.198+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:55.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:55.198+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:55.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:55.199+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:55.903 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:55.789+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:55.903 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:55.789+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:55.903 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:55.789+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:55.903 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:55.790+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:55.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:55.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:55.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:55.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:55.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:55.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:55.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:55.598+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:55.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:55.814+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:55.815968+0000) 2024-04-27T03:51:56.205 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:56.107+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:56.108931+0000) 2024-04-27T03:51:56.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:55.896+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:55.898758+0000) 2024-04-27T03:51:56.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:56.199+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:56.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:56.199+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:56.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:56.199+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:56.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:56.200+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:56.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:56.792+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:56.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:56.792+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:56.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:56.792+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:56.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:56.792+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:56.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:56.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:56.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:56.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:56.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:56.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:56.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:56.602+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:56.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:56.817+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:56.819183+0000) 2024-04-27T03:51:57.214 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:57.112+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:57.113970+0000) 2024-04-27T03:51:57.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:56.899+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:56.901379+0000) 2024-04-27T03:51:57.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:57.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:57.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:57.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:57.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:57.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:57.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:57.210+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:57.907 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:51:57.658+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74427 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:51:57.908 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:57.653+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:57.655094+0000) 2024-04-27T03:51:57.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:57.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:57.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:57.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:57.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:57.579+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:57.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:57.581+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:57.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:57.654+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:57.656600+0000) 2024-04-27T03:51:57.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:57.822+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:57.824332+0000) 2024-04-27T03:51:57.909 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:57.654+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:57.656658+0000) 2024-04-27T03:51:57.909 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:57.757+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:57.910 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:57.757+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:57.910 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:57.757+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:57.910 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:57.757+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:58.214 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:57.902+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:57.904104+0000) 2024-04-27T03:51:58.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:58.117+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:58.118989+0000) 2024-04-27T03:51:58.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:58.236+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:58.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:58.236+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:58.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:58.236+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:58.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:58.237+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:58.830 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:58.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:58.831 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:58.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:58.831 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:58.569+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:58.831 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:58.570+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:58.831 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:58.761+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:58.831 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:58.761+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:58.831 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:58.761+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:58.832 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:58.762+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:59.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:58.905+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:58.907582+0000) 2024-04-27T03:51:59.127 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:58.824+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:58.826411+0000) 2024-04-27T03:51:59.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:59.121+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:59.123242+0000) 2024-04-27T03:51:59.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:59.237+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:59.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:59.237+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:59.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:59.237+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:51:59.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:59.238+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:59.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:59.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:59.914 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:59.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:59.914 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:59.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:51:59.914 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:59.579+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:51:59.914 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:51:59.827+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:59.828884+0000) 2024-04-27T03:51:59.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:59.805+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:59.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:59.805+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:59.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:59.805+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:51:59.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:51:59.806+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:00.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:00.124+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:00.125539+0000) 2024-04-27T03:52:00.216 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:51:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:51:59.909+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:51:59.909912+0000) 2024-04-27T03:52:00.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:00.287+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:00.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:00.287+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:00.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:00.287+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:00.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:00.288+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:00.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:00.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:00.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:00.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:00.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:00.564+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:00.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:00.564+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:00.835 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:00.797+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:00.836 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:00.797+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:00.836 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:00.797+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:00.836 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:00.798+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:01.131 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:00.830+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:00.831155+0000) 2024-04-27T03:52:01.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:00.911+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:00.912283+0000) 2024-04-27T03:52:01.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:01.126+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:01.127725+0000) 2024-04-27T03:52:01.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:01.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:01.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:01.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:01.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:01.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:01.476 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:01.331+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:01.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:01.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:01.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:01.608+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:01.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:01.608+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:01.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:01.608+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:01.876 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:01.832+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:01.833538+0000) 2024-04-27T03:52:01.876 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:01.796+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:01.876 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:01.796+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:01.876 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:01.796+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:01.876 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:01.796+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:02.133 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:01.914+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:01.915342+0000) 2024-04-27T03:52:02.383 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:02.129+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:02.130000+0000) 2024-04-27T03:52:02.661 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:02.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:02.661 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:02.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:02.662 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:02.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:02.662 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:02.379+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:02.923 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:52:02.667+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74466 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:52:02.924 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:02.664+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:02.665254+0000) 2024-04-27T03:52:02.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:02.657+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:02.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:02.657+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:02.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:02.657+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:02.924 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:02.658+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:02.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:02.665+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:02.665828+0000) 2024-04-27T03:52:02.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:02.834+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:02.835779+0000) 2024-04-27T03:52:02.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:02.665+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:02.666050+0000) 2024-04-27T03:52:02.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:02.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:02.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:02.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:02.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:02.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:02.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:02.841+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:03.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:02.918+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:02.919706+0000) 2024-04-27T03:52:03.215 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:03.133+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:03.134309+0000) 2024-04-27T03:52:03.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:03.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:03.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:03.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:03.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:03.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:03.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:03.389+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:03.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:03.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:03.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:03.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:03.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:03.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:03.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:03.620+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:03.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:03.837+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:03.837826+0000) 2024-04-27T03:52:04.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:03.923+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:03.924223+0000) 2024-04-27T03:52:04.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:03.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:04.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:03.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:04.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:03.887+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:04.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:03.887+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:04.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:04.138+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:04.138967+0000) 2024-04-27T03:52:04.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:04.396+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:04.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:04.396+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:04.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:04.396+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:04.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:04.397+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:04.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:04.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:04.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:04.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:04.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:04.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:04.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:04.618+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:05.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:04.840+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:04.841029+0000) 2024-04-27T03:52:05.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:04.926+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:04.927221+0000) 2024-04-27T03:52:05.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:04.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:05.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:04.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:05.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:04.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:05.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:04.914+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:05.434 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:05.142+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:05.142837+0000) 2024-04-27T03:52:05.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:05.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:05.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:05.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:05.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:05.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:05.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:05.596+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:05.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:05.429+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:05.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:05.429+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:05.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:05.429+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:05.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:05.430+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:06.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:05.844+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:05.845266+0000) 2024-04-27T03:52:06.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:05.928+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:05.929320+0000) 2024-04-27T03:52:06.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:05.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:06.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:05.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:06.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:05.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:06.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:05.948+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:06.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:06.144+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:06.145009+0000) 2024-04-27T03:52:06.426 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:06.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:06.426 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:06.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:06.426 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:06.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:06.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:06.422+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:06.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:06.616+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:06.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:06.616+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:06.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:06.616+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:06.726 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:06.617+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:07.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:06.847+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:06.848320+0000) 2024-04-27T03:52:07.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:06.933+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:06.933951+0000) 2024-04-27T03:52:07.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:06.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:07.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:06.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:07.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:06.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:07.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:06.952+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:07.406 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:07.148+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:07.149758+0000) 2024-04-27T03:52:07.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:07.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:07.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:07.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:07.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:07.401+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:07.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:07.402+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:07.674 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:07.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:07.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:07.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:07.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:07.646+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:07.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:07.646+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:07.942 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:52:07.676+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74507 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:52:07.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:07.672+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:07.673394+0000) 2024-04-27T03:52:07.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:07.671+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:07.672339+0000) 2024-04-27T03:52:07.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:07.850+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:07.851143+0000) 2024-04-27T03:52:07.943 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:07.673+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:07.673950+0000) 2024-04-27T03:52:07.943 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:07.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:07.943 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:07.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:07.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:07.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:36.908197+0000 (oldest deadline 2024-04-27T03:41:56.908197+0000) 2024-04-27T03:52:07.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:07.915+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:08.215 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:07.937+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:07.938348+0000) 2024-04-27T03:52:08.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:08.152+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:08.153136+0000) 2024-04-27T03:52:08.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:08.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:08.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:08.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:08.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:08.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:30.693286+0000 (oldest deadline 2024-04-27T03:41:50.693286+0000) 2024-04-27T03:52:08.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:08.367+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:08.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:08.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:08.894 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:08.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:08.894 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:08.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:08.894 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:08.619+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:08.894 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:08.852+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:08.853362+0000) 2024-04-27T03:52:09.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:08.940+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:08.941542+0000) 2024-04-27T03:52:09.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:08.888+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:09.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:09.344+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:09.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:09.155+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:09.156237+0000) 2024-04-27T03:52:09.935 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:09.626+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:09.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:09.626+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:09.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:09.626+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:09.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:09.627+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:09.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:09.855+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:09.856697+0000) 2024-04-27T03:52:10.216 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:09.943+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:09.943869+0000) 2024-04-27T03:52:10.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:09.930+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:10.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:10.158+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:10.158866+0000) 2024-04-27T03:52:10.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:10.379+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:10.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:10.653+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:10.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:10.653+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:10.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:10.653+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:41:41.291902+0000 (oldest deadline 2024-04-27T03:42:01.291902+0000) 2024-04-27T03:52:10.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:10.654+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:10.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:10.859+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:10.860050+0000) 2024-04-27T03:52:11.216 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:10.945+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:10.946570+0000) 2024-04-27T03:52:11.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:10.939+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:11.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:11.161+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:11.162041+0000) 2024-04-27T03:52:11.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:11.335+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:11.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:11.687+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:11.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:11.861+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:11.862718+0000) 2024-04-27T03:52:12.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:11.985+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:12.216 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:12.163+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:12.164537+0000) 2024-04-27T03:52:12.217 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:11.947+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:11.948751+0000) 2024-04-27T03:52:12.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:12.309+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:12.955 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:52:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:52:12.685+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74546 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:52:12.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:12.681+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:12.682642+0000) 2024-04-27T03:52:12.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:12.681+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:12.682123+0000) 2024-04-27T03:52:12.957 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:12.681+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:12.682121+0000) 2024-04-27T03:52:12.957 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:12.731+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:12.957 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:12.865+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:12.866003+0000) 2024-04-27T03:52:13.216 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:12.951+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:12.952001+0000) 2024-04-27T03:52:13.217 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:13.014+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:13.217 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:13.166+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:13.167114+0000) 2024-04-27T03:52:13.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:13.287+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:14.023 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:13.954+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:13.955025+0000) 2024-04-27T03:52:14.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:13.760+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:14.024 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:13.870+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:13.870841+0000) 2024-04-27T03:52:14.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:14.019+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:14.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:14.170+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:14.171302+0000) 2024-04-27T03:52:14.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:14.315+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:14.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:14.957+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:14.958791+0000) 2024-04-27T03:52:14.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:14.759+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:14.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:14.873+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:14.874030+0000) 2024-04-27T03:52:15.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:14.988+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:15.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:15.174+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:15.175383+0000) 2024-04-27T03:52:15.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:15.345+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:16.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:15.992+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:16.146 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:15.792+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:16.146 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:15.877+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:15.878254+0000) 2024-04-27T03:52:16.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:15.960+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:15.960871+0000) 2024-04-27T03:52:16.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:16.373+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:16.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:16.176+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:16.177522+0000) 2024-04-27T03:52:17.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:16.962+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:16.963055+0000) 2024-04-27T03:52:17.146 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:16.817+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:17.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:16.879+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:16.880543+0000) 2024-04-27T03:52:17.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:16.985+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:17.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:17.360+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:17.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:17.179+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:17.179840+0000) 2024-04-27T03:52:17.819 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:52:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:52:17.693+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74587 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:52:17.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:17.690+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:17.690934+0000) 2024-04-27T03:52:17.820 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:17.690+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:17.691797+0000) 2024-04-27T03:52:17.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:17.690+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:17.690876+0000) 2024-04-27T03:52:17.820 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:17.806+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:17.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:17.881+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:17.882699+0000) 2024-04-27T03:52:18.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:18.000+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:18.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:18.181+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:18.182011+0000) 2024-04-27T03:52:18.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:17.964+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:17.965263+0000) 2024-04-27T03:52:18.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:18.319+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:19.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:18.966+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:18.967585+0000) 2024-04-27T03:52:19.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:18.965+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:19.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:18.843+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:19.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:18.886+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:18.886900+0000) 2024-04-27T03:52:19.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:19.184+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:19.184953+0000) 2024-04-27T03:52:19.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:19.303+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:20.146 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:19.856+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:20.146 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:19.889+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:19.890079+0000) 2024-04-27T03:52:20.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:19.974+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:20.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:19.969+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:19.970335+0000) 2024-04-27T03:52:20.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:20.344+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:20.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:20.186+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:20.187228+0000) 2024-04-27T03:52:21.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:20.971+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:20.972557+0000) 2024-04-27T03:52:21.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:20.963+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:21.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:20.875+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:21.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:20.893+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:20.894250+0000) 2024-04-27T03:52:21.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:21.332+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:21.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:21.188+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:21.189478+0000) 2024-04-27T03:52:22.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:21.953+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:22.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:21.836+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:22.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:21.896+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:21.897334+0000) 2024-04-27T03:52:22.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:21.975+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:21.975822+0000) 2024-04-27T03:52:22.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:22.191+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:22.191836+0000) 2024-04-27T03:52:22.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:22.289+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:22.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:22.698+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:22.699269+0000) 2024-04-27T03:52:22.975 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:52:22.702+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74626 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:52:22.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:22.700+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:22.700977+0000) 2024-04-27T03:52:22.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:22.699+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:22.700020+0000) 2024-04-27T03:52:22.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:22.829+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:22.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:22.898+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:22.899483+0000) 2024-04-27T03:52:23.334 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:22.977+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:22.978189+0000) 2024-04-27T03:52:23.335 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:22.984+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:23.335 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:23.193+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:23.194083+0000) 2024-04-27T03:52:23.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:23.332+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:24.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:23.979+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:23.980433+0000) 2024-04-27T03:52:24.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:24.008+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:24.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:23.845+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:24.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:23.900+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:23.901703+0000) 2024-04-27T03:52:24.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:24.352+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:24.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:24.195+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:24.196786+0000) 2024-04-27T03:52:25.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:25.044+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:25.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:24.857+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:25.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:24.903+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:24.903947+0000) 2024-04-27T03:52:25.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:24.981+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:24.982743+0000) 2024-04-27T03:52:25.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:25.361+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:25.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:25.198+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:25.199232+0000) 2024-04-27T03:52:26.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:26.028+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:26.147 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:25.984+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:25.984841+0000) 2024-04-27T03:52:26.148 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:25.852+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:26.148 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:25.905+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:25.906279+0000) 2024-04-27T03:52:26.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:26.392+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:26.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:26.200+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:26.201581+0000) 2024-04-27T03:52:27.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:26.986+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:26.987026+0000) 2024-04-27T03:52:27.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:26.980+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:27.135 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:26.811+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:27.135 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:26.907+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:26.908554+0000) 2024-04-27T03:52:27.420 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:27.202+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:27.203744+0000) 2024-04-27T03:52:27.708 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:27.415+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:27.974 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:52:27.711+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74667 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:52:27.975 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:27.707+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:27.708733+0000) 2024-04-27T03:52:27.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:27.708+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:27.709465+0000) 2024-04-27T03:52:27.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:27.768+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:27.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:27.909+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:27.910702+0000) 2024-04-27T03:52:27.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:27.708+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:27.709279+0000) 2024-04-27T03:52:28.335 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:27.991+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:27.991906+0000) 2024-04-27T03:52:28.335 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:27.982+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:28.335 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:28.205+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:28.205987+0000) 2024-04-27T03:52:28.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:28.421+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:29.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:28.993+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:28.994684+0000) 2024-04-27T03:52:29.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:28.986+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:29.046 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:28.789+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:29.046 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:28.914+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:28.914915+0000) 2024-04-27T03:52:29.335 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:29.208+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:29.208849+0000) 2024-04-27T03:52:29.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:29.378+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:30.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:29.997+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:29.998050+0000) 2024-04-27T03:52:30.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:29.747+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:30.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:29.918+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:29.919127+0000) 2024-04-27T03:52:30.335 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:30.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:30.335 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:30.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:30.335 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:30.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:30.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:30.032+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:30.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:30.211+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:30.212015+0000) 2024-04-27T03:52:30.705 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:30.341+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:30.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:30.701+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:30.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:30.921+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:30.922188+0000) 2024-04-27T03:52:31.335 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:30.999+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:31.000164+0000) 2024-04-27T03:52:31.335 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:31.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:31.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:31.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:31.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:31.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:31.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:31.052+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:31.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:31.213+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:31.214793+0000) 2024-04-27T03:52:31.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:31.349+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:31.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:31.678+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:31.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:31.678+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:31.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:31.678+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:31.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:31.679+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:31.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:31.925+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:31.926512+0000) 2024-04-27T03:52:32.335 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:32.002+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:32.003285+0000) 2024-04-27T03:52:32.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:32.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:32.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:32.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:32.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:32.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:32.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:32.041+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:32.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:32.216+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:32.217010+0000) 2024-04-27T03:52:32.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:32.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:32.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:32.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:32.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:32.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:32.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:32.398+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:32.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:32.716+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:32.716887+0000) 2024-04-27T03:52:32.975 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:52:32.721+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74706 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:52:32.975 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:32.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:32.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:32.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:32.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:32.682+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:32.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:32.684+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:32.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:32.716+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:32.717120+0000) 2024-04-27T03:52:32.976 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:32.927+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:32.928737+0000) 2024-04-27T03:52:32.977 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:32.716+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:32.716858+0000) 2024-04-27T03:52:33.335 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:33.005+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:33.006251+0000) 2024-04-27T03:52:33.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:33.091+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:33.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:33.091+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:33.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:33.091+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:33.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:33.091+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:33.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:33.219+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:33.220296+0000) 2024-04-27T03:52:33.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:33.377+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:33.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:33.377+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:33.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:33.377+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:33.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:33.378+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:34.012 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:33.727+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:34.012 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:33.727+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:34.012 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:33.727+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:34.012 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:33.728+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:34.013 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:33.931+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:33.932058+0000) 2024-04-27T03:52:34.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:34.059+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:34.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:34.059+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:34.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:34.059+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:34.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:34.060+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:34.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:34.222+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:34.223235+0000) 2024-04-27T03:52:34.337 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:34.008+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:34.008824+0000) 2024-04-27T03:52:34.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:34.396+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:34.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:34.396+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:34.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:34.396+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:34.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:34.398+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:35.046 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:35.011+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:35.011846+0000) 2024-04-27T03:52:35.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:35.023+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:35.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:35.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:35.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:35.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:35.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:35.024+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:35.047 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:34.777+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:35.047 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:34.777+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:35.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:34.777+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:35.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:34.777+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:35.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:34.934+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:34.935297+0000) 2024-04-27T03:52:35.335 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:35.225+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:35.226627+0000) 2024-04-27T03:52:35.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:35.427+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:35.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:35.427+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:35.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:35.427+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:35.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:35.428+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:36.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:36.014+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:36.015332+0000) 2024-04-27T03:52:36.148 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:35.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:36.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:35.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:36.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:35.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:36.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:35.988+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:36.149 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:35.803+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:36.149 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:35.803+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:36.149 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:35.803+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:36.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:35.804+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:36.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:35.937+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:35.938359+0000) 2024-04-27T03:52:36.427 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:36.228+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:36.229437+0000) 2024-04-27T03:52:36.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:36.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:36.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:36.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:36.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:36.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:36.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:36.424+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:37.046 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:37.016+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:37.017553+0000) 2024-04-27T03:52:37.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:37.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:37.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:37.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:37.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:37.031+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:37.047 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:37.032+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:37.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:36.789+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:37.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:36.789+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:37.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:36.789+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:37.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:36.790+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:37.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:36.940+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:36.941265+0000) 2024-04-27T03:52:37.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:37.231+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:37.232713+0000) 2024-04-27T03:52:37.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:37.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:37.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:37.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:37.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:37.420+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:37.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:37.422+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:38.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:37.725+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:37.726161+0000) 2024-04-27T03:52:38.024 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:52:37.730+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74747 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:52:38.025 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:37.726+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:37.727274+0000) 2024-04-27T03:52:38.025 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:37.726+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:37.727028+0000) 2024-04-27T03:52:38.025 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:37.774+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:38.025 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:37.774+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:38.025 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:37.774+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:38.026 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:37.775+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:38.026 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:37.943+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:37.943926+0000) 2024-04-27T03:52:38.336 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:38.020+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:38.020997+0000) 2024-04-27T03:52:38.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:38.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:38.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:38.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:38.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:38.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:38.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:38.024+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:38.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:38.234+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:38.235543+0000) 2024-04-27T03:52:38.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:38.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:38.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:38.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:38.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:38.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:38.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:38.452+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:39.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:39.023+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:39.024273+0000) 2024-04-27T03:52:39.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:39.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:39.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:39.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:39.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:39.049+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:39.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:39.049+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:39.149 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:38.817+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:39.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:38.817+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:39.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:38.817+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:39.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:38.817+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:39.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:38.945+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:38.946102+0000) 2024-04-27T03:52:39.433 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:39.239+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:39.240242+0000) 2024-04-27T03:52:39.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:39.429+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:39.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:39.429+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:39.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:39.429+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:39.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:39.430+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:40.057 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:40.025+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:40.026421+0000) 2024-04-27T03:52:40.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:39.795+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:40.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:39.795+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:40.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:39.795+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:40.058 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:39.797+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:40.058 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:39.947+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:39.948396+0000) 2024-04-27T03:52:40.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:40.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:40.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:40.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:40.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:40.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:40.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:40.053+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:40.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:40.244+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:40.244915+0000) 2024-04-27T03:52:40.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:40.480+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:40.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:40.480+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:40.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:40.480+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:40.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:40.480+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:41.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:41.027+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:41.028655+0000) 2024-04-27T03:52:41.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:41.064+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:41.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:41.064+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:41.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:41.064+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:41.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:41.065+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:41.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:40.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:41.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:40.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:41.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:40.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:41.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:40.838+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:41.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:40.949+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:40.950472+0000) 2024-04-27T03:52:41.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:41.246+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:41.247474+0000) 2024-04-27T03:52:41.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:41.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:41.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:41.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:41.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:41.523+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:41.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:41.524+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:42.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:42.032+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:42.033733+0000) 2024-04-27T03:52:42.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:42.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:42.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:42.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:42.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:42.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:42.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:42.083+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:42.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:41.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:42.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:41.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:42.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:41.851+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:42.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:41.852+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:42.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:41.953+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:41.954701+0000) 2024-04-27T03:52:42.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:42.250+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:42.251471+0000) 2024-04-27T03:52:42.852 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:52:42.739+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74786 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:52:42.852 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:42.735+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:42.736788+0000) 2024-04-27T03:52:42.853 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:42.733+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:42.734624+0000) 2024-04-27T03:52:42.853 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:42.847+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:42.853 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:42.847+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:42.853 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:42.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:42.853 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:42.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:42.853 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:42.556+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:42.854 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:42.556+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:42.854 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:42.735+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:42.735879+0000) 2024-04-27T03:52:43.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:43.037+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:43.038769+0000) 2024-04-27T03:52:43.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:43.062+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:43.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:43.062+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:43.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:43.062+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:43.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:43.063+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:43.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:42.847+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:43.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:42.849+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:43.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:42.956+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:42.957434+0000) 2024-04-27T03:52:43.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:43.253+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:43.254487+0000) 2024-04-27T03:52:43.848 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:43.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:43.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:43.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:43.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:43.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:43.849 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:43.513+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:44.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:44.042+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:44.043583+0000) 2024-04-27T03:52:44.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:44.018+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:44.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:44.018+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:44.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:44.018+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:44.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:44.019+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:44.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:43.844+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:44.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:43.844+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:44.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:43.844+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:44.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:43.845+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:44.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:43.961+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:43.962458+0000) 2024-04-27T03:52:44.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:44.256+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:44.257253+0000) 2024-04-27T03:52:44.879 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:44.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:44.880 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:44.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:44.880 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:44.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:44.880 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:44.546+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:45.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:45.045+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:45.045942+0000) 2024-04-27T03:52:45.149 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:44.875+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:45.149 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:44.875+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:45.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:44.875+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:45.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:44.876+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:45.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:44.963+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:44.964770+0000) 2024-04-27T03:52:45.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:45.068+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:45.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:45.068+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:45.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:45.068+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:45.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:45.069+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:45.441 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:45.259+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:45.260098+0000) 2024-04-27T03:52:45.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:45.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:45.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:45.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:45.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:45.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:45.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:45.511+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:46.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:46.047+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:46.048057+0000) 2024-04-27T03:52:46.149 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:46.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:46.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:46.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:46.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:46.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:46.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:46.096+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:46.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:45.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:46.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:45.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:46.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:45.873+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:46.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:45.874+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:46.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:45.966+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:45.966842+0000) 2024-04-27T03:52:46.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:46.261+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:46.262649+0000) 2024-04-27T03:52:46.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:46.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:46.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:46.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:46.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:46.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:46.878 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:46.498+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:47.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:47.049+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:47.050172+0000) 2024-04-27T03:52:47.144 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:46.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:47.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:46.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:47.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:46.874+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:47.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:46.875+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:47.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:46.968+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:46.968894+0000) 2024-04-27T03:52:47.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:47.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:47.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:47.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:47.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:47.140+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:47.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:47.141+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:47.475 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:47.264+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:47.264951+0000) 2024-04-27T03:52:47.875 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:52:47.750+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74827 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:52:47.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:47.744+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:47.745355+0000) 2024-04-27T03:52:47.876 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:47.744+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:47.745310+0000) 2024-04-27T03:52:47.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:47.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:47.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:47.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:47.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:47.546+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:47.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:47.547+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:47.876 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:47.743+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:47.744292+0000) 2024-04-27T03:52:48.149 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:48.051+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:48.052411+0000) 2024-04-27T03:52:48.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:47.913+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:48.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:47.913+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:48.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:47.913+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:48.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:47.914+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:48.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:47.970+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:47.971581+0000) 2024-04-27T03:52:48.150 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:48.107+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:48.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:48.107+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:48.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:48.107+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:48.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:48.108+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:48.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:48.266+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:48.267114+0000) 2024-04-27T03:52:48.915 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:48.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:48.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:48.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:48.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:48.547+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:48.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:48.548+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:49.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:49.053+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:49.054513+0000) 2024-04-27T03:52:49.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:49.097+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:49.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:49.097+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:49.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:49.097+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:49.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:49.098+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:49.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:48.911+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:49.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:48.911+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:49.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:48.911+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:49.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:48.912+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:49.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:48.975+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:48.976506+0000) 2024-04-27T03:52:49.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:49.267+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:49.269795+0000) 2024-04-27T03:52:49.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:49.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:49.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:49.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:49.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:49.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:49.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:49.556+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:50.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:50.055+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:50.057323+0000) 2024-04-27T03:52:50.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:50.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:50.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:50.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:50.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:50.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:50.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:50.100+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:50.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:49.952+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:50.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:49.952+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:50.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:49.952+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:50.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:49.953+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:50.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:49.979+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:49.981488+0000) 2024-04-27T03:52:50.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:50.270+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:50.272027+0000) 2024-04-27T03:52:50.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:50.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:50.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:50.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:50.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:50.567+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:50.965 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:50.568+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:51.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:51.057+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:51.059441+0000) 2024-04-27T03:52:51.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:51.085+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:51.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:51.086+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:51.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:51.086+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:51.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:51.086+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:51.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:50.958+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:51.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:50.958+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:51.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:50.958+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:51.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:50.959+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:51.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:50.981+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:50.983593+0000) 2024-04-27T03:52:51.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:51.272+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:51.274158+0000) 2024-04-27T03:52:51.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:51.565+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:51.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:51.565+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:51.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:51.565+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:51.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:51.566+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:52.243 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:52.059+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:52.061678+0000) 2024-04-27T03:52:52.244 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:51.984+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:51.985823+0000) 2024-04-27T03:52:52.244 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:51.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:52.244 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:51.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:52.244 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:51.998+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:52.244 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:51.999+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:52.244 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:52.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:52.245 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:52.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:52.245 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:52.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:52.245 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:52.121+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:52.560 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:52.274+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:52.276776+0000) 2024-04-27T03:52:52.880 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:52:52.758+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74866 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:52:52.881 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:52.754+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:52.756473+0000) 2024-04-27T03:52:52.881 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:52.753+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:52.755543+0000) 2024-04-27T03:52:52.881 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:52.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:52.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:52.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:52.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:52.555+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:52.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:52.557+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:52.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:52.753+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:52.755471+0000) 2024-04-27T03:52:53.150 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:53.062+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:53.063841+0000) 2024-04-27T03:52:53.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:52.988+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:52.990023+0000) 2024-04-27T03:52:53.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:53.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:53.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:53.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:53.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:53.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:53.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:53.041+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:53.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:53.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:53.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:53.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:53.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:53.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:53.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:53.101+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:53.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:53.277+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:53.279025+0000) 2024-04-27T03:52:53.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:53.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:53.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:53.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:53.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:53.595+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:53.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:53.595+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:54.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:54.064+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:54.065947+0000) 2024-04-27T03:52:54.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:53.990+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:53.992106+0000) 2024-04-27T03:52:54.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:54.006+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:54.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:54.006+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:54.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:54.006+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:54.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:54.007+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:54.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:54.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:54.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:54.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:54.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:54.055+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:54.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:54.056+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:54.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:54.279+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:54.281076+0000) 2024-04-27T03:52:54.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:54.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:54.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:54.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:54.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:54.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:54.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:54.625+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:55.249 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:55.066+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:55.068249+0000) 2024-04-27T03:52:55.249 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:55.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:55.249 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:55.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:55.249 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:55.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:55.250 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:55.059+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:55.250 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:54.992+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:54.994243+0000) 2024-04-27T03:52:55.250 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:55.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:55.250 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:55.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:55.250 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:55.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:55.250 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:55.003+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:55.599 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:55.283+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:55.285272+0000) 2024-04-27T03:52:55.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:55.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:55.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:55.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:55.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:55.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:55.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:55.595+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:56.290 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:56.070+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:56.072243+0000) 2024-04-27T03:52:56.291 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:56.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:56.291 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:56.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:56.291 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:56.077+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:56.291 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:56.077+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:56.291 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:55.996+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:55.998478+0000) 2024-04-27T03:52:56.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:56.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:56.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:56.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:56.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:56.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:56.292 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:56.011+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:56.620 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:56.285+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:56.287417+0000) 2024-04-27T03:52:56.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:56.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:56.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:56.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:56.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:56.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:56.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:56.616+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:57.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:57.073+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:57.075259+0000) 2024-04-27T03:52:57.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:57.101+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:57.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:57.101+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:57.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:57.101+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:57.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:57.101+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:57.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:56.999+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:57.001315+0000) 2024-04-27T03:52:57.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:57.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:57.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:57.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:57.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:57.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:57.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:57.056+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:57.627 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:57.288+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:57.290593+0000) 2024-04-27T03:52:57.891 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:52:57.769+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74907 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:52:57.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:57.762+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:57.764301+0000) 2024-04-27T03:52:57.927 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:57.762+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:57.764567+0000) 2024-04-27T03:52:57.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:57.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:57.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:57.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:57.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:57.622+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:57.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:57.623+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:57.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:57.761+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:57.763635+0000) 2024-04-27T03:52:58.150 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:58.076+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:58.078361+0000) 2024-04-27T03:52:58.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:58.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:58.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:58.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:58.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:58.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:58.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:58.055+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:58.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:58.001+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:58.003474+0000) 2024-04-27T03:52:58.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:58.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:58.152 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:58.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:58.152 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:58.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:58.152 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:58.082+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:58.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:58.293+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:58.295107+0000) 2024-04-27T03:52:58.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:58.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:58.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:58.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:58.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:58.663+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:58.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:58.663+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:59.300 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:59.080+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:59.082598+0000) 2024-04-27T03:52:59.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:59.003+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:59.005516+0000) 2024-04-27T03:52:59.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:59.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:59.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:59.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:59.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:59.128+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:52:59.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:52:59.129+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:59.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:59.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:59.302 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:59.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:59.302 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:59.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:52:59.302 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:59.081+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:52:59.302 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:52:59.295+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:52:59.297286+0000) 2024-04-27T03:52:59.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:59.643+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:59.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:59.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:59.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:59.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:52:59.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:52:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:52:59.646+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:00.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:00.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:00.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:00.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:00.304 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:00.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:00.304 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:00.056+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:00.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:00.005+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:00.007649+0000) 2024-04-27T03:53:00.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:00.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:00.304 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:00.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:00.305 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:00.081+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:00.305 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:00.082+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:00.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:00.083+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:00.084964+0000) 2024-04-27T03:53:00.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:00.298+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:00.299994+0000) 2024-04-27T03:53:00.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:00.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:00.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:00.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:00.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:00.694+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:00.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:00.696+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:01.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:01.085+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:01.087666+0000) 2024-04-27T03:53:01.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:01.105+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:01.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:01.105+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:01.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:01.105+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:01.306 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:01.106+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:01.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:01.008+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:01.009848+0000) 2024-04-27T03:53:01.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:01.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:01.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:01.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:01.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:01.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:01.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:01.047+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:01.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:01.301+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:01.302627+0000) 2024-04-27T03:53:01.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:01.650+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:01.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:01.650+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:01.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:01.650+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:01.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:01.651+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:02.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:02.090+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:02.090933+0000) 2024-04-27T03:53:02.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:02.014+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:02.015023+0000) 2024-04-27T03:53:02.310 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:02.028+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:02.310 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:02.028+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:02.310 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:02.028+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:02.310 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:02.029+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:02.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:02.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:02.311 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:02.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:02.311 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:02.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:02.311 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:02.109+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:02.666 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:02.305+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:02.305923+0000) 2024-04-27T03:53:02.974 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:02.775+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:02.776624+0000) 2024-04-27T03:53:02.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:02.776+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:02.777022+0000) 2024-04-27T03:53:02.975 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:53:02.778+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74946 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:53:02.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:02.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:02.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:02.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:02.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:02.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:02.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:02.663+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:02.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:02.775+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:02.776172+0000) 2024-04-27T03:53:03.311 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:03.092+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:03.093307+0000) 2024-04-27T03:53:03.311 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:03.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:03.312 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:03.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:03.312 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:03.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:03.312 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:03.004+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:03.312 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:03.017+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:03.018165+0000) 2024-04-27T03:53:03.312 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:03.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:03.312 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:03.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:03.313 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:03.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:03.313 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:03.073+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:03.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:03.307+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:03.308288+0000) 2024-04-27T03:53:03.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:03.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:03.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:03.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:03.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:03.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:03.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:03.706+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:04.316 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:04.094+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:04.095393+0000) 2024-04-27T03:53:04.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:04.019+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:04.020237+0000) 2024-04-27T03:53:04.317 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:04.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:04.317 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:04.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:04.317 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:04.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:04.317 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:04.041+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:04.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:04.073+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:04.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:04.073+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:04.318 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:04.073+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:04.318 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:04.074+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:04.318 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:04.312+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:04.313093+0000) 2024-04-27T03:53:05.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:04.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:05.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:04.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:05.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:04.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:05.029 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:04.725+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:05.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:05.099+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:05.100430+0000) 2024-04-27T03:53:05.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:05.024+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:05.025166+0000) 2024-04-27T03:53:05.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:05.064+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:05.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:05.064+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:05.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:05.064+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:05.319 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:05.065+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:05.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:05.062+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:05.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:05.062+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:05.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:05.062+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:05.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:05.062+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:05.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:05.314+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:05.315569+0000) 2024-04-27T03:53:06.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:05.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:06.034 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:05.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:06.034 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:05.752+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:06.034 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:05.753+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:06.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:06.104+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:06.105494+0000) 2024-04-27T03:53:06.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:06.029+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:06.030269+0000) 2024-04-27T03:53:06.321 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:06.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:06.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:06.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:06.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:06.072+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:06.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:06.073+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:06.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:06.075+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:06.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:06.075+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:06.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:06.075+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:06.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:06.076+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:06.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:06.317+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:06.318050+0000) 2024-04-27T03:53:07.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:06.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:07.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:06.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:07.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:06.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:07.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:06.761+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:07.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:07.109+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:07.110463+0000) 2024-04-27T03:53:07.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:07.095+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:07.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:07.095+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:07.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:07.095+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:07.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:07.096+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:07.325 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:07.034+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:07.035234+0000) 2024-04-27T03:53:07.325 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:07.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:07.325 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:07.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:07.325 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:07.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:07.326 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:07.056+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:07.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:07.320+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:07.321242+0000) 2024-04-27T03:53:08.042 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:53:07.787+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 74987 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:53:08.043 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:07.783+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:07.784133+0000) 2024-04-27T03:53:08.043 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:07.784+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:07.784902+0000) 2024-04-27T03:53:08.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:07.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:08.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:07.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:08.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:07.754+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:08.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:07.755+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:08.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:07.782+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:07.783393+0000) 2024-04-27T03:53:08.328 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:08.114+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:08.115612+0000) 2024-04-27T03:53:08.329 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:08.039+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:08.039837+0000) 2024-04-27T03:53:08.329 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:08.079+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:08.329 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:08.080+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:08.329 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:08.080+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:08.329 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:08.081+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:08.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:08.129+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:08.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:08.129+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:08.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:08.129+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:08.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:08.129+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:08.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:08.324+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:08.325530+0000) 2024-04-27T03:53:09.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:08.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:09.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:08.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:09.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:08.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:09.045 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:08.740+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:09.333 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:09.119+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:09.120685+0000) 2024-04-27T03:53:09.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:09.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:09.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:09.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:09.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:09.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:09.334 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:09.146+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:09.334 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:09.041+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:09.041827+0000) 2024-04-27T03:53:09.335 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:09.125+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:09.335 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:09.125+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:09.335 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:09.125+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:09.335 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:09.127+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:09.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:09.329+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:09.330514+0000) 2024-04-27T03:53:09.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:09.709+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:09.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:09.709+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:09.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:09.709+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:09.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:09.710+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:10.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:10.043+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:10.044052+0000) 2024-04-27T03:53:10.339 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:10.096+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:10.339 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:10.096+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:10.339 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:10.096+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:10.339 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:10.098+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:10.339 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:10.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:10.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:10.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:10.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:10.104+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:10.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:10.104+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:10.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:10.334+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:10.335543+0000) 2024-04-27T03:53:10.340 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:10.124+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:10.125777+0000) 2024-04-27T03:53:11.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:10.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:11.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:10.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:11.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:10.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:11.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:10.759+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:11.339 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:11.130+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:11.130923+0000) 2024-04-27T03:53:11.339 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:11.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:11.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:11.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:11.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:11.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:11.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:11.124+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:11.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:11.045+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:11.046732+0000) 2024-04-27T03:53:11.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:11.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:11.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:11.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:11.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:11.093+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:11.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:11.094+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:11.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:11.340+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:11.341023+0000) 2024-04-27T03:53:11.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:11.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:11.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:11.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:11.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:11.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:11.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:11.717+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:12.339 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:12.132+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:12.133063+0000) 2024-04-27T03:53:12.339 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:12.049+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:12.049842+0000) 2024-04-27T03:53:12.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:12.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:12.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:12.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:12.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:12.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:12.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:12.112+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:12.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:12.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:12.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:12.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:12.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:12.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:12.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:12.100+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:12.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:12.342+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:12.343172+0000) 2024-04-27T03:53:13.050 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:53:12.797+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75026 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:53:13.050 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:12.795+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:12.796428+0000) 2024-04-27T03:53:13.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:12.792+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:12.792908+0000) 2024-04-27T03:53:13.051 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:12.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:13.051 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:12.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:13.051 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:12.726+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:13.051 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:12.727+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:13.052 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:12.792+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:12.793040+0000) 2024-04-27T03:53:13.339 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:13.134+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:13.135404+0000) 2024-04-27T03:53:13.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:13.051+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:13.051975+0000) 2024-04-27T03:53:13.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:13.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:13.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:13.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:13.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:13.091+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:13.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:13.092+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:13.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:13.097+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:13.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:13.097+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:13.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:13.097+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:13.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:13.098+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:13.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:13.345+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:13.346274+0000) 2024-04-27T03:53:13.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:13.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:13.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:13.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:13.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:13.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:13.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:13.694+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:14.339 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:14.137+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:14.138239+0000) 2024-04-27T03:53:14.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:14.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:14.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:14.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:14.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:14.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:14.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:14.052+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:14.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:14.045+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:14.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:14.045+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:14.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:14.045+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:14.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:14.045+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:14.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:14.053+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:14.054619+0000) 2024-04-27T03:53:14.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:14.347+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:14.348465+0000) 2024-04-27T03:53:14.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:14.695+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:14.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:14.695+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:14.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:14.695+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:14.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:14.696+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:15.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:15.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:15.277 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:15.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:15.277 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:15.011+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:15.277 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:15.011+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:15.277 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:15.056+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:15.057717+0000) 2024-04-27T03:53:15.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:15.140+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:15.140924+0000) 2024-04-27T03:53:15.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:15.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:15.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:15.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:15.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:15.052+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:15.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:15.053+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:15.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:15.351+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:15.352670+0000) 2024-04-27T03:53:15.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:15.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:15.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:15.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:15.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:15.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:15.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:15.716+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:16.339 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:16.142+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:16.143205+0000) 2024-04-27T03:53:16.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:16.044+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:16.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:16.044+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:16.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:16.044+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:16.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:16.045+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:16.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:16.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:16.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:16.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:16.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:16.047+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:16.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:16.048+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:16.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:16.059+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:16.059838+0000) 2024-04-27T03:53:16.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:16.354+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:16.355533+0000) 2024-04-27T03:53:17.023 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:16.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:17.023 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:16.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:17.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:16.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:17.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:16.751+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:17.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:17.144+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:17.145457+0000) 2024-04-27T03:53:17.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:17.019+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:17.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:17.019+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:17.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:17.019+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:17.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:17.020+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:17.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:17.061+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:17.062476+0000) 2024-04-27T03:53:17.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:17.086+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:17.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:17.086+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:17.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:17.086+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:17.279 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:17.087+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:17.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:17.356+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:17.357783+0000) 2024-04-27T03:53:17.930 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:53:17.806+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75067 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:53:17.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:17.802+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:17.803108+0000) 2024-04-27T03:53:17.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:17.801+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:17.801875+0000) 2024-04-27T03:53:17.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:17.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:17.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:17.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:17.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:17.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:17.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:17.725+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:17.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:17.802+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:17.802959+0000) 2024-04-27T03:53:18.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:18.146+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:18.147755+0000) 2024-04-27T03:53:18.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:17.979+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:18.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:17.979+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:18.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:17.979+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:18.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:17.979+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:18.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:18.063+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:18.064792+0000) 2024-04-27T03:53:18.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:18.083+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:18.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:18.083+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:18.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:18.083+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:18.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:18.085+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:18.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:18.359+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:18.360660+0000) 2024-04-27T03:53:18.995 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:18.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:18.995 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:18.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:18.995 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:18.739+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:18.995 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:18.740+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:19.280 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:19.066+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:19.067449+0000) 2024-04-27T03:53:19.280 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:19.097+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:19.280 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:19.097+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:19.280 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:19.097+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:19.280 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:19.097+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:19.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:18.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:19.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:18.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:19.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:18.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:19.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:18.991+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:19.282 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:19.149+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:19.149913+0000) 2024-04-27T03:53:19.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:19.362+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:19.362856+0000) 2024-04-27T03:53:19.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:19.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:19.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:19.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:19.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:19.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:19.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:19.697+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:20.340 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:20.151+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:20.152083+0000) 2024-04-27T03:53:20.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:20.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:20.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:20.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:20.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:20.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:20.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:20.031+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:20.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:20.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:20.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:20.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:20.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:20.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:20.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:20.056+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:20.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:20.068+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:20.069676+0000) 2024-04-27T03:53:20.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:20.364+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:20.365174+0000) 2024-04-27T03:53:20.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:20.709+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:20.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:20.709+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:20.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:20.709+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:20.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:20.710+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:21.340 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:21.153+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:21.154220+0000) 2024-04-27T03:53:21.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:21.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:21.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:21.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:21.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:21.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:21.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:21.066+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:21.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:21.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:21.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:21.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:21.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:21.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:21.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:21.043+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:21.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:21.072+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:21.072914+0000) 2024-04-27T03:53:21.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:21.366+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:21.367301+0000) 2024-04-27T03:53:21.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:21.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:21.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:21.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:21.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:21.677+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:21.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:21.679+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:22.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:22.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:22.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:22.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:22.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:22.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:22.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:22.054+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:22.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:22.074+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:22.075031+0000) 2024-04-27T03:53:22.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:22.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:22.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:22.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:22.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:22.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:22.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:22.044+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:22.342 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:22.157+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:22.158607+0000) 2024-04-27T03:53:22.659 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:22.369+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:22.370520+0000) 2024-04-27T03:53:22.936 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:53:22.814+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75106 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:53:22.937 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:22.810+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:22.811173+0000) 2024-04-27T03:53:22.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:22.811+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:22.812285+0000) 2024-04-27T03:53:22.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:22.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:22.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:22.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:22.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:22.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:22.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:22.684+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:22.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:22.811+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:22.811991+0000) 2024-04-27T03:53:23.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:23.159+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:23.160689+0000) 2024-04-27T03:53:23.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:23.076+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:23.077749+0000) 2024-04-27T03:53:23.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:23.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:23.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:23.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:23.224 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:23.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:23.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:23.086+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:23.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:23.046+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:23.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:23.046+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:23.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:23.046+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:23.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:23.046+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:23.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:23.371+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:23.372640+0000) 2024-04-27T03:53:23.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:23.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:23.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:23.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:23.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:23.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:23.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:23.643+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:24.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:24.162+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:24.163528+0000) 2024-04-27T03:53:24.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:24.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:24.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:24.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:24.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:24.001+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:24.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:24.002+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:24.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:24.080+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:24.080967+0000) 2024-04-27T03:53:24.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:24.112+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:24.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:24.112+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:24.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:24.112+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:24.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:24.113+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:24.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:24.374+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:24.375418+0000) 2024-04-27T03:53:24.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:24.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:24.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:24.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:24.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:24.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:24.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:24.658+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:25.289 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:25.164+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:25.165650+0000) 2024-04-27T03:53:25.289 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:25.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:25.290 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:25.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:25.290 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:25.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:25.290 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:25.025+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:25.290 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:25.082+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:25.083106+0000) 2024-04-27T03:53:25.290 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:25.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:25.290 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:25.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:25.291 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:25.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:25.291 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:25.109+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:25.621 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:25.376+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:25.377563+0000) 2024-04-27T03:53:25.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:25.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:25.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:25.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:25.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:25.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:25.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:25.618+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:26.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:26.066+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:26.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:26.066+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:26.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:26.066+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:26.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:26.067+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:26.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:26.085+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:26.086079+0000) 2024-04-27T03:53:26.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:26.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:26.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:26.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:26.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:26.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:26.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:26.158+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:26.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:26.168+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:26.168922+0000) 2024-04-27T03:53:26.596 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:26.379+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:26.380708+0000) 2024-04-27T03:53:26.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:26.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:26.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:26.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:26.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:26.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:26.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:26.593+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:27.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:27.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:27.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:27.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:27.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:27.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:27.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:27.096+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:27.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:27.087+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:27.088338+0000) 2024-04-27T03:53:27.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:27.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:27.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:27.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:27.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:27.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:27.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:27.110+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:27.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:27.170+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:27.171062+0000) 2024-04-27T03:53:27.640 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:27.382+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:27.382933+0000) 2024-04-27T03:53:27.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:27.821+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:27.822061+0000) 2024-04-27T03:53:27.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:27.820+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:27.820860+0000) 2024-04-27T03:53:27.949 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:53:27.823+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75147 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:53:27.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:27.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:27.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:27.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:27.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:27.636+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:27.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:27.638+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:27.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:27.819+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:27.820332+0000) 2024-04-27T03:53:28.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:28.172+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:28.173226+0000) 2024-04-27T03:53:28.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:28.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:28.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:28.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:28.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:28.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:28.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:28.127+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:28.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:28.089+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:28.090479+0000) 2024-04-27T03:53:28.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:28.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:28.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:28.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:28.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:28.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:28.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:28.130+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:28.621 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:28.386+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:28.387229+0000) 2024-04-27T03:53:28.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:28.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:28.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:28.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:28.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:28.617+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:28.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:28.619+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:29.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:29.175+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:29.176234+0000) 2024-04-27T03:53:29.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:29.144+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:29.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:29.144+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:29.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:29.144+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:29.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:29.145+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:29.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:29.092+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:29.093302+0000) 2024-04-27T03:53:29.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:29.143+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:29.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:29.143+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:29.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:29.143+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:29.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:29.143+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:29.636 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:29.388+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:29.389348+0000) 2024-04-27T03:53:29.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:29.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:29.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:29.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:29.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:29.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:29.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:29.633+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:30.342 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:30.177+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:30.178561+0000) 2024-04-27T03:53:30.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:30.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:30.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:30.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:30.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:30.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:30.343 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:30.147+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:30.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:30.094+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:30.095433+0000) 2024-04-27T03:53:30.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:30.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:30.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:30.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:30.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:30.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:30.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:30.142+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:30.593 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:30.391+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:30.392390+0000) 2024-04-27T03:53:30.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:30.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:30.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:30.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:30.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:30.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:30.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:30.591+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:31.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:31.180+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:31.181669+0000) 2024-04-27T03:53:31.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:31.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:31.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:31.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:31.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:31.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:31.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:31.122+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:31.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:31.099+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:31.100550+0000) 2024-04-27T03:53:31.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:31.151+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:31.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:31.151+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:31.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:31.151+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:31.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:31.152+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:31.617 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:31.395+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:31.396554+0000) 2024-04-27T03:53:31.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:31.613+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:31.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:31.613+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:31.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:31.613+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:31.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:31.614+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:32.341 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:32.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:32.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:32.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:32.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:32.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:32.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:32.152+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:32.342 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:32.182+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:32.183779+0000) 2024-04-27T03:53:32.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:32.104+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:32.105700+0000) 2024-04-27T03:53:32.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:32.199+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:32.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:32.199+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:32.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:32.199+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:32.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:32.200+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:32.596 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:32.398+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:32.398832+0000) 2024-04-27T03:53:32.857 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:53:32.833+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75186 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:53:32.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:32.829+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:32.830531+0000) 2024-04-27T03:53:32.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:32.829+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:32.830009+0000) 2024-04-27T03:53:32.858 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:32.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:32.858 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:32.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:32.858 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:32.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:32.859 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:32.593+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:32.859 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:32.829+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:32.829854+0000) 2024-04-27T03:53:33.405 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:33.110+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:33.110823+0000) 2024-04-27T03:53:33.405 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:33.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:33.405 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:33.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:33.405 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:33.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:33.406 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:33.157+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:33.406 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:33.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:33.406 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:33.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:33.406 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:33.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:33.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:33.171+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:33.407 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:33.401+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:33.402058+0000) 2024-04-27T03:53:33.407 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:33.187+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:33.188128+0000) 2024-04-27T03:53:33.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:33.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:33.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:33.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:33.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:33.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:33.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:33.625+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:34.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:34.112+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:34.113445+0000) 2024-04-27T03:53:34.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:34.181+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:34.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:34.181+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:34.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:34.181+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:34.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:34.182+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:34.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:34.190+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:34.191192+0000) 2024-04-27T03:53:34.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:34.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:34.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:34.219+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:34.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:34.219+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:34.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:34.219+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:34.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:34.404+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:34.405227+0000) 2024-04-27T03:53:34.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:34.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:34.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:34.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:34.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:34.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:34.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:34.659+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:35.308 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:35.192+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:35.193772+0000) 2024-04-27T03:53:35.308 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:35.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:35.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:35.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:35.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:35.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:35.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:35.215+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:35.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:35.115+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:35.116302+0000) 2024-04-27T03:53:35.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:35.232+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:35.309 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:35.232+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:35.310 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:35.232+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:35.310 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:35.233+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:35.682 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:35.407+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:35.407867+0000) 2024-04-27T03:53:35.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:35.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:35.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:35.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:35.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:35.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:35.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:35.680+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:36.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:36.196+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:36.197049+0000) 2024-04-27T03:53:36.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:36.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:36.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:36.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:36.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:36.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:36.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:36.257+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:36.310 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:36.118+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:36.119317+0000) 2024-04-27T03:53:36.310 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:36.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:36.310 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:36.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:36.310 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:36.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:36.311 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:36.194+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:36.706 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:36.410+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:36.411061+0000) 2024-04-27T03:53:36.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:36.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:36.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:36.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:36.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:36.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:36.974 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:36.703+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:37.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:37.200+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:37.201465+0000) 2024-04-27T03:53:37.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:37.229+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:37.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:37.229+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:37.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:37.229+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:37.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:37.229+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:37.310 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:37.122+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:37.123502+0000) 2024-04-27T03:53:37.311 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:37.236+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:37.311 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:37.236+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:37.311 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:37.236+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:37.311 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:37.236+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:37.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:37.412+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:37.413745+0000) 2024-04-27T03:53:38.052 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:53:37.842+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75227 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:53:38.053 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:37.838+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:37.839095+0000) 2024-04-27T03:53:38.053 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:37.838+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:37.839017+0000) 2024-04-27T03:53:38.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:37.742+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:38.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:37.742+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:38.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:37.742+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:38.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:37.742+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:38.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:37.838+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:37.839122+0000) 2024-04-27T03:53:38.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:38.203+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:38.204422+0000) 2024-04-27T03:53:38.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:38.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:38.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:38.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:38.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:38.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:38.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:38.199+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:38.311 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:38.126+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:38.127564+0000) 2024-04-27T03:53:38.311 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:38.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:38.311 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:38.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:38.311 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:38.277+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:38.311 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:38.279+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:38.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:38.416+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:38.417095+0000) 2024-04-27T03:53:39.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:38.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:39.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:38.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:39.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:38.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:39.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:38.724+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:39.307 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:39.129+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:39.130223+0000) 2024-04-27T03:53:39.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:39.206+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:39.207311+0000) 2024-04-27T03:53:39.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:39.189+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:39.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:39.189+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:39.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:39.189+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:39.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:39.190+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:39.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:39.418+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:39.419340+0000) 2024-04-27T03:53:39.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:39.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:39.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:39.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:39.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:39.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:39.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:39.305+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:40.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:39.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:40.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:39.750+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:40.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:39.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:40.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:39.752+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:40.314 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:40.132+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:40.132913+0000) 2024-04-27T03:53:40.315 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:40.209+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:40.210498+0000) 2024-04-27T03:53:40.315 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:40.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:40.315 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:40.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:40.316 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:40.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:40.316 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:40.199+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:40.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:40.420+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:40.421488+0000) 2024-04-27T03:53:40.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:40.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:40.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:40.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:40.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:40.322+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:40.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:40.322+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:41.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:40.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:41.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:40.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:41.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:40.791+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:41.053 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:40.792+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:41.316 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:41.134+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:41.135110+0000) 2024-04-27T03:53:41.316 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:41.211+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:41.212727+0000) 2024-04-27T03:53:41.316 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:41.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:41.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:41.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:41.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:41.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:41.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:41.174+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:41.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:41.423+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:41.424326+0000) 2024-04-27T03:53:41.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:41.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:41.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:41.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:41.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:41.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:41.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:41.350+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:42.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:41.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:42.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:41.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:42.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:41.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:42.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:41.814+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:42.430 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:42.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:42.430 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:42.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:42.430 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:42.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:42.431 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:42.205+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:42.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:42.136+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:42.137219+0000) 2024-04-27T03:53:42.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:42.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:42.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:42.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:42.431 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:42.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:42.432 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:42.383+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:42.432 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:42.214+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:42.214882+0000) 2024-04-27T03:53:42.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:42.426+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:42.427284+0000) 2024-04-27T03:53:43.142 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:53:42.850+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75266 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:53:43.143 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:42.848+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:42.849067+0000) 2024-04-27T03:53:43.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:42.847+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:42.848225+0000) 2024-04-27T03:53:43.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:42.843+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:43.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:42.843+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:43.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:42.843+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:43.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:42.843+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:43.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:42.847+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:42.848500+0000) 2024-04-27T03:53:43.401 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:43.138+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:43.139725+0000) 2024-04-27T03:53:43.401 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:43.218+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:43.219276+0000) 2024-04-27T03:53:43.401 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:43.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:43.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:43.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:43.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:43.170+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:43.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:43.170+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:43.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:43.429+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:43.430103+0000) 2024-04-27T03:53:43.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:43.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:43.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:43.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:43.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:43.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:43.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:43.398+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:44.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:43.843+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:44.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:43.843+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:44.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:43.843+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:44.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:43.843+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:44.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:44.141+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:44.142249+0000) 2024-04-27T03:53:44.425 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:44.222+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:44.223377+0000) 2024-04-27T03:53:44.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:44.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:44.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:44.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:44.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:44.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:44.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:44.199+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:44.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:44.431+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:44.432665+0000) 2024-04-27T03:53:44.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:44.421+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:44.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:44.421+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:44.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:44.421+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:44.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:44.422+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:45.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:44.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:45.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:44.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:45.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:44.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:45.148 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:44.861+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:45.438 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:45.227+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:45.227946+0000) 2024-04-27T03:53:45.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:45.144+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:45.145059+0000) 2024-04-27T03:53:45.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:45.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:45.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:45.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:45.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:45.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:45.439 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:45.375+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:45.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:45.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:45.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:45.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:45.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:45.178+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:45.440 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:45.179+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:45.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:45.435+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:45.435901+0000) 2024-04-27T03:53:46.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:45.840+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:46.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:45.840+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:46.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:45.840+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:46.152 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:45.841+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:46.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:46.229+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:46.230569+0000) 2024-04-27T03:53:46.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:46.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:46.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:46.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:46.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:46.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:46.442 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:46.228+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:46.442 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:46.148+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:46.149097+0000) 2024-04-27T03:53:46.442 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:46.393+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:46.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:46.393+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:46.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:46.393+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:46.443 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:46.394+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:46.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:46.438+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:46.438797+0000) 2024-04-27T03:53:47.155 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:46.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:47.155 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:46.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:47.155 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:46.820+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:47.155 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:46.821+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:47.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:47.152+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:47.153758+0000) 2024-04-27T03:53:47.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:47.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:47.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:47.232+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:47.233724+0000) 2024-04-27T03:53:47.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:47.234+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:47.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:47.234+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:47.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:47.234+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:47.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:47.235+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:47.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:47.440+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:47.441382+0000) 2024-04-27T03:53:47.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:47.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:47.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:47.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:47.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:47.436+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:48.155 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:53:47.859+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75307 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:53:48.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:47.857+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:47.857971+0000) 2024-04-27T03:53:48.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:47.855+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:47.855825+0000) 2024-04-27T03:53:48.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:47.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:48.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:47.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:48.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:47.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:48.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:47.812+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:48.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:47.855+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:47.856620+0000) 2024-04-27T03:53:48.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:48.235+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:48.236409+0000) 2024-04-27T03:53:48.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:48.157+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:48.158015+0000) 2024-04-27T03:53:48.435 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:48.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:48.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:48.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:48.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:48.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:48.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:48.241+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:48.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:48.241+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:48.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:48.241+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:48.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:48.242+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:48.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:48.442+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:48.443600+0000) 2024-04-27T03:53:48.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:48.432+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:49.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:48.778+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:49.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:48.778+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:49.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:48.778+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:49.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:48.779+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:49.328 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:49.159+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:49.160197+0000) 2024-04-27T03:53:49.329 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:49.237+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:49.238661+0000) 2024-04-27T03:53:49.329 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:49.245+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:49.329 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:49.245+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:49.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:49.245+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:49.330 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:49.246+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:49.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:49.445+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:49.446391+0000) 2024-04-27T03:53:49.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:49.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:49.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:49.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:49.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:49.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:49.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:49.421+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:50.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:49.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:50.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:49.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:50.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:49.804+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:50.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:49.805+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:50.451 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:50.241+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:50.241831+0000) 2024-04-27T03:53:50.451 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:50.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:50.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:50.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:50.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:50.215+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:50.452 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:50.215+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:50.452 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:50.162+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:50.162900+0000) 2024-04-27T03:53:50.452 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:50.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:50.452 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:50.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:50.453 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:50.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:50.453 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:50.374+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:50.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:50.446+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:50.448618+0000) 2024-04-27T03:53:51.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:50.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:51.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:50.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:51.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:50.805+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:51.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:50.806+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:51.454 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:51.244+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:51.246160+0000) 2024-04-27T03:53:51.455 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:51.164+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:51.166110+0000) 2024-04-27T03:53:51.455 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:51.361+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:51.456 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:51.361+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:51.456 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:51.361+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:51.456 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:51.362+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:51.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:51.245+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:51.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:51.245+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:51.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:51.245+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:51.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:51.245+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:51.458 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:51.450+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:51.451841+0000) 2024-04-27T03:53:52.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:51.855+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:52.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:51.855+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:52.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:51.855+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:52.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:51.856+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:52.457 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:52.248+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:52.250638+0000) 2024-04-27T03:53:52.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:52.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:52.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:52.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:52.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:52.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:52.457 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:52.257+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:52.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:52.167+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:52.168899+0000) 2024-04-27T03:53:52.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:52.317+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:52.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:52.317+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:52.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:52.317+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:52.458 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:52.318+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:52.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:52.452+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:52.454302+0000) 2024-04-27T03:53:53.156 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:53:52.867+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75346 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:53:53.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:52.863+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:52.864996+0000) 2024-04-27T03:53:53.157 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:52.864+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:52.866560+0000) 2024-04-27T03:53:53.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:52.863+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:52.865416+0000) 2024-04-27T03:53:53.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:52.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:53.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:52.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:53.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:52.886+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:53.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:52.887+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:53.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:53.170+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:53.172111+0000) 2024-04-27T03:53:53.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:53.294+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:53.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:53.294+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:53.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:53.294+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:53.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:53.294+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:53.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:53.290+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:53.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:53.290+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:53.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:53.290+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:53.461 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:53.291+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:53.462 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:53.251+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:53.252909+0000) 2024-04-27T03:53:53.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:53.455+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:53.457235+0000) 2024-04-27T03:53:54.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:53.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:54.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:53.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:54.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:53.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:54.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:53.884+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:54.462 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:54.255+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:54.257185+0000) 2024-04-27T03:53:54.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:54.245+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:54.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:54.245+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:54.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:54.245+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:54.463 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:54.246+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:54.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:54.175+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:54.177050+0000) 2024-04-27T03:53:54.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:54.337+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:54.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:54.337+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:54.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:54.337+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:54.464 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:54.338+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:54.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:54.458+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:54.460307+0000) 2024-04-27T03:53:55.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:54.875+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:55.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:54.875+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:55.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:54.875+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:55.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:54.875+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:55.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:55.180+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:55.182090+0000) 2024-04-27T03:53:55.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:55.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:55.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:55.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:55.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:55.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:55.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:55.305+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:55.467 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:55.291+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:55.467 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:55.291+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:55.467 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:55.291+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:55.468 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:55.291+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:55.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:55.258+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:55.260179+0000) 2024-04-27T03:53:55.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:55.461+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:55.463545+0000) 2024-04-27T03:53:56.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:55.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:56.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:55.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:56.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:55.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:56.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:55.852+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:56.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:56.262+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:56.263902+0000) 2024-04-27T03:53:56.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:56.321+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:56.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:56.321+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:56.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:56.321+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:56.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:56.322+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:56.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:56.184+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:56.186319+0000) 2024-04-27T03:53:56.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:56.324+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:56.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:56.324+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:56.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:56.324+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:56.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:56.326+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:56.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:56.464+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:56.466568+0000) 2024-04-27T03:53:57.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:56.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:57.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:56.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:57.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:56.841+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:57.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:56.842+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:57.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:57.264+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:57.266080+0000) 2024-04-27T03:53:57.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:57.186+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:57.188522+0000) 2024-04-27T03:53:57.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:57.298+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:57.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:57.298+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:57.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:57.298+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:57.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:57.298+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:57.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:57.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:57.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:57.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:57.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:57.352+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:57.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:57.353+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:57.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:57.468+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:57.469799+0000) 2024-04-27T03:53:58.077 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:53:57.875+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75387 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:53:58.077 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:57.872+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:57.874362+0000) 2024-04-27T03:53:58.078 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:57.872+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:57.874338+0000) 2024-04-27T03:53:58.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:57.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:58.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:57.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:58.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:57.812+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:58.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:57.813+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:58.079 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:57.871+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:57.873670+0000) 2024-04-27T03:53:58.339 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:58.189+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:58.191566+0000) 2024-04-27T03:53:58.339 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:58.266+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:58.268732+0000) 2024-04-27T03:53:58.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:58.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:58.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:58.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:58.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:58.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:58.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:58.335+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:58.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:58.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:58.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:58.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:58.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:58.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:58.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:58.342+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:58.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:58.470+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:58.472679+0000) 2024-04-27T03:53:59.156 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:58.831+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:59.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:58.831+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:59.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:58.831+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:53:59.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:58.832+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:59.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:59.271+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:59.273389+0000) 2024-04-27T03:53:59.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:59.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:59.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:59.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:59.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:59.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:53:59.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:59.388+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:59.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:59.192+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:59.193819+0000) 2024-04-27T03:53:59.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:59.374+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:59.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:59.374+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:59.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:59.374+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:53:59.475 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:53:59.375+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:53:59.876 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:53:59.472+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:53:59.474724+0000) 2024-04-27T03:54:00.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:59.872+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:00.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:59.872+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:00.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:59.872+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:00.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:53:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:53:59.873+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:00.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:00.276+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:00.278082+0000) 2024-04-27T03:54:00.440 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:00.194+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:00.196611+0000) 2024-04-27T03:54:00.440 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:00.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:00.440 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:00.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:00.440 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:00.364+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:00.440 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:00.366+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:00.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:00.435+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:00.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:00.435+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:00.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:00.435+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:00.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:00.435+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:00.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:00.477+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:00.478932+0000) 2024-04-27T03:54:01.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:00.848+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:01.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:00.848+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:01.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:00.848+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:01.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:00.849+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:01.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:01.278+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:01.280740+0000) 2024-04-27T03:54:01.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:01.196+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:01.198760+0000) 2024-04-27T03:54:01.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:01.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:01.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:01.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:01.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:01.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:01.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:01.359+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:01.893 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:01.478+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:01.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:01.478+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:01.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:01.478+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:01.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:01.479+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:01.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:01.481+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:01.483192+0000) 2024-04-27T03:54:02.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:01.890+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:02.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:01.890+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:02.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:01.890+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:02.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:01.890+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:02.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:02.283+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:02.283983+0000) 2024-04-27T03:54:02.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:02.201+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:02.203540+0000) 2024-04-27T03:54:02.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:02.337+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:02.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:02.337+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:02.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:02.337+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:02.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:02.338+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:02.869 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:02.484+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:02.485398+0000) 2024-04-27T03:54:02.870 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:02.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:02.870 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:02.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:02.870 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:02.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:02.870 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:02.526+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:03.122 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:54:02.885+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75427 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:54:03.122 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:02.881+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:02.882693+0000) 2024-04-27T03:54:03.123 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:02.881+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:02.882030+0000) 2024-04-27T03:54:03.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:02.866+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:03.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:02.866+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:03.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:02.866+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:03.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:02.867+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:03.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:02.881+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:02.882151+0000) 2024-04-27T03:54:03.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:03.286+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:03.287455+0000) 2024-04-27T03:54:03.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:03.204+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:03.205759+0000) 2024-04-27T03:54:03.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:03.343+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:03.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:03.343+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:03.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:03.343+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:03.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:03.343+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:03.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:03.488+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:03.489792+0000) 2024-04-27T03:54:03.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:03.543+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:03.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:03.543+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:03.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:03.543+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:03.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:03.544+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:04.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:03.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:04.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:03.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:04.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:03.908+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:04.211 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:03.909+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:04.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:04.289+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:04.290309+0000) 2024-04-27T03:54:04.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:04.207+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:04.207882+0000) 2024-04-27T03:54:04.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:04.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:04.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:04.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:04.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:04.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:04.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:04.346+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:04.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:04.493+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:04.494003+0000) 2024-04-27T03:54:04.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:04.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:04.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:04.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:04.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:04.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:04.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:04.519+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:05.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:04.941+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:05.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:04.941+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:05.214 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:04.941+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:05.214 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:04.943+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:05.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:05.293+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:05.294659+0000) 2024-04-27T03:54:05.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:05.209+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:05.210792+0000) 2024-04-27T03:54:05.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:05.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:05.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:05.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:05.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:05.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:05.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:05.338+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:05.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:05.495+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:05.496657+0000) 2024-04-27T03:54:05.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:05.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:05.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:05.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:05.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:05.508+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:05.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:05.509+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:06.299 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:06.212+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:06.213165+0000) 2024-04-27T03:54:06.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:05.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:06.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:05.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:06.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:05.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:06.300 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:05.979+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:06.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:06.295+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:06.296716+0000) 2024-04-27T03:54:06.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:06.311+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:06.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:06.311+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:06.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:06.311+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:06.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:06.311+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:06.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:06.500+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:06.501327+0000) 2024-04-27T03:54:06.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:06.509+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:06.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:06.509+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:06.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:06.509+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:06.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:06.509+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:07.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:07.215+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:07.216172+0000) 2024-04-27T03:54:07.303 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:06.981+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:07.303 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:06.981+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:07.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:06.981+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:07.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:06.982+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:07.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:07.299+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:07.300500+0000) 2024-04-27T03:54:07.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:07.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:07.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:07.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:07.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:07.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:07.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:07.308+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:07.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:07.503+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:07.504398+0000) 2024-04-27T03:54:07.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:07.524+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:07.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:07.524+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:07.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:07.524+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:07.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:07.525+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:08.157 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:54:07.893+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75468 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:54:08.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:07.889+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:07.890558+0000) 2024-04-27T03:54:08.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:07.890+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:07.891461+0000) 2024-04-27T03:54:08.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:07.890+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:07.891621+0000) 2024-04-27T03:54:08.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:07.972+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:08.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:07.972+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:08.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:07.972+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:08.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:07.973+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:08.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:08.302+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:08.302797+0000) 2024-04-27T03:54:08.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:08.217+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:08.218321+0000) 2024-04-27T03:54:08.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:08.296+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:08.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:08.296+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:08.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:08.296+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:08.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:08.297+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:08.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:08.506+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:08.507268+0000) 2024-04-27T03:54:08.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:08.570+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:08.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:08.570+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:08.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:08.570+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:08.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:08.571+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:09.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:08.961+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:09.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:08.961+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:09.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:08.961+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:09.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:08.962+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:09.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:09.304+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:09.304841+0000) 2024-04-27T03:54:09.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:09.219+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:09.220579+0000) 2024-04-27T03:54:09.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:09.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:09.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:09.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:09.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:09.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:09.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:09.283+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:09.957 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:09.508+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:09.509474+0000) 2024-04-27T03:54:09.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:09.574+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:09.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:09.574+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:09.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:09.574+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:09.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:09.574+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:10.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:09.954+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:10.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:09.954+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:10.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:09.954+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:10.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:09.955+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:10.515 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:10.306+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:10.306957+0000) 2024-04-27T03:54:10.515 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:10.222+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:10.223300+0000) 2024-04-27T03:54:10.515 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:10.258+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:10.515 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:10.258+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:10.515 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:10.258+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:10.515 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:10.259+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:10.943 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:10.511+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:10.512459+0000) 2024-04-27T03:54:10.943 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:10.554+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:10.943 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:10.554+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:10.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:10.554+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:10.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:10.554+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:11.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:10.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:11.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:10.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:11.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:10.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:11.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:10.941+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:11.519 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:11.310+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:11.311296+0000) 2024-04-27T03:54:11.520 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:11.225+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:11.225919+0000) 2024-04-27T03:54:11.520 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:11.298+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:11.520 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:11.298+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:11.520 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:11.298+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:11.521 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:11.299+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:11.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:11.516+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:11.517182+0000) 2024-04-27T03:54:11.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:11.599+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:11.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:11.599+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:11.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:11.599+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:11.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:11.599+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:12.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:11.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:12.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:11.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:12.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:11.892+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:12.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:11.893+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:12.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:12.312+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:12.313390+0000) 2024-04-27T03:54:12.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:12.227+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:12.228233+0000) 2024-04-27T03:54:12.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:12.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:12.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:12.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:12.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:12.336+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:12.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:12.337+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:12.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:12.518+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:12.519543+0000) 2024-04-27T03:54:12.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:12.597+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:12.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:12.597+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:12.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:12.597+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:12.900 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:12.597+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:13.158 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:54:12.902+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75507 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:54:13.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:12.899+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:12.899898+0000) 2024-04-27T03:54:13.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:12.898+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:12.899075+0000) 2024-04-27T03:54:13.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:12.899+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:12.899893+0000) 2024-04-27T03:54:13.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:12.939+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:13.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:12.939+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:13.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:12.939+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:13.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:12.940+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:13.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:13.315+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:13.316380+0000) 2024-04-27T03:54:13.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:13.229+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:13.230365+0000) 2024-04-27T03:54:13.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:13.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:13.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:13.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:13.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:13.306+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:13.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:13.307+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:13.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:13.521+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:13.521827+0000) 2024-04-27T03:54:13.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:13.556+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:13.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:13.556+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:13.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:13.556+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:13.920 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:13.556+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:14.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:13.916+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:14.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:13.916+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:14.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:13.916+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:14.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:13.917+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:14.526 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:14.317+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:14.318619+0000) 2024-04-27T03:54:14.527 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:14.232+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:14.233305+0000) 2024-04-27T03:54:14.527 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:14.263+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:14.527 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:14.263+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:14.527 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:14.263+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:14.527 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:14.263+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:14.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:14.523+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:14.524442+0000) 2024-04-27T03:54:14.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:14.564+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:14.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:14.564+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:14.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:14.564+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:14.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:14.565+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:15.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:14.966+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:15.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:14.966+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:15.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:14.966+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:15.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:14.967+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:15.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:15.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:15.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:15.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:15.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:15.216+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:15.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:15.217+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:15.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:15.319+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:15.320757+0000) 2024-04-27T03:54:15.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:15.234+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:15.235411+0000) 2024-04-27T03:54:15.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:15.526+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:15.526847+0000) 2024-04-27T03:54:15.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:15.538+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:15.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:15.538+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:15.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:15.538+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:15.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:15.538+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:16.266 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:16.007+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:16.267 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:16.007+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:16.267 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:16.007+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:16.267 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:16.008+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:16.267 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:16.236+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:16.237582+0000) 2024-04-27T03:54:16.267 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:16.263+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:16.267 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:16.263+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:16.268 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:16.263+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:16.532 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:16.264+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:16.532 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:16.322+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:16.323420+0000) 2024-04-27T03:54:16.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:16.528+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:16.529500+0000) 2024-04-27T03:54:16.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:16.586+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:16.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:16.586+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:16.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:16.586+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:16.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:16.586+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:17.330 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:17.239+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:17.240485+0000) 2024-04-27T03:54:17.330 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:17.283+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:17.331 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:17.283+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:17.331 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:17.283+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:17.331 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:17.284+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:17.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:17.051+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:17.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:17.051+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:17.331 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:17.051+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:17.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:17.051+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:17.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:17.326+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:17.327741+0000) 2024-04-27T03:54:17.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:17.531+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:17.531957+0000) 2024-04-27T03:54:17.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:17.536+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:17.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:17.536+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:17.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:17.536+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:17.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:17.537+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:18.158 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:54:17.910+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75548 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:54:18.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:17.906+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:17.907540+0000) 2024-04-27T03:54:18.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:17.906+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:17.907566+0000) 2024-04-27T03:54:18.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:17.907+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:17.907802+0000) 2024-04-27T03:54:18.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:18.067+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:18.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:18.067+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:18.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:18.067+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:18.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:18.068+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:18.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:18.329+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:18.330742+0000) 2024-04-27T03:54:18.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:18.242+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:18.242824+0000) 2024-04-27T03:54:18.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:18.326+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:18.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:18.326+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:18.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:18.326+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:18.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:18.327+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:18.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:18.535+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:18.536248+0000) 2024-04-27T03:54:18.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:18.541+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:18.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:18.541+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:18.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:18.541+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:18.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:18.541+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:19.337 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:19.244+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:19.245065+0000) 2024-04-27T03:54:19.338 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:19.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:19.338 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:19.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:19.338 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:19.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:19.338 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:19.093+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:19.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:19.332+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:19.332826+0000) 2024-04-27T03:54:19.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:19.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:19.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:19.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:19.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:19.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:19.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:19.356+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:19.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:19.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:19.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:19.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:19.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:19.516+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:19.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:19.516+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:19.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:19.537+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:19.538556+0000) 2024-04-27T03:54:20.337 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:20.246+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:20.247159+0000) 2024-04-27T03:54:20.338 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:20.119+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:20.338 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:20.119+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:20.338 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:20.119+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:20.338 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:20.119+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:20.338 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:20.334+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:20.335172+0000) 2024-04-27T03:54:20.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:20.391+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:20.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:20.391+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:20.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:20.391+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:20.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:20.392+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:20.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:20.517+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:20.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:20.517+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:20.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:20.517+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:20.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:20.517+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:20.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:20.540+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:20.540816+0000) 2024-04-27T03:54:21.340 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:21.248+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:21.249295+0000) 2024-04-27T03:54:21.340 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:21.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:21.340 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:21.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:21.340 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:21.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:21.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:21.097+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:21.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:21.336+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:21.337595+0000) 2024-04-27T03:54:21.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:21.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:21.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:21.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:21.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:21.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:21.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:21.384+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:21.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:21.520+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:21.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:21.520+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:21.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:21.520+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:21.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:21.520+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:21.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:21.542+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:21.543060+0000) 2024-04-27T03:54:22.343 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:22.251+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:22.251958+0000) 2024-04-27T03:54:22.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:22.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:22.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:22.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:22.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:22.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:22.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:22.129+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:22.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:22.340+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:22.340923+0000) 2024-04-27T03:54:22.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:22.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:22.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:22.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:22.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:22.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:22.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:22.420+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:22.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:22.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:22.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:22.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:22.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:22.493+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:22.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:22.493+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:22.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:22.545+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:22.545782+0000) 2024-04-27T03:54:23.174 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:54:22.920+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75587 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:54:23.174 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:22.915+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:22.916580+0000) 2024-04-27T03:54:23.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:22.915+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:22.916470+0000) 2024-04-27T03:54:23.175 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:22.915+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:22.916510+0000) 2024-04-27T03:54:23.175 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:23.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:23.175 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:23.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:23.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:23.170+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:23.430 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:23.254+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:23.255160+0000) 2024-04-27T03:54:23.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:23.172+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:23.430 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:23.342+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:23.343275+0000) 2024-04-27T03:54:23.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:23.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:23.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:23.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:23.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:23.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:23.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:23.427+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:23.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:23.507+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:23.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:23.507+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:23.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:23.507+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:23.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:23.507+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:23.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:23.548+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:23.549067+0000) 2024-04-27T03:54:24.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:24.166+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:24.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:24.166+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:24.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:24.166+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:24.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:24.167+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:24.508 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:24.344+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:24.345406+0000) 2024-04-27T03:54:24.508 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:24.256+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:24.257346+0000) 2024-04-27T03:54:24.508 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:24.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:24.508 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:24.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:24.508 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:24.418+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:24.509 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:24.419+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:24.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:24.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:24.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:24.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:24.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:24.504+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:24.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:24.505+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:24.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:24.552+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:24.553301+0000) 2024-04-27T03:54:25.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:25.258+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:25.259333+0000) 2024-04-27T03:54:25.347 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:25.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:25.347 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:25.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:25.347 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:25.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:25.347 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:25.148+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:25.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:25.347+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:25.348287+0000) 2024-04-27T03:54:25.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:25.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:25.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:25.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:25.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:25.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:25.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:25.518+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:25.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:25.557+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:25.558057+0000) 2024-04-27T03:54:25.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:25.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:25.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:25.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:25.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:25.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:25.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:25.414+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:26.346 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:26.260+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:26.261593+0000) 2024-04-27T03:54:26.347 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:26.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:26.347 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:26.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:26.347 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:26.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:26.347 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:26.176+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:26.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:26.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:26.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:26.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:26.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:26.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:26.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:26.383+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:26.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:26.349+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:26.350611+0000) 2024-04-27T03:54:26.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:26.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:26.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:26.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:26.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:26.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:26.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:26.477+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:26.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:26.562+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:26.562903+0000) 2024-04-27T03:54:27.461 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:27.263+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:27.263918+0000) 2024-04-27T03:54:27.461 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:27.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:27.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:27.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:27.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:27.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:27.462 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:27.374+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:27.462 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:27.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:27.462 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:27.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:27.462 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:27.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:27.463 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:27.133+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:27.463 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:27.352+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:27.352847+0000) 2024-04-27T03:54:27.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:27.458+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:27.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:27.458+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:27.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:27.458+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:27.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:27.458+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:27.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:27.564+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:27.565258+0000) 2024-04-27T03:54:28.223 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:54:27.929+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75628 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:54:28.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:27.925+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:27.926006+0000) 2024-04-27T03:54:28.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:27.925+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:27.926791+0000) 2024-04-27T03:54:28.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:27.926+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:27.927262+0000) 2024-04-27T03:54:28.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:28.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:28.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:28.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:28.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:28.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:28.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:28.092+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:28.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:28.354+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:28.355151+0000) 2024-04-27T03:54:28.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:28.429+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:28.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:28.429+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:28.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:28.429+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:28.570 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:28.430+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:28.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:28.265+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:28.266085+0000) 2024-04-27T03:54:28.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:28.326+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:28.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:28.326+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:28.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:28.326+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:28.571 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:28.326+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:28.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:28.566+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:28.567404+0000) 2024-04-27T03:54:29.347 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:29.269+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:29.270283+0000) 2024-04-27T03:54:29.347 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:29.117+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:29.348 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:29.117+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:29.348 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:29.117+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:29.348 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:29.118+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:29.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:29.356+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:29.357327+0000) 2024-04-27T03:54:29.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:29.355+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:29.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:29.355+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:29.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:29.355+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:29.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:29.355+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:29.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:29.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:29.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:29.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:29.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:29.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:29.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:29.391+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:29.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:29.570+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:29.570969+0000) 2024-04-27T03:54:30.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:30.271+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:30.272419+0000) 2024-04-27T03:54:30.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:30.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:30.344 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:30.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:30.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:30.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:30.344 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:30.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:30.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:30.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:30.345 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:30.076+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:30.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:30.358+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:30.359434+0000) 2024-04-27T03:54:30.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:30.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:30.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:30.341+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:30.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:30.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:30.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:30.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:30.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:30.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:30.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:30.360+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:30.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:30.572+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:30.573341+0000) 2024-04-27T03:54:31.311 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:31.275+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:31.276374+0000) 2024-04-27T03:54:31.311 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:31.307+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:31.311 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:31.307+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:31.311 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:31.307+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:31.311 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:31.061+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:31.312 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:31.061+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:31.312 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:31.061+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:31.312 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:31.062+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:31.580 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:31.361+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:31.361892+0000) 2024-04-27T03:54:31.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:31.308+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:31.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:31.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:31.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:31.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:31.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:31.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:31.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:31.344+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:31.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:31.577+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:31.578144+0000) 2024-04-27T03:54:32.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:32.277+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:32.278596+0000) 2024-04-27T03:54:32.323 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:32.320+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:32.324 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:32.320+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:32.324 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:32.320+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:32.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:32.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:32.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:32.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:32.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:32.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:32.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:32.098+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:32.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:32.321+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:32.586 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:32.363+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:32.364207+0000) 2024-04-27T03:54:32.586 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:32.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:32.586 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:32.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:32.586 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:32.384+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:32.586 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:32.385+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:32.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:32.582+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:32.583345+0000) 2024-04-27T03:54:33.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:32.935+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:32.936624+0000) 2024-04-27T03:54:33.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:32.934+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:32.935587+0000) 2024-04-27T03:54:33.224 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:54:32.938+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75667 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:54:33.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:32.935+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:32.936103+0000) 2024-04-27T03:54:33.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:33.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:33.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:33.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:33.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:33.122+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:33.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:33.124+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:33.588 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:33.367+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:33.368299+0000) 2024-04-27T03:54:33.588 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:33.279+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:33.280776+0000) 2024-04-27T03:54:33.588 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:33.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:33.588 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:33.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:33.589 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:33.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:33.589 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:33.347+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:33.589 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:33.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:33.589 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:33.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:33.589 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:33.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:33.589 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:33.359+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:33.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:33.584+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:33.585671+0000) 2024-04-27T03:54:34.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:34.172+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:34.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:34.172+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:34.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:34.172+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:34.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:34.173+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:34.591 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:34.369+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:34.370236+0000) 2024-04-27T03:54:34.591 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:34.285+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:34.285845+0000) 2024-04-27T03:54:34.591 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:34.303+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:34.591 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:34.303+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:34.591 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:34.303+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:34.592 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:34.304+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:34.592 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:34.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:34.592 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:34.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:34.592 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:34.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:34.592 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:34.351+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:34.592 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:34.587+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:34.588590+0000) 2024-04-27T03:54:35.325 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:35.287+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:35.287940+0000) 2024-04-27T03:54:35.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:35.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:35.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:35.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:35.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:35.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:35.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:35.162+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:35.594 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:35.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:35.594 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:35.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:35.594 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:35.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:35.594 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:35.323+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:35.595 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:35.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:35.595 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:35.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:35.595 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:35.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:35.595 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:35.366+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:35.595 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:35.371+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:35.372456+0000) 2024-04-27T03:54:35.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:35.590+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:35.591540+0000) 2024-04-27T03:54:36.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:36.157+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:36.293 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:36.157+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:36.294 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:36.157+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:36.294 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:36.158+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:36.596 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:36.290+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:36.291092+0000) 2024-04-27T03:54:36.596 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:36.319+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:36.597 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:36.319+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:36.597 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:36.319+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:36.597 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:36.319+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:36.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:36.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:36.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:36.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:36.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:36.351+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:36.598 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:36.351+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:36.598 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:36.373+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:36.374754+0000) 2024-04-27T03:54:36.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:36.593+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:36.594058+0000) 2024-04-27T03:54:37.348 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:37.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:37.348 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:37.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:37.348 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:37.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:37.348 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:37.144+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:37.348 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:37.294+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:37.295402+0000) 2024-04-27T03:54:37.349 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:37.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:37.349 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:37.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:37.349 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:37.304+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:37.349 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:37.304+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:37.598 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:37.377+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:37.378040+0000) 2024-04-27T03:54:37.599 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:37.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:37.599 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:37.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:37.599 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:37.348+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:37.599 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:37.349+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:37.942 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:37.595+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:37.596505+0000) 2024-04-27T03:54:38.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:37.943+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:37.944491+0000) 2024-04-27T03:54:38.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:37.942+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:37.943327+0000) 2024-04-27T03:54:38.224 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:54:37.949+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75708 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:54:38.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:37.942+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:37.943752+0000) 2024-04-27T03:54:38.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:38.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:38.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:38.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:38.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:38.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:38.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:38.160+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:38.601 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:38.297+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:38.298470+0000) 2024-04-27T03:54:38.601 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:38.350+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:38.601 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:38.350+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:38.601 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:38.350+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:38.602 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:38.351+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:38.602 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:38.370+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:38.602 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:38.370+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:38.602 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:38.370+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:38.602 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:38.371+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:38.603 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:38.380+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:38.381211+0000) 2024-04-27T03:54:38.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:38.598+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:38.598825+0000) 2024-04-27T03:54:39.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:39.301+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:39.302309+0000) 2024-04-27T03:54:39.342 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:39.148+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:39.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:39.148+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:39.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:39.148+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:39.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:39.149+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:39.606 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:39.385+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:39.386229+0000) 2024-04-27T03:54:39.606 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:39.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:39.606 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:39.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:39.606 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:39.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:39.607 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:39.401+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:39.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:39.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:39.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:39.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:39.607 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:39.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:39.608 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:39.339+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:39.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:39.603+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:39.603798+0000) 2024-04-27T03:54:40.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:40.113+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:40.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:40.113+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:40.310 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:40.113+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:40.310 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:40.114+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:40.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:40.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:40.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:40.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:40.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:40.297+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:40.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:40.297+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:40.611 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:40.390+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:40.391213+0000) 2024-04-27T03:54:40.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:40.306+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:40.307031+0000) 2024-04-27T03:54:40.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:40.394+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:40.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:40.394+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:40.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:40.394+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:40.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:40.396+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:40.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:40.608+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:40.608840+0000) 2024-04-27T03:54:41.312 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:41.300+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:41.312 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:41.300+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:41.312 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:41.300+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:41.312 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:41.301+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:41.313 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:41.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:41.313 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:41.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:41.313 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:41.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:41.313 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:41.067+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:41.616 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:41.393+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:41.393883+0000) 2024-04-27T03:54:41.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:41.309+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:41.309824+0000) 2024-04-27T03:54:41.617 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:41.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:41.617 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:41.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:41.617 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:41.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:41.617 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:41.416+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:41.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:41.613+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:41.613966+0000) 2024-04-27T03:54:42.315 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:42.036+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:42.315 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:42.036+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:42.315 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:42.036+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:42.315 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:42.036+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:42.621 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:42.397+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:42.398747+0000) 2024-04-27T03:54:42.621 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:42.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:42.622 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:42.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:42.622 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:42.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:42.622 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:42.340+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:42.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:42.311+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:42.312463+0000) 2024-04-27T03:54:42.623 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:42.371+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:42.623 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:42.371+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:42.623 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:42.371+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:42.623 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:42.372+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:42.954 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:42.617+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:42.618777+0000) 2024-04-27T03:54:43.223 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:54:42.959+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75747 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:54:43.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:42.955+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:42.956438+0000) 2024-04-27T03:54:43.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:42.954+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:42.955493+0000) 2024-04-27T03:54:43.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:42.954+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:42.955435+0000) 2024-04-27T03:54:43.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:43.061+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:43.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:43.061+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:43.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:43.061+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:43.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:43.061+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:43.485 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:43.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:43.485 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:43.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:43.485 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:43.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:43.485 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:43.367+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:43.486 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:43.314+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:43.315141+0000) 2024-04-27T03:54:43.486 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:43.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:43.486 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:43.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:43.486 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:43.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:43.486 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:43.340+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:43.487 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:43.401+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:43.402001+0000) 2024-04-27T03:54:43.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:43.623+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:43.623861+0000) 2024-04-27T03:54:44.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:44.027+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:44.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:44.027+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:44.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:44.027+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:44.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:44.028+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:44.628 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:44.403+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:44.404263+0000) 2024-04-27T03:54:44.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:44.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:44.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:44.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:44.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:44.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:44.629 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:44.301+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:44.629 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:44.318+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:44.319400+0000) 2024-04-27T03:54:44.629 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:44.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:44.629 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:44.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:44.629 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:44.337+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:44.629 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:44.338+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:44.629 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:44.625+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:44.626023+0000) 2024-04-27T03:54:45.248 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:44.992+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:45.248 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:44.992+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:45.249 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:44.992+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:45.249 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:44.993+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:45.631 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:45.406+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:45.407232+0000) 2024-04-27T03:54:45.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:45.319+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:45.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:45.319+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:45.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:45.319+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:45.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:45.320+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:45.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:45.320+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:45.321560+0000) 2024-04-27T03:54:45.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:45.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:45.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:45.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:45.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:45.336+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:45.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:45.337+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:45.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:45.628+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:45.629390+0000) 2024-04-27T03:54:46.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:46.012+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:46.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:46.012+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:46.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:46.012+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:46.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:46.014+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:46.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:46.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:46.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:46.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:46.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:46.368+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:46.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:46.368+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:46.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:46.323+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:46.324797+0000) 2024-04-27T03:54:46.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:46.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:46.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:46.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:46.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:46.340+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:46.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:46.341+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:46.636 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:46.409+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:46.409897+0000) 2024-04-27T03:54:46.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:46.631+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:46.632251+0000) 2024-04-27T03:54:47.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:47.021+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:47.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:47.021+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:47.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:47.021+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:47.328 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:47.022+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:47.637 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:47.413+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:47.414222+0000) 2024-04-27T03:54:47.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:47.327+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:47.328344+0000) 2024-04-27T03:54:47.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:47.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:47.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:47.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:47.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:47.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:47.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:47.374+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:47.639 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:47.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:47.639 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:47.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:47.639 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:47.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:47.639 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:47.325+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:47.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:47.634+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:47.635468+0000) 2024-04-27T03:54:48.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:47.964+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:47.965272+0000) 2024-04-27T03:54:48.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:47.964+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:47.965677+0000) 2024-04-27T03:54:48.223 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:54:47.970+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75788 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:54:48.235 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:47.964+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:47.965186+0000) 2024-04-27T03:54:48.235 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:47.999+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:48.235 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:47.999+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:48.235 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:47.999+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:48.235 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:48.000+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:48.531 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:48.417+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:48.418289+0000) 2024-04-27T03:54:48.531 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:48.329+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:48.330716+0000) 2024-04-27T03:54:48.532 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:48.404+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:48.532 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:48.404+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:48.532 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:48.404+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:48.532 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:48.405+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:48.532 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:48.304+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:48.533 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:48.304+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:48.533 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:48.304+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:48.533 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:48.304+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:48.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:48.637+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:48.638444+0000) 2024-04-27T03:54:49.333 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:49.004+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:49.334 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:49.004+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:49.334 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:49.004+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:49.334 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:49.005+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:49.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:49.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:49.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:49.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:49.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:49.330+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:49.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:49.331+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:49.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:49.332+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:49.333013+0000) 2024-04-27T03:54:49.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:49.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:49.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:49.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:49.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:49.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:49.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:49.444+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:49.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:49.422+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:49.422931+0000) 2024-04-27T03:54:49.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:49.639+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:49.640650+0000) 2024-04-27T03:54:50.337 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:50.047+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:50.337 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:50.047+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:50.338 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:50.047+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:50.338 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:50.048+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:50.645 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:50.425+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:50.426335+0000) 2024-04-27T03:54:50.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:50.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:50.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:50.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:50.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:50.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:50.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:50.365+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:50.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:50.334+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:50.335286+0000) 2024-04-27T03:54:50.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:50.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:50.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:50.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:50.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:50.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:50.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:50.458+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:50.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:50.642+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:50.643274+0000) 2024-04-27T03:54:51.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:51.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:51.326 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:51.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:51.326 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:51.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:51.326 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:51.067+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:51.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:51.336+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:51.337573+0000) 2024-04-27T03:54:51.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:51.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:51.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:51.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:51.649 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:51.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:51.649 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:51.442+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:51.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:51.322+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:51.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:51.322+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:51.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:51.322+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:51.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:51.323+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:51.650 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:51.645+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:51.646062+0000) 2024-04-27T03:54:51.650 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:51.427+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:51.428503+0000) 2024-04-27T03:54:52.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:52.102+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:52.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:52.102+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:52.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:52.102+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:52.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:52.104+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:52.650 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:52.428+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:52.430668+0000) 2024-04-27T03:54:52.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:52.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:52.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:52.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:52.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:52.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:52.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:52.321+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:52.652 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:52.340+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:52.342333+0000) 2024-04-27T03:54:52.652 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:52.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:52.652 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:52.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:52.652 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:52.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:52.652 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:52.408+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:52.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:52.646+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:52.648570+0000) 2024-04-27T03:54:53.234 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:54:52.978+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75827 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:54:53.234 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:52.975+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:52.976834+0000) 2024-04-27T03:54:53.235 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:52.975+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:52.977343+0000) 2024-04-27T03:54:53.235 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:52.974+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:52.976566+0000) 2024-04-27T03:54:53.235 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:53.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:53.235 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:53.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:53.235 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:53.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:53.236 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:53.093+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:53.580 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:53.432+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:53.434001+0000) 2024-04-27T03:54:53.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:53.280+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:53.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:53.280+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:53.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:53.280+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:53.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:53.280+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:53.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:53.342+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:53.344468+0000) 2024-04-27T03:54:53.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:53.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:53.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:53.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:53.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:53.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:53.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:53.374+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:53.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:53.649+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:53.651649+0000) 2024-04-27T03:54:54.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:54.080+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:54.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:54.080+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:54.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:54.080+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:54.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:54.080+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:54.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:54.344+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:54.346562+0000) 2024-04-27T03:54:54.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:54.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:54.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:54.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:54.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:54.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:54.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:54.347+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:54.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:54.273+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:54.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:54.273+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:54.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:54.273+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:54.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:54.273+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:54.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:54.652+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:54.653924+0000) 2024-04-27T03:54:54.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:54.436+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:54.438315+0000) 2024-04-27T03:54:55.349 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:55.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:55.350 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:55.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:55.350 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:55.300+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:55.350 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:55.301+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:55.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:55.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:55.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:55.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:55.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:55.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:55.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:55.253+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:55.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:55.087+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:55.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:55.087+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:55.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:55.087+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:55.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:55.088+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:55.660 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:55.348+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:55.350178+0000) 2024-04-27T03:54:55.660 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:55.438+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:55.440430+0000) 2024-04-27T03:54:55.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:55.656+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:55.658452+0000) 2024-04-27T03:54:56.349 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:56.275+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:56.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:56.275+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:56.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:56.275+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:56.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:56.275+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:56.350 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:56.254+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:56.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:56.254+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:56.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:56.254+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:56.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:56.255+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:56.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:56.132+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:56.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:56.132+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:56.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:56.132+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:56.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:56.133+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:56.662 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:56.351+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:56.352811+0000) 2024-04-27T03:54:56.663 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:56.440+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:56.442764+0000) 2024-04-27T03:54:56.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:56.658+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:56.660783+0000) 2024-04-27T03:54:57.350 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:57.138+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:57.350 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:57.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:57.350 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:57.139+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:57.350 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:57.139+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:57.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:57.227+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:57.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:57.227+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:57.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:57.227+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:57.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:57.228+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:57.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:57.275+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:57.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:57.275+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:57.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:57.275+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:57.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:57.276+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:57.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:57.353+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:57.355062+0000) 2024-04-27T03:54:57.665 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:57.445+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:57.447113+0000) 2024-04-27T03:54:57.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:57.661+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:57.663399+0000) 2024-04-27T03:54:58.234 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:54:57.987+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75868 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:54:58.235 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:57.982+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:57.984679+0000) 2024-04-27T03:54:58.235 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:57.983+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:57.985606+0000) 2024-04-27T03:54:58.235 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:58.207+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:58.235 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:58.207+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:58.235 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:58.207+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:58.235 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:58.207+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:58.236 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:57.983+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:57.985040+0000) 2024-04-27T03:54:58.236 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:58.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:58.236 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:58.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:58.236 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:58.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:58.236 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:58.093+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:58.636 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:58.450+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:58.452164+0000) 2024-04-27T03:54:58.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:58.272+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:58.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:58.272+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:58.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:58.272+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:58.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:58.273+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:58.637 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:58.355+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:58.357736+0000) 2024-04-27T03:54:58.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:58.663+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:58.665421+0000) 2024-04-27T03:54:59.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:59.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:59.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:59.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:59.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:59.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:54:59.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:59.181+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:59.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:59.270+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:59.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:59.270+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:59.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:59.270+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:54:59.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:59.270+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:59.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:59.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:59.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:59.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:59.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:59.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:54:59.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:59.066+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:54:59.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:54:59.358+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:59.359904+0000) 2024-04-27T03:54:59.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:54:59.452+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:59.454445+0000) 2024-04-27T03:54:59.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:54:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:54:59.666+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:54:59.667795+0000) 2024-04-27T03:55:00.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:00.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:00.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:00.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:00.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:00.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:00.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:00.179+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:00.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:00.248+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:00.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:00.248+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:00.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:00.248+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:00.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:00.249+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:00.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:00.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:00.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:00.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:00.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:00.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:00.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:00.065+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:00.608 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:00.362+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:00.364088+0000) 2024-04-27T03:55:00.608 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:00.454+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:00.456709+0000) 2024-04-27T03:55:00.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:00.668+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:00.670302+0000) 2024-04-27T03:55:01.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:01.177+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:01.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:01.177+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:01.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:01.177+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:01.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:01.177+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:01.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:01.227+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:01.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:01.227+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:01.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:01.227+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:01.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:01.229+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:01.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:01.071+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:01.354 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:01.071+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:01.354 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:01.071+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:01.354 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:01.072+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:01.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:01.364+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:01.366257+0000) 2024-04-27T03:55:01.676 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:01.459+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:01.461478+0000) 2024-04-27T03:55:01.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:01.671+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:01.673599+0000) 2024-04-27T03:55:02.350 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:02.049+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:02.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:02.049+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:02.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:02.049+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:02.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:02.050+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:02.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:02.279+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:02.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:02.279+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:02.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:02.279+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:02.352 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:02.279+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:02.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:02.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:02.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:02.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:02.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:02.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:02.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:02.162+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:02.678 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:02.461+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:02.463650+0000) 2024-04-27T03:55:02.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:02.369+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:02.370950+0000) 2024-04-27T03:55:02.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:02.674+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:02.676176+0000) 2024-04-27T03:55:03.318 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:55:02.996+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75907 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:55:03.318 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:02.991+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:02.993645+0000) 2024-04-27T03:55:03.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:02.991+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:02.993480+0000) 2024-04-27T03:55:03.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:03.019+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:03.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:03.019+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:03.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:03.019+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:03.319 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:03.019+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:02.992+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:02.994353+0000) 2024-04-27T03:55:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:03.186+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:03.186+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:03.186+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:03.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:03.186+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:03.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:03.314+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:03.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:03.314+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:03.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:03.314+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:03.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:03.315+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:03.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:03.375+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:03.375866+0000) 2024-04-27T03:55:03.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:03.466+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:03.466926+0000) 2024-04-27T03:55:03.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:03.677+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:03.678567+0000) 2024-04-27T03:55:04.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:04.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:04.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:04.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:04.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:04.010+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:04.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:04.011+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:04.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:04.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:04.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:04.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:04.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:04.198+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:04.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:04.199+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:04.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:04.309+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:04.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:04.309+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:04.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:04.309+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:04.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:04.310+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:04.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:04.378+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:04.379219+0000) 2024-04-27T03:55:04.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:04.468+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:04.469076+0000) 2024-04-27T03:55:04.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:04.680+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:04.681347+0000) 2024-04-27T03:55:05.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:05.196+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:05.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:05.196+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:05.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:05.196+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:05.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:05.196+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:05.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:05.029+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:05.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:05.029+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:05.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:05.029+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:05.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:05.030+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:05.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:05.314+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:05.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:05.314+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:05.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:05.314+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:05.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:05.315+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:05.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:05.381+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:05.381931+0000) 2024-04-27T03:55:05.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:05.471+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:05.471980+0000) 2024-04-27T03:55:05.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:05.683+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:05.684224+0000) 2024-04-27T03:55:06.313 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:06.309+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:06.313 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:06.309+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:06.313 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:06.309+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:06.313 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:06.310+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:06.314 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:06.177+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:06.314 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:06.177+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:06.314 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:06.177+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:06.314 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:06.178+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:06.314 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:06.074+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:06.314 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:06.074+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:06.315 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:06.074+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:06.315 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:06.075+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:06.689 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:06.384+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:06.385113+0000) 2024-04-27T03:55:06.689 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:06.474+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:06.475240+0000) 2024-04-27T03:55:06.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:06.686+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:06.686857+0000) 2024-04-27T03:55:07.348 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:07.105+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:07.348 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:07.105+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:07.348 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:07.105+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:07.349 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:07.106+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:07.349 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:07.154+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:07.349 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:07.154+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:07.349 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:07.154+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:07.349 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:07.155+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:07.691 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:07.479+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:07.479823+0000) 2024-04-27T03:55:07.692 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:07.344+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:07.692 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:07.344+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:07.692 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:07.344+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:07.692 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:07.346+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:07.692 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:07.387+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:07.388060+0000) 2024-04-27T03:55:07.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:07.688+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:07.689646+0000) 2024-04-27T03:55:08.351 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:55:08.006+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75948 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:55:08.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:08.002+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:08.003464+0000) 2024-04-27T03:55:08.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:08.002+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:08.003360+0000) 2024-04-27T03:55:08.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:08.147+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:08.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:08.147+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:08.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:08.147+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:08.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:08.148+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:08.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:08.002+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:08.002959+0000) 2024-04-27T03:55:08.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:08.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:08.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:08.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:08.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:08.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:08.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:08.162+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:08.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:08.482+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:08.483015+0000) 2024-04-27T03:55:08.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:08.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:08.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:08.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:08.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:08.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:08.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:08.390+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:08.391154+0000) 2024-04-27T03:55:08.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:08.390+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:08.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:08.692+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:08.692842+0000) 2024-04-27T03:55:09.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:09.123+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:09.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:09.123+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:09.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:09.123+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:09.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:09.124+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:09.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:09.186+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:09.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:09.186+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:09.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:09.186+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:09.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:09.186+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:09.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:09.485+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:09.486127+0000) 2024-04-27T03:55:09.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:09.392+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:09.393327+0000) 2024-04-27T03:55:09.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:09.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:09.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:09.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:09.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:09.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:09.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:09.426+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:09.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:09.694+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:09.695186+0000) 2024-04-27T03:55:10.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:10.158+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:10.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:10.158+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:10.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:10.158+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:10.351 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:10.158+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:10.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:10.157+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:10.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:10.157+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:10.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:10.157+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:10.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:10.157+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:10.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:10.489+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:10.490408+0000) 2024-04-27T03:55:10.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:10.394+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:10.395560+0000) 2024-04-27T03:55:10.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:10.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:10.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:10.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:10.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:10.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:10.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:10.425+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:10.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:10.698+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:10.698840+0000) 2024-04-27T03:55:11.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:11.120+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:11.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:11.120+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:11.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:11.120+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:11.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:11.120+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:11.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:11.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:11.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:11.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:11.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:11.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:11.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:11.176+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:11.705 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:11.491+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:11.492602+0000) 2024-04-27T03:55:11.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:11.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:11.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:11.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:11.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:11.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:11.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:11.386+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:11.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:11.398+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:11.398889+0000) 2024-04-27T03:55:11.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:11.702+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:11.703569+0000) 2024-04-27T03:55:12.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:12.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:12.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:12.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:12.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:12.121+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:12.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:12.121+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:12.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:12.187+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:12.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:12.187+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:12.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:12.187+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:12.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:12.187+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:12.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:12.494+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:12.494812+0000) 2024-04-27T03:55:12.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:12.400+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:12.401581+0000) 2024-04-27T03:55:12.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:12.407+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:12.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:12.407+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:12.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:12.407+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:12.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:12.408+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:12.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:12.707+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:12.708277+0000) 2024-04-27T03:55:13.351 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:13.011+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:13.012557+0000) 2024-04-27T03:55:13.352 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:55:13.015+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 75987 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:55:13.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:13.010+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:13.011518+0000) 2024-04-27T03:55:13.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:13.169+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:13.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:13.169+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:13.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:13.169+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:13.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:13.169+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:13.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:13.010+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:13.011133+0000) 2024-04-27T03:55:13.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:13.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:13.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:13.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:13.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:13.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:13.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:13.168+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:13.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:13.496+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:13.496982+0000) 2024-04-27T03:55:13.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:13.403+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:13.403844+0000) 2024-04-27T03:55:13.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:13.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:13.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:13.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:13.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:13.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:13.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:13.416+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:13.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:13.711+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:13.712472+0000) 2024-04-27T03:55:14.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:14.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:14.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:14.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:14.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:14.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:14.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:14.145+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:14.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:14.204+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:14.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:14.204+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:14.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:14.204+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:14.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:14.205+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:14.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:14.498+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:14.499039+0000) 2024-04-27T03:55:14.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:14.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:14.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:14.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:14.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:14.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:14.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:14.395+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:14.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:14.407+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:14.408065+0000) 2024-04-27T03:55:14.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:14.714+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:14.715371+0000) 2024-04-27T03:55:15.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:15.243+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:15.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:15.243+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:15.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:15.243+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:15.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:15.244+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:15.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:15.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:15.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:15.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:15.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:15.174+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:15.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:15.175+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:15.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:15.501+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:15.502127+0000) 2024-04-27T03:55:15.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:15.411+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:15.412254+0000) 2024-04-27T03:55:15.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:15.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:15.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:15.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:15.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:15.433+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:15.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:15.434+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:15.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:15.717+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:15.718016+0000) 2024-04-27T03:55:16.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:16.234+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:16.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:16.234+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:16.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:16.234+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:16.352 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:16.235+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:16.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:16.183+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:16.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:16.183+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:16.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:16.183+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:16.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:16.183+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:16.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:16.505+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:16.506445+0000) 2024-04-27T03:55:16.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:16.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:16.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:16.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:16.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:16.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:16.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:16.386+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:16.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:16.416+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:16.416912+0000) 2024-04-27T03:55:17.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:16.722+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:16.723011+0000) 2024-04-27T03:55:17.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:17.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:17.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:17.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:17.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:17.202+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:17.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:17.203+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:17.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:17.284+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:17.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:17.284+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:17.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:17.284+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:17.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:17.285+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:17.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:17.508+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:17.509468+0000) 2024-04-27T03:55:17.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:17.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:17.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:17.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:17.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:17.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:17.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:17.414+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:17.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:17.418+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:17.419608+0000) 2024-04-27T03:55:18.019 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:17.724+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:17.725678+0000) 2024-04-27T03:55:18.296 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:18.020+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:18.021047+0000) 2024-04-27T03:55:18.297 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:55:18.024+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76028 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:55:18.297 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:18.021+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:18.022029+0000) 2024-04-27T03:55:18.297 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:18.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:18.297 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:18.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:18.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:18.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:18.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:18.020+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:18.021193+0000) 2024-04-27T03:55:18.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:18.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:18.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:18.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:18.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:18.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:18.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:18.160+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:18.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:18.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:18.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:18.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:18.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:18.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:18.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:18.403+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:18.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:18.422+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:18.422930+0000) 2024-04-27T03:55:18.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:18.295+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:18.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:18.513+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:18.513827+0000) 2024-04-27T03:55:19.063 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:18.728+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:18.729068+0000) 2024-04-27T03:55:19.337 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:19.152+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:19.338 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:19.152+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:19.338 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:19.152+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:19.338 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:19.152+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:19.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:19.424+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:19.425144+0000) 2024-04-27T03:55:19.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:19.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:19.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:19.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:19.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:19.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:19.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:19.443+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:19.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:19.334+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:19.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:19.334+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:19.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:19.334+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:19.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:19.335+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:19.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:19.515+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:19.516025+0000) 2024-04-27T03:55:20.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:19.730+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:19.731300+0000) 2024-04-27T03:55:20.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:20.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:20.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:20.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:20.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:20.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:20.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:20.179+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:20.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:20.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:20.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:20.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:20.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:20.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:20.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:20.311+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:20.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:20.517+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:20.518102+0000) 2024-04-27T03:55:20.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:20.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:20.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:20.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:20.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:20.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:20.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:20.416+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:20.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:20.428+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:20.429359+0000) 2024-04-27T03:55:21.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:20.732+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:20.733536+0000) 2024-04-27T03:55:21.351 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:21.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:21.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:21.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:21.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:21.188+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:21.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:21.188+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:21.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:21.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:21.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:21.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:21.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:21.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:21.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:21.416+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:21.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:21.430+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:21.431594+0000) 2024-04-27T03:55:21.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:21.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:21.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:21.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:21.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:21.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:21.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:21.349+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:21.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:21.520+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:21.521168+0000) 2024-04-27T03:55:22.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:21.735+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:21.736641+0000) 2024-04-27T03:55:22.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:22.227+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:22.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:22.227+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:22.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:22.227+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:22.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:22.227+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:22.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:22.391+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:22.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:22.391+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:22.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:22.391+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:22.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:22.392+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:22.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:22.522+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:22.523289+0000) 2024-04-27T03:55:22.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:22.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:22.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:22.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:22.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:22.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:22.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:22.390+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:22.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:22.433+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:22.434733+0000) 2024-04-27T03:55:23.028 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:22.738+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:22.738795+0000) 2024-04-27T03:55:23.352 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:55:23.034+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76067 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:55:23.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:23.030+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:23.030902+0000) 2024-04-27T03:55:23.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:23.029+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:23.030481+0000) 2024-04-27T03:55:23.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:23.030+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:23.031712+0000) 2024-04-27T03:55:23.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:23.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:23.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:23.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:23.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:23.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:23.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:23.255+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:23.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:23.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:23.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:23.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:23.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:23.435+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:23.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:23.436+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:23.437062+0000) 2024-04-27T03:55:23.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:23.436+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:23.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:23.376+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:23.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:23.376+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:23.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:23.376+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:23.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:23.377+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:23.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:23.524+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:23.525434+0000) 2024-04-27T03:55:24.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:23.742+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:23.743044+0000) 2024-04-27T03:55:24.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:24.252+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:24.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:24.252+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:24.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:24.252+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:24.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:24.252+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:24.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:24.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:24.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:24.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:24.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:24.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:24.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:24.407+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:24.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:24.439+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:24.440191+0000) 2024-04-27T03:55:24.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:24.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:24.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:24.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:24.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:24.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:24.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:24.381+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:24.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:24.527+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:24.528284+0000) 2024-04-27T03:55:25.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:24.744+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:24.745172+0000) 2024-04-27T03:55:25.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:25.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:25.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:25.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:25.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:25.216+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:25.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:25.217+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:25.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:25.416+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:25.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:25.416+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:25.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:25.416+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:25.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:25.417+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:25.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:25.442+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:25.442883+0000) 2024-04-27T03:55:25.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:25.370+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:25.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:25.370+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:25.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:25.370+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:25.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:25.372+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:25.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:25.530+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:25.531229+0000) 2024-04-27T03:55:26.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:25.746+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:25.747411+0000) 2024-04-27T03:55:26.352 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:26.181+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:26.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:26.181+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:26.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:26.181+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:26.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:26.181+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:26.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:26.415+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:26.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:26.415+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:26.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:26.415+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:26.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:26.416+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:26.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:26.534+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:26.535574+0000) 2024-04-27T03:55:26.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:26.446+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:26.447602+0000) 2024-04-27T03:55:26.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:26.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:26.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:26.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:26.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:26.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:26.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:26.465+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:27.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:26.751+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:26.752144+0000) 2024-04-27T03:55:27.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:27.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:27.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:27.144+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:27.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:27.144+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:27.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:27.144+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:27.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:27.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:27.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:27.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:27.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:27.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:27.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:27.432+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:27.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:27.451+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:27.452330+0000) 2024-04-27T03:55:27.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:27.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:27.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:27.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:27.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:27.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:27.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:27.464+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:27.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:27.537+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:27.538366+0000) 2024-04-27T03:55:28.038 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:27.756+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:27.756907+0000) 2024-04-27T03:55:28.353 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:55:28.044+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76108 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:55:28.353 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:28.039+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:28.039964+0000) 2024-04-27T03:55:28.354 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:28.038+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:28.039749+0000) 2024-04-27T03:55:28.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:28.039+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:28.040522+0000) 2024-04-27T03:55:28.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:28.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:28.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:28.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:28.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:28.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:28.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:28.174+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:28.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:28.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:28.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:28.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:28.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:28.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:28.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:28.440+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:28.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:28.456+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:28.457058+0000) 2024-04-27T03:55:28.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:28.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:28.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:28.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:28.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:28.482+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:28.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:28.483+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:28.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:28.540+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:28.541208+0000) 2024-04-27T03:55:29.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:28.760+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:28.761786+0000) 2024-04-27T03:55:29.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:29.189+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:29.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:29.189+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:29.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:29.189+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:29.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:29.190+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:29.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:29.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:29.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:29.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:29.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:29.463+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:29.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:29.465+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:29.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:29.542+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:29.543333+0000) 2024-04-27T03:55:29.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:29.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:29.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:29.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:29.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:29.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:29.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:29.426+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:29.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:29.458+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:29.459156+0000) 2024-04-27T03:55:30.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:29.763+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:29.763996+0000) 2024-04-27T03:55:30.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:30.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:30.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:30.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:30.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:30.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:30.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:30.220+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:30.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:30.429+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:30.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:30.429+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:30.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:30.429+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:30.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:30.430+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:30.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:30.544+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:30.545417+0000) 2024-04-27T03:55:30.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:30.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:30.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:30.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:30.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:30.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:30.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:30.457+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:30.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:30.463+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:30.464218+0000) 2024-04-27T03:55:31.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:30.765+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:30.766298+0000) 2024-04-27T03:55:31.327 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:31.227+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:31.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:31.227+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:31.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:31.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:31.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:31.228+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:31.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:31.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:31.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:31.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:31.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:31.466+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:31.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:31.468+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:31.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:31.546+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:31.547531+0000) 2024-04-27T03:55:31.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:31.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:31.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:31.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:31.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:31.413+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:31.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:31.414+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:31.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:31.465+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:31.466324+0000) 2024-04-27T03:55:32.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:31.767+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:31.768706+0000) 2024-04-27T03:55:32.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:32.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:32.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:32.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:32.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:32.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:32.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:32.180+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:32.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:32.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:32.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:32.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:32.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:32.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:32.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:32.374+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:32.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:32.468+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:32.469241+0000) 2024-04-27T03:55:32.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:32.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:32.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:32.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:32.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:32.492+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:32.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:32.492+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:32.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:32.551+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:32.552623+0000) 2024-04-27T03:55:33.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:32.770+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:32.770842+0000) 2024-04-27T03:55:33.354 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:55:33.053+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76147 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:55:33.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:33.049+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:33.050007+0000) 2024-04-27T03:55:33.355 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:33.049+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:33.050174+0000) 2024-04-27T03:55:33.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:33.049+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:33.049834+0000) 2024-04-27T03:55:33.356 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:33.176+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:33.356 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:33.176+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:33.356 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:33.176+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:33.356 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:33.177+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:33.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:33.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:33.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:33.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:33.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:33.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:33.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:33.359+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:33.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:33.471+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:33.471914+0000) 2024-04-27T03:55:33.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:33.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:33.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:33.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:33.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:33.486+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:33.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:33.487+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:33.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:33.554+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:33.555415+0000) 2024-04-27T03:55:34.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:33.772+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:33.773654+0000) 2024-04-27T03:55:34.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:34.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:34.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:34.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:34.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:34.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:34.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:34.220+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:34.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:34.384+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:34.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:34.384+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:34.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:34.384+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:34.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:34.385+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:34.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:34.475+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:34.476105+0000) 2024-04-27T03:55:34.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:34.489+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:34.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:34.489+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:34.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:34.489+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:34.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:34.490+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:34.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:34.558+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:34.559716+0000) 2024-04-27T03:55:35.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:34.775+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:34.775831+0000) 2024-04-27T03:55:35.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:35.172+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:35.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:35.172+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:35.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:35.172+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:35.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:35.173+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:35.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:35.372+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:35.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:35.372+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:35.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:35.372+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:35.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:35.373+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:35.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:35.477+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:35.478329+0000) 2024-04-27T03:55:35.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:35.510+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:35.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:35.510+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:35.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:35.510+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:35.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:35.512+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:35.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:35.561+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:35.562459+0000) 2024-04-27T03:55:36.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:35.779+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:35.780052+0000) 2024-04-27T03:55:36.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:36.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:36.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:36.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:36.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:36.171+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:36.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:36.171+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:36.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:36.407+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:36.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:36.407+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:36.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:36.407+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:36.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:36.408+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:36.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:36.480+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:36.481236+0000) 2024-04-27T03:55:36.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:36.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:36.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:36.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:36.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:36.554+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:36.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:36.555+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:36.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:36.565+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:36.566796+0000) 2024-04-27T03:55:37.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:36.782+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:36.783167+0000) 2024-04-27T03:55:37.353 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:37.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:37.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:37.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:37.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:37.158+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:37.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:37.159+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:37.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:37.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:37.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:37.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:37.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:37.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:37.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:37.451+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:37.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:37.483+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:37.484683+0000) 2024-04-27T03:55:37.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:37.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:37.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:37.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:37.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:37.534+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:37.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:37.535+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:37.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:37.568+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:37.568884+0000) 2024-04-27T03:55:38.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:37.784+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:37.785311+0000) 2024-04-27T03:55:38.354 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:55:38.065+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76188 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:55:38.354 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:38.059+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:38.060543+0000) 2024-04-27T03:55:38.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:38.059+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:38.060196+0000) 2024-04-27T03:55:38.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:38.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:38.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:38.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:38.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:38.193+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:38.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:38.193+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:38.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:38.058+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:38.059349+0000) 2024-04-27T03:55:38.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:38.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:38.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:38.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:38.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:38.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:38.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:38.464+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:38.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:38.488+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:38.489394+0000) 2024-04-27T03:55:38.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:38.570+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:38.571723+0000) 2024-04-27T03:55:38.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:38.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:38.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:38.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:38.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:38.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:38.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:38.573+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:39.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:38.787+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:38.788255+0000) 2024-04-27T03:55:39.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:39.165+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:39.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:39.165+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:39.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:39.165+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:39.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:39.165+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:39.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:39.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:39.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:39.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:39.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:39.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:39.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:39.551+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:39.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:39.573+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:39.574138+0000) 2024-04-27T03:55:39.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:39.491+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:39.492176+0000) 2024-04-27T03:55:39.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:39.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:39.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:39.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:39.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:39.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:39.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:39.492+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:40.066 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:39.789+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:39.790765+0000) 2024-04-27T03:55:40.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:40.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:40.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:40.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:40.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:40.159+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:40.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:40.160+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:40.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:40.469+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:40.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:40.469+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:40.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:40.469+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:40.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:40.470+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:40.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:40.495+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:40.496349+0000) 2024-04-27T03:55:40.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:40.575+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:40.576389+0000) 2024-04-27T03:55:40.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:40.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:40.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:40.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:40.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:40.582+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:40.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:40.583+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:41.066 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:40.792+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:40.793062+0000) 2024-04-27T03:55:41.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:41.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:41.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:41.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:41.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:41.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:41.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:41.174+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:41.625 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:41.438+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:41.626 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:41.438+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:41.626 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:41.438+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:41.626 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:41.439+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:41.626 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:41.500+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:41.501268+0000) 2024-04-27T03:55:41.626 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:41.577+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:41.578604+0000) 2024-04-27T03:55:41.627 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:41.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:41.627 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:41.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:41.627 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:41.623+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:41.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:41.624+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:41.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:41.797+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:41.798217+0000) 2024-04-27T03:55:42.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:42.210+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:42.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:42.210+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:42.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:42.210+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:42.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:42.211+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:42.647 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:42.580+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:42.580920+0000) 2024-04-27T03:55:42.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:42.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:42.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:42.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:42.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:42.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:42.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:42.487+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:42.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:42.505+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:42.506036+0000) 2024-04-27T03:55:42.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:42.800+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:42.800903+0000) 2024-04-27T03:55:42.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:42.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:42.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:42.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:42.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:42.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:42.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:42.645+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:43.354 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:55:43.074+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76227 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:55:43.355 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:43.070+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:43.071030+0000) 2024-04-27T03:55:43.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:43.069+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:43.069901+0000) 2024-04-27T03:55:43.356 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:43.069+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:43.070346+0000) 2024-04-27T03:55:43.356 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:43.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:43.356 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:43.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:43.356 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:43.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:43.356 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:43.258+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:43.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:43.510+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:43.511048+0000) 2024-04-27T03:55:43.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:43.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:43.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:43.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:43.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:43.536+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:43.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:43.537+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:43.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:43.582+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:43.583254+0000) 2024-04-27T03:55:43.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:43.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:43.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:43.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:43.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:43.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:43.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:43.596+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:44.066 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:43.803+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:43.804229+0000) 2024-04-27T03:55:44.354 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:44.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:44.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:44.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:44.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:44.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:44.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:44.282+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:44.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:44.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:44.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:44.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:44.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:44.571+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:44.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:44.572+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:44.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:44.586+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:44.587606+0000) 2024-04-27T03:55:44.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:44.514+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:44.515269+0000) 2024-04-27T03:55:44.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:44.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:44.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:44.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:44.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:44.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:44.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:44.559+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:45.066 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:44.806+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:44.807551+0000) 2024-04-27T03:55:45.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:45.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:45.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:45.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:45.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:45.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:45.355 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:45.312+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:45.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:45.516+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:45.517428+0000) 2024-04-27T03:55:45.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:45.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:45.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:45.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:45.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:45.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:45.610 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:45.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:45.610 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:45.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:45.611 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:45.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:45.611 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:45.560+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:45.611 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:45.589+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:45.589828+0000) 2024-04-27T03:55:45.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:45.607+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:45.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:45.809+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:45.810359+0000) 2024-04-27T03:55:46.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:46.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:46.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:46.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:46.572 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:46.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:46.572 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:46.568+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:46.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:46.318+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:46.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:46.318+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:46.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:46.318+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:46.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:46.319+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:46.573 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:46.519+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:46.520335+0000) 2024-04-27T03:55:46.573 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:46.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:46.573 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:46.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:46.573 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:46.558+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:46.573 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:46.559+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:46.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:46.812+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:46.813584+0000) 2024-04-27T03:55:46.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:46.591+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:46.592525+0000) 2024-04-27T03:55:47.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:47.521+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:47.522567+0000) 2024-04-27T03:55:47.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:47.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:47.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:47.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:47.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:47.578+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:47.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:47.579+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:47.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:47.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:47.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:47.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:47.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:47.568+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:47.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:47.569+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:47.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:47.594+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:47.594895+0000) 2024-04-27T03:55:47.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:47.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:47.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:47.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:47.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:47.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:47.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:47.365+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:48.078 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:47.816+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:47.816910+0000) 2024-04-27T03:55:48.355 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:55:48.082+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76268 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:55:48.355 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:48.080+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:48.080841+0000) 2024-04-27T03:55:48.356 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:48.080+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:48.080917+0000) 2024-04-27T03:55:48.356 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:48.078+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:48.079662+0000) 2024-04-27T03:55:48.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:48.524+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:48.524888+0000) 2024-04-27T03:55:48.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:48.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:48.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:48.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:48.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:48.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:48.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:48.543+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:48.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:48.596+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:48.597258+0000) 2024-04-27T03:55:48.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:48.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:48.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:48.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:48.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:48.596+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:48.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:48.597+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:48.724 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:48.411+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:48.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:48.411+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:48.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:48.411+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:48.725 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:48.411+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:49.153 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:48.819+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:48.820132+0000) 2024-04-27T03:55:49.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:49.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:49.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:49.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:49.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:49.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:49.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:49.367+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:49.824 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:49.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:49.825 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:49.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:49.825 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:49.583+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:49.825 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:49.584+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:49.825 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:49.601+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:49.602073+0000) 2024-04-27T03:55:49.825 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:49.498+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:49.826 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:49.498+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:49.826 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:49.498+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:49.826 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:49.499+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:49.826 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:49.526+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:49.526935+0000) 2024-04-27T03:55:50.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:49.822+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:49.822809+0000) 2024-04-27T03:55:50.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:50.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:50.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:50.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:50.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:50.367+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:50.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:50.367+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:50.828 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:50.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:50.828 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:50.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:50.828 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:50.572+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:50.828 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:50.573+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:50.828 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:50.603+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:50.604171+0000) 2024-04-27T03:55:50.829 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:50.530+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:50.531142+0000) 2024-04-27T03:55:50.829 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:50.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:50.829 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:50.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:50.829 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:50.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:50.829 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:50.541+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:51.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:50.825+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:50.826036+0000) 2024-04-27T03:55:51.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:51.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:51.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:51.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:51.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:51.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:51.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:51.365+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:51.831 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:51.605+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:51.606479+0000) 2024-04-27T03:55:51.831 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:51.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:51.831 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:51.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:51.832 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:51.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:51.832 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:51.618+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:51.832 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:51.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:51.832 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:51.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:51.832 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:51.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:51.832 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:51.516+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:51.833 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:51.532+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:51.533256+0000) 2024-04-27T03:55:52.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:51.828+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:51.829356+0000) 2024-04-27T03:55:52.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:52.362+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:52.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:52.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:52.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:52.363+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:52.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:52.363+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:52.832 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:52.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:52.832 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:52.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:52.832 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:52.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:52.832 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:52.508+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:52.832 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:52.537+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:52.538375+0000) 2024-04-27T03:55:52.833 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:52.610+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:52.611503+0000) 2024-04-27T03:55:52.833 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:52.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:52.833 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:52.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:52.833 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:52.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:52.833 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:52.657+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:53.086 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:52.831+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:52.832140+0000) 2024-04-27T03:55:53.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:53.087+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:53.088293+0000) 2024-04-27T03:55:53.339 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:53.086+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:53.087342+0000) 2024-04-27T03:55:53.339 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:55:53.093+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76307 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:55:53.339 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:53.088+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:53.089032+0000) 2024-04-27T03:55:53.339 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:53.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:53.339 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:53.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:53.340 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:53.334+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:53.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:53.335+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:53.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:53.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:53.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:53.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:53.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:53.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:53.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:53.528+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:53.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:53.541+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:53.543477+0000) 2024-04-27T03:55:53.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:53.832+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:53.834444+0000) 2024-04-27T03:55:53.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:53.614+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:53.616394+0000) 2024-04-27T03:55:53.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:53.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:53.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:53.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:53.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:53.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:53.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:53.650+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:54.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:54.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:54.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:54.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:54.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:54.358+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:54.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:54.359+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:54.839 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:54.619+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:54.620950+0000) 2024-04-27T03:55:54.840 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:54.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:54.840 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:54.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:54.840 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:54.689+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:54.840 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:54.690+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:54.840 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:54.508+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:54.841 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:54.508+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:54.841 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:54.508+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:54.841 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:54.509+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:54.841 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:54.546+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:54.548114+0000) 2024-04-27T03:55:55.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:54.836+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:54.837904+0000) 2024-04-27T03:55:55.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:55.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:55.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:55.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:55.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:55.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:55.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:55.359+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:55.842 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:55.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:55.842 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:55.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:55.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:55.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:55.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:55.530+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:55.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:55.550+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:55.552343+0000) 2024-04-27T03:55:55.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:55.621+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:55.623034+0000) 2024-04-27T03:55:55.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:55.654+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:55.843 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:55.654+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:55.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:55.654+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:55.844 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:55.656+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:56.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:55.838+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:55.840598+0000) 2024-04-27T03:55:56.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:56.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:56.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:56.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:56.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:56.396+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:56.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:56.397+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:56.846 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:56.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:56.846 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:56.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:56.846 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:56.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:56.846 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:56.543+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:56.846 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:56.553+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:56.555255+0000) 2024-04-27T03:55:56.847 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:56.623+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:56.625713+0000) 2024-04-27T03:55:56.847 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:56.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:56.847 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:56.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:56.847 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:56.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:56.847 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:56.679+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:57.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:56.842+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:56.844025+0000) 2024-04-27T03:55:57.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:57.434+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:57.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:57.434+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:57.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:57.434+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:57.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:57.435+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:57.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:57.558+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:57.560247+0000) 2024-04-27T03:55:57.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:57.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:57.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:57.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:57.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:57.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:57.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:57.575+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:57.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:57.629+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:57.630815+0000) 2024-04-27T03:55:57.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:57.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:57.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:57.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:57.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:57.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:57.725 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:57.647+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:58.101 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:57.847+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:57.849054+0000) 2024-04-27T03:55:58.356 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:58.097+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:58.099262+0000) 2024-04-27T03:55:58.356 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:58.098+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:58.099925+0000) 2024-04-27T03:55:58.357 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:55:58.102+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76348 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:55:58.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:58.097+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:58.099166+0000) 2024-04-27T03:55:58.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:58.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:58.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:58.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:58.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:58.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:58.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:58.445+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:58.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:58.562+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:58.564571+0000) 2024-04-27T03:55:58.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:58.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:58.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:58.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:58.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:58.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:58.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:58.565+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:58.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:58.849+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:58.851260+0000) 2024-04-27T03:55:58.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:58.631+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:58.632998+0000) 2024-04-27T03:55:58.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:58.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:58.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:58.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:58.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:58.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:58.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:58.697+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:59.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:59.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:59.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:59.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:59.638 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:59.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:55:59.638 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:59.407+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:59.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:59.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:59.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:59.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:59.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:59.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:55:59.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:59.543+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:55:59.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:55:59.564+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:59.566760+0000) 2024-04-27T03:55:59.639 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:59.633+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:59.635769+0000) 2024-04-27T03:55:59.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:55:59.852+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:55:59.853978+0000) 2024-04-27T03:55:59.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:59.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:59.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:59.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:59.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:59.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:55:59.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:55:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:55:59.720+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:00.639 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:00.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:00.640 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:00.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:00.640 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:00.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:00.640 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:00.389+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:00.640 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:00.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:00.640 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:00.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:00.640 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:00.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:00.640 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:00.562+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:00.641 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:00.569+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:00.570992+0000) 2024-04-27T03:56:00.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:00.854+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:00.856285+0000) 2024-04-27T03:56:00.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:00.636+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:00.637900+0000) 2024-04-27T03:56:00.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:00.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:00.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:00.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:00.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:00.751+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:00.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:00.751+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:01.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:01.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:01.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:01.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:01.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:01.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:01.643 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:01.517+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:01.643 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:01.573+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:01.575218+0000) 2024-04-27T03:56:01.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:01.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:01.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:01.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:01.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:01.378+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:01.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:01.379+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:01.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:01.856+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:01.858418+0000) 2024-04-27T03:56:01.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:01.638+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:01.640369+0000) 2024-04-27T03:56:01.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:01.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:01.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:01.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:01.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:01.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:01.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:01.783+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:02.644 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:02.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:02.644 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:02.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:02.644 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:02.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:02.645 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:02.339+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:02.645 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:02.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:02.645 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:02.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:02.645 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:02.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:02.645 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:02.529+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:02.645 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:02.575+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:02.577289+0000) 2024-04-27T03:56:02.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:02.858+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:02.860664+0000) 2024-04-27T03:56:02.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:02.640+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:02.642664+0000) 2024-04-27T03:56:02.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:02.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:02.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:02.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:02.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:02.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:02.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:02.816+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:03.356 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:56:03.111+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76387 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:56:03.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:03.108+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:03.109952+0000) 2024-04-27T03:56:03.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:03.107+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:03.109024+0000) 2024-04-27T03:56:03.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:03.107+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:03.109676+0000) 2024-04-27T03:56:03.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:03.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:03.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:03.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:03.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:03.364+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:03.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:03.365+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:03.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:03.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:03.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:03.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:03.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:03.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:03.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:03.536+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:03.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:03.580+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:03.581515+0000) 2024-04-27T03:56:03.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:03.862+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:03.863414+0000) 2024-04-27T03:56:03.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:03.644+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:03.644891+0000) 2024-04-27T03:56:03.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:03.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:03.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:03.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:03.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:03.813+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:03.948 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:03.814+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:04.491 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:04.357+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:04.492 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:04.357+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:04.492 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:04.357+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:04.492 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:04.357+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:04.779 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:04.647+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:04.649407+0000) 2024-04-27T03:56:04.779 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:04.488+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:04.779 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:04.488+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:04.780 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:04.488+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:04.780 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:04.489+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:04.780 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:04.583+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:04.583838+0000) 2024-04-27T03:56:05.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:04.863+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:04.865641+0000) 2024-04-27T03:56:05.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:04.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:05.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:04.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:05.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:04.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:05.069 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:04.776+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:05.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:05.314+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:05.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:05.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:05.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:05.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:05.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:05.315+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:05.655 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:05.539+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:05.655 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:05.539+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:05.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:05.539+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:05.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:05.540+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:05.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:05.585+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:05.586081+0000) 2024-04-27T03:56:05.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:05.867+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:05.868392+0000) 2024-04-27T03:56:05.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:05.653+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:05.653801+0000) 2024-04-27T03:56:05.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:05.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:05.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:05.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:05.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:05.760+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:05.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:05.761+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:06.356 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:06.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:06.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:06.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:06.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:06.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:06.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:06.279+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:06.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:06.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:06.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:06.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:06.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:06.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:06.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:06.509+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:06.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:06.587+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:06.588751+0000) 2024-04-27T03:56:06.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:06.870+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:06.871334+0000) 2024-04-27T03:56:06.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:06.655+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:06.656546+0000) 2024-04-27T03:56:06.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:06.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:06.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:06.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:06.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:06.756+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:06.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:06.757+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:07.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:07.273+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:07.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:07.273+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:07.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:07.273+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:07.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:07.273+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:07.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:07.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:07.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:07.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:07.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:07.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:07.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:07.530+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:07.662 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:07.590+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:07.591079+0000) 2024-04-27T03:56:07.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:07.873+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:07.874272+0000) 2024-04-27T03:56:07.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:07.658+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:07.659660+0000) 2024-04-27T03:56:07.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:07.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:07.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:07.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:07.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:07.720+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:07.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:07.721+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:08.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:08.118+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:08.118840+0000) 2024-04-27T03:56:08.357 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:56:08.122+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76428 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:56:08.358 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:08.116+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:08.117424+0000) 2024-04-27T03:56:08.358 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:08.117+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:08.118127+0000) 2024-04-27T03:56:08.358 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:08.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:08.358 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:08.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:08.358 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:08.228+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:08.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:08.229+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:08.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:08.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:08.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:08.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:08.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:08.557+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:08.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:08.559+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:08.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:08.592+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:08.593273+0000) 2024-04-27T03:56:08.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:08.877+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:08.878577+0000) 2024-04-27T03:56:08.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:08.661+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:08.662494+0000) 2024-04-27T03:56:08.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:08.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:08.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:08.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:08.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:08.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:08.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:08.707+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:09.344 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:09.191+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:09.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:09.191+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:09.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:09.191+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:09.345 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:09.191+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:09.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:09.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:09.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:09.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:09.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:09.518+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:09.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:09.519+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:09.882 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:09.597+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:09.597956+0000) 2024-04-27T03:56:09.882 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:09.666+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:09.667190+0000) 2024-04-27T03:56:09.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:09.697+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:09.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:09.697+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:09.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:09.697+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:09.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:09.698+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:10.148 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:09.880+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:09.880832+0000) 2024-04-27T03:56:10.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:10.195+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:10.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:10.195+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:10.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:10.195+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:10.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:10.196+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:10.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:10.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:10.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:10.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:10.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:10.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:10.887 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:10.534+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:10.888 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:10.599+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:10.600267+0000) 2024-04-27T03:56:10.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:10.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:10.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:10.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:10.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:10.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:10.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:10.663+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:10.888 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:10.670+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:10.671514+0000) 2024-04-27T03:56:11.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:10.884+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:10.885612+0000) 2024-04-27T03:56:11.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:11.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:11.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:11.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:11.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:11.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:11.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:11.206+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:11.891 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:11.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:11.891 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:11.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:11.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:11.553+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:11.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:11.553+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:11.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:11.601+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:11.602425+0000) 2024-04-27T03:56:11.892 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:11.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:11.892 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:11.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:11.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:11.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:11.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:11.646+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:11.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:11.672+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:11.673718+0000) 2024-04-27T03:56:12.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:11.888+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:11.889729+0000) 2024-04-27T03:56:12.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:12.235+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:12.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:12.235+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:12.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:12.235+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:12.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:12.236+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:12.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:12.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:12.894 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:12.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:12.894 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:12.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:12.894 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:12.528+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:12.894 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:12.605+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:12.606724+0000) 2024-04-27T03:56:12.894 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:12.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:12.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:12.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:12.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:12.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:12.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:12.647+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:12.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:12.675+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:12.675840+0000) 2024-04-27T03:56:13.150 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:56:13.131+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76467 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:56:13.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:13.128+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:13.128875+0000) 2024-04-27T03:56:13.151 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:13.127+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:13.128631+0000) 2024-04-27T03:56:13.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:12.891+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:12.892152+0000) 2024-04-27T03:56:13.152 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:13.126+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:13.127198+0000) 2024-04-27T03:56:13.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:13.210+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:13.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:13.210+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:13.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:13.210+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:13.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:13.210+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:13.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:13.612+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:13.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:13.612+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:13.897 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:13.612+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:13.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:13.613+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:13.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:13.677+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:13.678572+0000) 2024-04-27T03:56:13.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:13.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:13.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:13.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:13.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:13.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:13.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:13.483+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:13.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:13.608+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:13.608931+0000) 2024-04-27T03:56:14.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:13.895+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:13.895793+0000) 2024-04-27T03:56:14.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:14.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:14.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:14.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:14.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:14.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:14.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:14.237+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:14.901 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:14.603+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:14.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:14.603+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:14.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:14.603+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:14.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:14.603+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:14.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:14.682+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:14.683239+0000) 2024-04-27T03:56:14.902 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:14.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:14.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:14.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:14.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:14.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:14.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:14.531+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:14.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:14.610+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:14.611002+0000) 2024-04-27T03:56:15.170 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:14.899+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:14.899985+0000) 2024-04-27T03:56:15.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:15.207+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:15.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:15.207+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:15.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:15.207+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:15.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:15.207+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:15.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:15.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:15.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:15.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:15.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:15.627+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:15.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:15.628+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:15.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:15.684+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:15.685496+0000) 2024-04-27T03:56:15.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:15.551+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:15.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:15.551+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:15.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:15.551+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:15.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:15.551+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:15.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:15.612+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:15.613225+0000) 2024-04-27T03:56:16.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:15.901+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:15.902357+0000) 2024-04-27T03:56:16.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:16.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:16.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:16.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:16.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:16.161+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:16.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:16.162+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:16.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:16.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:16.907 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:16.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:16.907 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:16.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:16.907 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:16.547+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:16.907 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:16.616+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:16.617481+0000) 2024-04-27T03:56:16.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:16.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:16.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:16.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:16.908 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:16.657+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:16.908 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:16.657+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:16.908 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:16.689+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:16.690260+0000) 2024-04-27T03:56:17.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:16.904+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:16.905107+0000) 2024-04-27T03:56:17.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:17.163+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:17.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:17.163+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:17.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:17.163+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:17.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:17.163+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:17.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:17.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:17.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:17.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:17.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:17.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:17.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:17.534+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:17.910 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:17.620+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:17.621732+0000) 2024-04-27T03:56:17.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:17.665+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:17.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:17.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:17.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:17.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:17.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:17.667+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:17.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:17.692+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:17.692909+0000) 2024-04-27T03:56:18.163 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:56:18.140+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76508 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:56:18.164 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:18.137+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:18.138079+0000) 2024-04-27T03:56:18.164 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:18.136+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:18.136850+0000) 2024-04-27T03:56:18.164 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:17.906+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:17.907648+0000) 2024-04-27T03:56:18.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:18.135+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:18.136407+0000) 2024-04-27T03:56:18.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:18.197+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:18.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:18.197+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:18.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:18.197+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:18.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:18.198+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:18.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:18.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:18.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:18.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:18.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:18.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:18.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:18.543+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:18.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:18.623+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:18.623841+0000) 2024-04-27T03:56:18.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:18.696+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:18.697788+0000) 2024-04-27T03:56:18.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:18.703+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:18.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:18.703+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:18.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:18.703+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:18.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:18.705+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:19.169 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:18.909+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:18.910153+0000) 2024-04-27T03:56:19.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:19.166+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:19.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:19.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:19.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:19.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:19.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:19.167+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:19.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:19.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:19.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:19.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:19.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:19.554+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:19.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:19.556+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:19.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:19.625+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:19.625962+0000) 2024-04-27T03:56:19.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:19.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:19.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:19.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:19.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:19.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:19.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:19.674+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:19.724 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:19.699+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:19.700457+0000) 2024-04-27T03:56:20.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:19.912+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:19.912926+0000) 2024-04-27T03:56:20.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:20.163+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:20.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:20.163+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:20.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:20.163+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:20.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:20.164+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:20.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:20.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:20.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:20.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:20.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:20.574+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:20.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:20.575+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:20.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:20.627+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:20.628037+0000) 2024-04-27T03:56:20.918 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:20.697+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:20.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:20.697+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:20.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:20.697+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:20.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:20.697+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:20.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:20.702+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:20.703423+0000) 2024-04-27T03:56:21.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:20.915+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:20.916096+0000) 2024-04-27T03:56:21.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:21.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:21.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:21.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:21.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:21.145+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:21.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:21.145+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:21.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:21.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:21.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:21.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:21.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:21.537+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:21.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:21.539+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:21.920 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:21.629+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:21.630092+0000) 2024-04-27T03:56:21.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:21.704+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:21.705515+0000) 2024-04-27T03:56:21.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:21.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:21.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:21.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:21.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:21.710+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:21.921 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:21.712+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:22.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:21.917+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:21.918382+0000) 2024-04-27T03:56:22.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:22.122+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:22.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:22.122+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:22.171 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:22.122+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:22.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:22.123+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:22.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:22.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:22.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:22.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:22.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:22.491+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:22.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:22.492+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:22.926 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:22.632+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:22.632813+0000) 2024-04-27T03:56:22.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:22.709+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:22.710623+0000) 2024-04-27T03:56:22.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:22.714+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:22.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:22.714+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:22.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:22.714+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:22.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:22.715+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:23.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:23.146+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:23.146931+0000) 2024-04-27T03:56:23.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:23.145+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:23.146376+0000) 2024-04-27T03:56:23.223 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:56:23.150+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76547 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:56:23.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:22.922+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:22.923507+0000) 2024-04-27T03:56:23.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:23.145+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:23.146160+0000) 2024-04-27T03:56:23.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:23.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:23.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:23.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:23.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:23.149+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:23.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:23.149+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:23.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:23.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:23.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:23.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:23.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:23.542+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:23.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:23.543+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:23.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:23.635+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:23.636050+0000) 2024-04-27T03:56:23.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:23.715+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:23.716033+0000) 2024-04-27T03:56:23.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:23.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:23.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:23.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:23.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:23.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:23.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:23.746+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:24.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:23.927+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:23.928475+0000) 2024-04-27T03:56:24.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:24.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:24.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:24.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:24.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:24.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:24.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:24.148+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:24.783 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:24.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:24.783 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:24.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:24.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:24.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:24.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:24.510+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:24.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:24.637+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:24.638732+0000) 2024-04-27T03:56:24.784 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:24.717+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:24.718672+0000) 2024-04-27T03:56:24.784 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:24.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:24.784 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:24.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:24.784 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:24.781+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:25.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:24.781+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:25.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:24.930+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:24.931536+0000) 2024-04-27T03:56:25.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:25.138+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:25.361 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:25.138+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:25.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:25.138+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:25.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:25.139+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:25.643 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:25.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:25.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:25.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:25.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:25.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:25.644 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:25.515+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:25.935 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:25.641+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:25.642039+0000) 2024-04-27T03:56:25.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:25.721+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:25.721903+0000) 2024-04-27T03:56:25.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:25.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:25.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:25.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:25.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:25.758+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:25.936 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:25.759+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:26.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:25.933+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:25.933823+0000) 2024-04-27T03:56:26.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:26.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:26.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:26.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:26.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:26.184+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:26.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:26.185+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:26.778 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:26.723+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:26.724531+0000) 2024-04-27T03:56:26.778 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:26.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:26.779 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:26.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:26.779 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:26.776+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:26.779 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:26.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:26.779 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:26.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:26.779 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:26.481+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:26.779 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:26.482+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:26.780 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:26.644+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:26.645166+0000) 2024-04-27T03:56:27.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:26.776+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:27.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:26.936+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:26.936911+0000) 2024-04-27T03:56:27.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:27.224+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:27.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:27.224+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:27.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:27.224+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:27.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:27.225+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:27.649 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:27.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:27.650 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:27.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:27.650 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:27.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:27.650 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:27.441+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:27.650 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:27.647+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:27.648214+0000) 2024-04-27T03:56:27.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:27.725+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:27.726758+0000) 2024-04-27T03:56:27.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:27.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:27.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:27.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:27.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:27.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:27.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:27.823+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:28.222 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:56:28.159+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76588 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:56:28.223 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:28.154+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:28.155585+0000) 2024-04-27T03:56:28.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:28.155+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:28.156326+0000) 2024-04-27T03:56:28.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:27.938+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:27.939226+0000) 2024-04-27T03:56:28.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:28.154+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:28.155421+0000) 2024-04-27T03:56:28.572 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:28.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:28.572 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:28.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:28.573 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:28.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:28.573 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:28.414+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:28.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:28.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:28.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:28.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:28.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:28.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:28.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:28.262+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:28.827 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:28.649+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:28.650627+0000) 2024-04-27T03:56:28.828 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:28.728+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:28.728925+0000) 2024-04-27T03:56:28.828 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:28.824+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:28.828 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:28.824+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:28.828 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:28.825+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:29.171 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:28.826+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:29.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:28.941+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:28.942264+0000) 2024-04-27T03:56:29.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:29.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:29.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:29.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:29.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:29.279+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:29.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:29.279+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:29.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:29.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:29.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:29.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:29.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:29.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:29.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:29.452+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:29.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:29.654+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:29.654825+0000) 2024-04-27T03:56:30.071 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:29.946+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:29.946984+0000) 2024-04-27T03:56:30.071 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:29.730+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:29.731029+0000) 2024-04-27T03:56:30.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:29.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:30.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:29.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:30.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:29.823+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:30.072 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:29.824+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:30.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:30.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:30.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:30.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:30.359 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:30.310+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:30.360 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:30.310+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:30.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:30.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:30.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:30.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:30.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:30.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:30.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:30.420+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:30.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:30.656+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:30.657542+0000) 2024-04-27T03:56:30.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:30.732+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:30.733571+0000) 2024-04-27T03:56:30.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:30.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:30.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:30.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:30.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:30.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:30.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:30.840+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:31.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:30.950+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:30.951230+0000) 2024-04-27T03:56:31.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:31.394+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:31.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:31.394+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:31.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:31.394+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:31.580 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:31.395+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:31.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:31.267+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:31.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:31.267+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:31.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:31.267+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:31.581 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:31.268+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:31.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:31.659+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:31.660740+0000) 2024-04-27T03:56:31.891 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:31.734+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:31.735653+0000) 2024-04-27T03:56:32.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:31.954+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:31.955445+0000) 2024-04-27T03:56:32.172 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:31.888+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:32.172 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:31.888+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:32.173 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:31.888+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:32.173 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:31.889+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:32.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:32.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:32.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:32.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:32.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:32.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:32.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:32.237+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:32.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:32.352+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:32.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:32.352+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:32.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:32.352+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:32.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:32.354+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:32.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:32.664+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:32.665023+0000) 2024-04-27T03:56:32.931 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:32.738+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:32.738960+0000) 2024-04-27T03:56:33.191 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:56:33.169+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76627 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:56:33.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:33.164+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:33.164861+0000) 2024-04-27T03:56:33.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:32.957+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:32.958412+0000) 2024-04-27T03:56:33.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:33.164+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:33.165020+0000) 2024-04-27T03:56:33.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:32.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:33.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:32.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:33.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:32.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:33.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:32.930+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:33.193 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:33.164+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:33.165222+0000) 2024-04-27T03:56:33.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:33.310+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:33.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:33.310+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:33.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:33.310+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:33.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:33.312+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:33.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:33.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:33.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:33.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:33.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:33.190+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:33.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:33.191+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:33.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:33.740+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:33.741199+0000) 2024-04-27T03:56:33.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:33.669+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:33.669823+0000) 2024-04-27T03:56:34.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:33.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:34.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:33.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:34.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:33.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:34.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:33.975+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:34.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:33.959+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:33.960620+0000) 2024-04-27T03:56:34.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:34.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:34.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:34.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:34.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:34.151+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:34.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:34.152+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:34.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:34.314+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:34.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:34.314+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:34.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:34.314+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:34.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:34.315+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:34.965 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:34.673+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:34.674032+0000) 2024-04-27T03:56:34.965 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:34.743+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:34.744163+0000) 2024-04-27T03:56:35.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:34.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:35.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:34.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:35.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:34.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:35.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:34.991+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:35.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:34.962+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:34.963633+0000) 2024-04-27T03:56:35.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:35.123+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:35.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:35.123+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:35.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:35.123+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:35.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:35.123+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:35.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:35.285+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:35.588 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:35.285+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:35.588 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:35.285+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:35.588 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:35.285+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:35.839 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:35.675+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:35.676752+0000) 2024-04-27T03:56:35.839 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:35.746+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:35.746867+0000) 2024-04-27T03:56:36.103 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:35.965+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:35.966020+0000) 2024-04-27T03:56:36.103 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:35.981+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:36.103 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:35.981+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:36.103 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:35.981+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:36.103 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:35.983+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:36.360 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:36.324+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:36.360 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:36.324+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:36.360 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:36.324+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:36.360 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:36.325+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:36.361 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:36.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:36.361 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:36.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:36.361 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:36.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:36.361 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:36.101+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:36.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:36.750+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:36.751194+0000) 2024-04-27T03:56:36.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:36.678+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:36.678961+0000) 2024-04-27T03:56:37.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:36.967+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:36.968419+0000) 2024-04-27T03:56:37.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:37.119+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:37.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:37.119+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:37.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:37.119+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:37.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:37.120+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:37.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:37.012+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:37.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:37.012+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:37.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:37.012+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:37.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:37.013+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:37.589 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:37.325+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:37.589 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:37.325+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:37.589 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:37.325+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:37.589 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:37.325+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:37.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:37.681+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:37.682089+0000) 2024-04-27T03:56:37.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:37.753+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:37.754229+0000) 2024-04-27T03:56:38.245 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:56:38.177+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76668 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:56:38.245 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:38.173+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:38.174781+0000) 2024-04-27T03:56:38.246 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:37.991+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:38.246 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:37.991+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:38.246 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:37.991+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:38.246 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:37.993+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:38.246 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:38.173+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:38.174781+0000) 2024-04-27T03:56:38.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:37.970+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:37.970855+0000) 2024-04-27T03:56:38.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:38.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:38.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:38.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:38.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:38.100+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:38.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:38.101+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:38.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:38.175+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:38.176217+0000) 2024-04-27T03:56:38.589 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:38.298+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:38.589 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:38.298+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:38.589 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:38.298+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:38.589 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:38.299+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:38.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:38.683+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:38.684467+0000) 2024-04-27T03:56:38.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:38.756+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:38.757177+0000) 2024-04-27T03:56:39.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:38.964+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:39.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:38.964+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:39.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:38.964+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:39.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:38.965+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:39.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:38.972+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:38.973417+0000) 2024-04-27T03:56:39.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:39.087+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:39.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:39.087+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:39.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:39.087+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:39.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:39.088+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:39.593 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:39.273+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:39.593 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:39.273+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:39.593 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:39.273+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:39.593 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:39.275+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:39.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:39.685+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:39.686624+0000) 2024-04-27T03:56:39.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:39.758+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:39.759311+0000) 2024-04-27T03:56:40.245 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:39.974+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:39.975650+0000) 2024-04-27T03:56:40.246 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:40.060+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:40.246 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:40.060+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:40.246 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:40.060+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:40.246 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:40.061+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:40.246 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:39.999+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:40.247 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:39.999+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:40.247 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:39.999+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:40.247 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:40.000+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:40.598 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:40.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:40.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:40.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:40.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:40.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:40.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:40.321+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:40.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:40.690+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:40.690871+0000) 2024-04-27T03:56:40.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:40.762+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:40.763165+0000) 2024-04-27T03:56:40.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:40.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:40.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:40.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:40.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:40.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:41.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:40.970+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:41.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:40.977+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:40.977910+0000) 2024-04-27T03:56:41.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:41.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:41.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:41.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:41.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:41.084+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:41.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:41.084+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:41.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:41.278+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:41.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:41.278+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:41.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:41.278+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:41.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:41.279+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:41.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:41.692+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:41.692939+0000) 2024-04-27T03:56:41.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:41.764+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:41.765653+0000) 2024-04-27T03:56:42.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:41.959+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:42.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:41.960+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:42.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:41.960+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:42.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:41.961+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:42.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:41.979+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:41.980210+0000) 2024-04-27T03:56:42.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:42.095+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:42.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:42.095+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:42.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:42.095+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:42.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:42.096+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:42.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:42.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:42.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:42.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:42.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:42.280+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:42.600 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:42.281+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:42.942 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:42.694+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:42.695126+0000) 2024-04-27T03:56:42.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:42.767+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:42.767832+0000) 2024-04-27T03:56:42.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:42.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:42.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:42.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:42.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:42.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:43.194 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:56:43.186+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76707 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:56:43.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:43.183+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:43.184778+0000) 2024-04-27T03:56:43.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:42.981+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:42.982621+0000) 2024-04-27T03:56:43.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:43.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:43.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:43.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:43.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:43.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:43.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:43.119+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:43.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:43.183+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:43.184530+0000) 2024-04-27T03:56:43.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:42.942+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:43.196 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:43.182+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:43.183388+0000) 2024-04-27T03:56:43.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:43.291+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:43.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:43.291+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:43.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:43.291+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:43.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:43.293+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:43.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:43.696+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:43.697339+0000) 2024-04-27T03:56:43.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:43.769+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:43.769991+0000) 2024-04-27T03:56:44.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:43.984+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:43.984864+0000) 2024-04-27T03:56:44.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:44.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:44.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:44.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:44.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:44.148+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:44.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:44.148+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:44.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:43.969+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:44.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:43.969+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:44.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:43.969+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:44.224 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:43.969+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:44.604 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:44.313+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:44.604 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:44.313+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:44.605 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:44.313+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:44.605 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:44.314+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:44.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:44.699+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:44.700300+0000) 2024-04-27T03:56:44.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:44.771+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:44.772747+0000) 2024-04-27T03:56:44.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:44.964+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:44.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:44.964+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:44.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:44.964+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:45.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:44.965+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:45.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:44.986+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:44.987081+0000) 2024-04-27T03:56:45.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:45.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:45.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:45.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:45.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:45.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:45.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:45.135+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:45.607 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:45.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:45.608 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:45.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:45.608 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:45.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:45.608 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:45.339+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:45.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:45.702+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:45.702910+0000) 2024-04-27T03:56:45.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:45.774+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:45.775065+0000) 2024-04-27T03:56:46.246 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:45.988+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:45.989083+0000) 2024-04-27T03:56:46.246 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:46.134+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:46.246 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:46.134+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:46.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:46.134+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:46.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:46.134+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:46.247 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:45.983+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:46.247 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:45.983+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:46.247 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:45.983+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:46.247 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:45.983+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:46.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:46.291+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:46.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:46.291+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:46.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:46.291+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:46.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:46.291+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:46.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:46.704+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:46.705296+0000) 2024-04-27T03:56:46.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:46.777+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:46.778198+0000) 2024-04-27T03:56:47.246 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:46.991+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:46.992178+0000) 2024-04-27T03:56:47.246 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:47.088+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:47.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:47.088+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:47.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:47.088+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:47.247 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:47.089+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:47.247 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:46.986+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:47.247 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:46.986+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:47.247 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:46.986+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:47.247 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:46.987+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:47.608 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:47.308+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:47.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:47.308+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:47.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:47.308+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:47.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:47.310+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:47.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:47.707+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:47.708249+0000) 2024-04-27T03:56:47.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:47.780+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:47.780902+0000) 2024-04-27T03:56:48.246 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:56:48.195+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76748 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:56:48.247 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:48.191+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:48.192549+0000) 2024-04-27T03:56:48.247 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:48.028+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:48.247 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:48.028+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:48.247 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:48.028+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:48.247 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:48.029+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:48.248 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:48.191+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:48.191908+0000) 2024-04-27T03:56:48.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:47.994+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:47.995002+0000) 2024-04-27T03:56:48.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:48.092+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:48.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:48.092+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:48.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:48.092+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:48.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:48.093+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:48.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:48.191+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:48.192761+0000) 2024-04-27T03:56:48.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:48.325+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:48.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:48.325+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:48.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:48.325+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:48.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:48.326+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:48.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:48.709+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:48.710697+0000) 2024-04-27T03:56:48.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:48.783+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:48.784097+0000) 2024-04-27T03:56:49.361 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:48.999+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:49.361 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:48.999+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:49.361 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:48.999+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:49.361 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:49.000+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:49.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:48.997+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:48.998165+0000) 2024-04-27T03:56:49.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:49.105+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:49.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:49.105+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:49.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:49.105+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:49.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:49.106+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:49.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:49.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:49.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:49.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:49.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:49.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:49.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:49.370+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:49.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:49.712+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:49.712840+0000) 2024-04-27T03:56:49.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:49.785+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:49.786356+0000) 2024-04-27T03:56:50.361 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:49.999+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:50.000297+0000) 2024-04-27T03:56:50.361 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:50.144+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:50.361 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:50.144+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:50.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:50.144+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:50.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:50.144+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:50.362 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:50.006+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:50.362 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:50.006+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:50.362 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:50.006+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:50.362 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:50.006+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:50.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:50.394+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:50.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:50.394+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:50.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:50.394+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:50.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:50.395+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:50.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:50.788+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:50.789322+0000) 2024-04-27T03:56:50.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:50.714+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:50.715499+0000) 2024-04-27T03:56:51.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:50.966+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:51.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:50.966+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:51.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:50.966+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:51.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:50.967+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:51.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:51.001+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:51.002543+0000) 2024-04-27T03:56:51.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:51.139+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:51.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:51.139+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:51.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:51.139+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:51.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:51.140+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:51.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:51.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:51.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:51.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:51.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:51.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:51.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:51.428+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:51.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:51.718+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:51.719764+0000) 2024-04-27T03:56:51.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:51.790+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:51.791460+0000) 2024-04-27T03:56:52.361 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:52.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:52.361 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:52.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:52.362 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:52.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:52.362 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:52.016+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:52.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:52.003+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:52.004732+0000) 2024-04-27T03:56:52.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:52.172+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:52.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:52.172+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:52.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:52.172+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:52.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:52.173+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:52.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:52.404+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:52.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:52.404+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:52.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:52.404+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:52.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:52.405+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:52.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:52.721+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:52.721845+0000) 2024-04-27T03:56:52.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:52.795+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:52.796599+0000) 2024-04-27T03:56:53.324 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:56:53.204+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76788 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:56:53.325 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:53.200+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:53.200878+0000) 2024-04-27T03:56:53.325 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:53.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:53.326 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:53.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:53.326 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:53.016+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:53.326 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:53.017+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:53.326 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:53.199+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:53.200675+0000) 2024-04-27T03:56:53.326 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:53.006+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:53.006906+0000) 2024-04-27T03:56:53.326 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:53.200+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:53.201086+0000) 2024-04-27T03:56:53.327 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:53.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:53.327 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:53.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:53.327 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:53.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:53.327 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:53.205+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:53.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:53.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:53.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:53.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:53.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:53.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:53.616 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:53.421+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:53.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:53.798+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:53.798821+0000) 2024-04-27T03:56:53.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:53.723+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:53.723987+0000) 2024-04-27T03:56:54.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:54.008+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:54.009023+0000) 2024-04-27T03:56:54.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:54.033+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:54.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:54.033+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:54.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:54.033+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:54.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:54.033+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:54.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:54.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:54.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:54.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:54.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:54.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:54.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:54.236+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:54.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:54.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:54.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:54.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:54.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:54.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:54.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:54.399+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:54.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:54.801+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:54.803393+0000) 2024-04-27T03:56:54.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:54.724+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:54.726026+0000) 2024-04-27T03:56:55.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:55.010+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:55.012211+0000) 2024-04-27T03:56:55.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:55.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:55.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:55.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:55.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:55.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:55.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:55.254+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:55.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:55.031+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:55.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:55.031+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:55.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:55.031+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:55.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:55.033+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:55.620 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:55.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:55.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:55.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:55.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:55.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:55.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:55.432+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:55.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:55.726+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:55.728696+0000) 2024-04-27T03:56:55.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:55.806+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:55.808102+0000) 2024-04-27T03:56:56.252 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:56.013+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:56.014953+0000) 2024-04-27T03:56:56.252 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:56.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:56.252 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:56.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:56.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:56.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:56.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:56.249+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:56.253 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:55.999+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:56.253 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:55.999+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:56.253 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:55.999+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:56.253 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:56.000+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:56.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:56.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:56.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:56.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:56.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:56.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:56.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:56.456+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:56.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:56.730+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:56.731923+0000) 2024-04-27T03:56:56.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:56.811+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:56.812939+0000) 2024-04-27T03:56:57.362 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:57.040+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:57.362 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:57.040+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:57.362 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:57.040+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:57.362 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:57.041+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:57.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:57.016+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:57.018103+0000) 2024-04-27T03:56:57.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:57.220+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:57.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:57.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:57.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:57.221+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:57.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:57.221+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:57.621 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:57.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:57.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:57.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:57.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:57.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:57.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:57.415+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:57.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:57.815+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:57.817780+0000) 2024-04-27T03:56:57.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:57.733+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:57.735329+0000) 2024-04-27T03:56:58.334 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:56:58.211+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76828 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:56:58.335 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:58.208+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:58.210452+0000) 2024-04-27T03:56:58.335 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:58.039+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:58.335 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:58.039+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:58.335 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:58.039+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:58.335 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:58.040+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:58.335 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:58.208+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:58.209919+0000) 2024-04-27T03:56:58.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:58.020+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:58.022346+0000) 2024-04-27T03:56:58.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:58.207+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:58.209652+0000) 2024-04-27T03:56:58.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:58.250+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:58.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:58.250+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:58.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:58.250+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:58.336 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:58.251+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:58.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:58.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:58.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:58.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:58.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:58.432+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:58.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:58.433+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:58.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:58.737+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:58.739635+0000) 2024-04-27T03:56:58.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:58.818+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:58.820487+0000) 2024-04-27T03:56:59.362 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:58.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:59.362 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:58.997+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:59.362 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:58.997+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:56:59.362 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:58.997+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:59.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:59.023+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:59.025305+0000) 2024-04-27T03:56:59.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:59.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:59.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:59.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:59.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:59.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:56:59.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:56:59.225+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:59.626 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:59.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:59.627 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:59.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:59.627 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:59.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:56:59.628 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:59.406+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:56:59.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:56:59.741+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:59.742953+0000) 2024-04-27T03:56:59.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:59.821+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:56:59.823347+0000) 2024-04-27T03:56:59.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:59.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:00.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:00.026+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:00.028204+0000) 2024-04-27T03:57:00.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:00.210+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:00.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:00.210+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:00.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:00.210+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:00.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:00.211+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:00.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:59.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:00.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:59.967+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:00.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:56:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:56:59.969+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:00.629 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:00.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:00.630 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:00.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:00.630 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:00.378+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:00.630 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:00.380+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:00.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:00.743+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:00.745777+0000) 2024-04-27T03:57:00.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:00.823+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:00.825736+0000) 2024-04-27T03:57:01.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:01.029+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:01.031243+0000) 2024-04-27T03:57:01.362 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:01.199+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:01.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:01.199+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:01.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:01.199+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:01.363 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:01.200+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:01.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:01.003+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:01.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:01.003+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:01.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:01.003+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:01.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:01.004+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:01.631 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:01.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:01.631 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:01.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:01.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:01.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:01.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:01.430+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:01.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:01.746+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:01.748516+0000) 2024-04-27T03:57:01.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:01.826+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:01.828503+0000) 2024-04-27T03:57:01.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:01.959+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:01.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:01.959+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:01.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:01.959+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:02.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:02.032+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:02.034245+0000) 2024-04-27T03:57:02.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:01.961+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:02.631 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:02.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:02.631 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:02.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:02.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:02.451+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:02.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:02.452+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:02.632 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:02.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:02.632 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:02.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:02.632 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:02.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:02.632 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:02.247+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:02.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:02.749+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:02.751601+0000) 2024-04-27T03:57:02.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:02.829+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:02.831751+0000) 2024-04-27T03:57:03.239 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:57:03.220+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76868 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:57:03.240 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:03.216+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:03.218642+0000) 2024-04-27T03:57:03.240 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:03.036+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:03.036898+0000) 2024-04-27T03:57:03.240 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:03.216+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:03.218606+0000) 2024-04-27T03:57:03.240 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:02.986+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:03.241 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:02.986+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:03.241 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:02.986+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:03.241 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:02.986+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:03.241 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:03.218+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:03.219920+0000) 2024-04-27T03:57:03.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:03.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:03.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:03.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:03.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:03.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:03.632 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:03.446+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:03.632 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:03.293+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:03.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:03.293+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:03.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:03.293+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:03.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:03.293+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:03.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:03.751+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:03.753753+0000) 2024-04-27T03:57:03.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:03.832+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:03.833911+0000) 2024-04-27T03:57:04.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:04.039+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:04.041158+0000) 2024-04-27T03:57:04.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:04.027+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:04.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:04.027+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:04.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:04.027+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:04.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:04.028+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:04.634 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:04.301+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:04.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:04.301+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:04.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:04.301+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:04.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:04.302+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:04.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:04.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:04.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:04.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:04.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:04.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:04.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:04.458+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:04.897 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:04.754+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:04.756398+0000) 2024-04-27T03:57:04.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:04.834+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:04.836590+0000) 2024-04-27T03:57:05.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:05.042+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:05.044560+0000) 2024-04-27T03:57:05.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:05.017+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:05.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:05.017+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:05.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:05.017+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:05.176 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:05.018+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:05.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:05.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:05.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:05.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:05.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:05.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:05.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:05.308+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:05.760 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:05.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:05.760 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:05.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:05.761 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:05.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:05.761 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:05.507+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:06.049 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:05.757+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:05.759262+0000) 2024-04-27T03:57:06.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:05.838+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:05.839837+0000) 2024-04-27T03:57:06.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:05.987+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:06.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:05.987+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:06.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:05.987+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:06.050 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:05.989+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:06.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:06.046+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:06.048601+0000) 2024-04-27T03:57:06.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:06.484+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:06.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:06.484+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:06.640 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:06.484+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:06.640 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:06.486+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:06.640 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:06.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:06.640 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:06.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:06.640 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:06.339+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:06.640 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:06.339+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:06.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:06.762+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:06.763543+0000) 2024-04-27T03:57:06.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:06.842+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:06.843112+0000) 2024-04-27T03:57:07.297 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:07.030+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:07.297 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:07.030+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:07.297 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:07.030+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:07.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:07.031+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:07.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:07.051+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:07.052385+0000) 2024-04-27T03:57:07.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:07.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:07.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:07.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:07.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:07.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:07.639 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:07.295+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:07.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:07.508+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:07.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:07.508+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:07.640 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:07.508+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:07.640 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:07.509+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:07.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:07.765+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:07.766596+0000) 2024-04-27T03:57:07.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:07.844+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:07.845330+0000) 2024-04-27T03:57:08.355 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:08.227+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:08.228119+0000) 2024-04-27T03:57:08.356 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:57:08.232+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76908 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:57:08.356 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:08.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:08.356 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:08.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:08.356 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:08.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:08.356 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:08.066+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:08.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:08.227+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:08.227856+0000) 2024-04-27T03:57:08.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:08.053+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:08.054690+0000) 2024-04-27T03:57:08.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:08.226+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:08.227068+0000) 2024-04-27T03:57:08.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:08.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:08.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:08.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:08.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:08.282+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:08.357 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:08.282+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:08.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:08.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:08.640 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:08.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:08.640 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:08.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:08.640 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:08.508+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:08.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:08.770+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:08.770844+0000) 2024-04-27T03:57:08.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:08.847+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:08.848080+0000) 2024-04-27T03:57:09.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:09.056+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:09.056935+0000) 2024-04-27T03:57:09.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:09.106+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:09.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:09.106+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:09.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:09.106+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:09.323 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:09.107+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:09.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:09.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:09.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:09.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:09.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:09.521+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:09.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:09.523+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:09.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:09.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:09.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:09.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:09.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:09.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:09.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:09.320+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:09.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:09.772+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:09.772973+0000) 2024-04-27T03:57:09.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:09.849+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:09.850327+0000) 2024-04-27T03:57:10.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:10.058+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:10.059149+0000) 2024-04-27T03:57:10.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:10.102+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:10.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:10.102+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:10.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:10.102+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:10.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:10.103+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:10.646 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:10.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:10.646 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:10.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:10.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:10.482+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:10.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:10.483+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:10.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:10.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:10.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:10.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:10.647 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:10.312+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:10.648 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:10.312+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:10.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:10.774+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:10.775602+0000) 2024-04-27T03:57:10.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:10.851+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:10.852415+0000) 2024-04-27T03:57:11.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:11.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:11.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:11.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:11.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:11.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:11.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:11.066+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:11.364 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:11.061+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:11.062109+0000) 2024-04-27T03:57:11.364 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:11.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:11.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:11.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:11.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:11.264+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:11.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:11.265+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:11.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:11.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:11.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:11.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:11.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:11.446+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:11.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:11.447+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:11.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:11.853+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:11.854761+0000) 2024-04-27T03:57:11.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:11.779+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:11.780631+0000) 2024-04-27T03:57:12.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:12.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:12.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:12.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:12.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:12.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:12.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:12.062+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:12.364 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:12.064+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:12.065321+0000) 2024-04-27T03:57:12.364 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:12.243+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:12.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:12.243+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:12.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:12.243+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:12.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:12.243+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:12.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:12.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:12.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:12.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:12.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:12.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:12.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:12.407+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:12.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:12.784+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:12.785681+0000) 2024-04-27T03:57:12.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:12.856+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:12.856962+0000) 2024-04-27T03:57:13.362 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:57:13.242+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76948 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:57:13.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:13.239+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:13.240194+0000) 2024-04-27T03:57:13.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:13.027+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:13.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:13.027+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:13.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:13.027+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:13.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:13.028+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:13.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:13.237+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:13.238105+0000) 2024-04-27T03:57:13.364 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:13.066+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:13.067549+0000) 2024-04-27T03:57:13.364 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:13.237+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:13.237992+0000) 2024-04-27T03:57:13.364 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:13.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:13.364 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:13.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:13.364 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:13.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:13.364 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:13.263+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:13.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:13.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:13.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:13.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:13.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:13.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:13.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:13.424+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:13.950 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:13.787+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:13.788490+0000) 2024-04-27T03:57:13.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:13.858+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:13.859120+0000) 2024-04-27T03:57:14.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:14.071+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:14.072275+0000) 2024-04-27T03:57:14.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:14.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:14.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:14.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:14.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:14.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:14.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:14.076+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:14.653 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:14.245+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:14.653 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:14.245+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:14.653 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:14.245+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:14.653 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:14.246+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:14.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:14.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:14.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:14.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:14.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:14.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:14.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:14.444+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:14.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:14.860+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:14.861252+0000) 2024-04-27T03:57:14.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:14.789+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:14.790581+0000) 2024-04-27T03:57:15.249 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:15.036+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:15.250 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:15.036+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:15.250 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:15.036+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:15.250 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:15.036+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:15.250 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:15.073+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:15.074511+0000) 2024-04-27T03:57:15.250 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:15.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:15.250 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:15.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:15.251 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:15.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:15.251 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:15.215+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:15.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:15.480+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:15.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:15.480+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:15.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:15.480+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:15.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:15.481+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:15.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:15.794+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:15.795293+0000) 2024-04-27T03:57:15.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:15.863+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:15.863939+0000) 2024-04-27T03:57:16.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:16.049+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:16.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:16.049+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:16.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:16.049+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:16.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:16.050+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:16.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:16.076+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:16.077031+0000) 2024-04-27T03:57:16.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:16.238+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:16.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:16.238+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:16.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:16.238+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:16.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:16.239+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:16.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:16.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:16.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:16.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:16.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:16.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:16.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:16.461+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:16.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:16.798+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:16.799270+0000) 2024-04-27T03:57:16.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:16.865+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:16.866138+0000) 2024-04-27T03:57:17.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:17.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:17.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:17.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:17.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:17.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:17.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:17.014+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:17.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:17.078+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:17.079733+0000) 2024-04-27T03:57:17.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:17.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:17.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:17.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:17.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:17.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:17.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:17.254+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:17.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:17.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:17.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:17.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:17.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:17.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:17.660 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:17.436+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:17.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:17.800+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:17.801457+0000) 2024-04-27T03:57:17.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:17.869+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:17.870545+0000) 2024-04-27T03:57:18.300 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:57:18.250+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 76988 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:57:18.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:18.246+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:18.247418+0000) 2024-04-27T03:57:18.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:18.082+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:18.082983+0000) 2024-04-27T03:57:18.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:18.246+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:18.247463+0000) 2024-04-27T03:57:18.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:18.298+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:18.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:18.298+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:18.301 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:18.048+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:18.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:18.048+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:18.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:18.048+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:18.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:18.049+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:18.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:18.246+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:18.247311+0000) 2024-04-27T03:57:18.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:18.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:18.660 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:18.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:18.660 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:18.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:18.660 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:18.487+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:18.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:18.298+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:18.660 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:18.298+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:18.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:18.804+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:18.805719+0000) 2024-04-27T03:57:18.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:18.871+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:18.872685+0000) 2024-04-27T03:57:19.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:19.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:19.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:19.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:19.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:19.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:19.365 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:19.014+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:19.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:19.085+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:19.086393+0000) 2024-04-27T03:57:19.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:19.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:19.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:19.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:19.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:19.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:19.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:19.262+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:19.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:19.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:19.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:19.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:19.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:19.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:19.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:19.531+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:19.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:19.807+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:19.807838+0000) 2024-04-27T03:57:19.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:19.874+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:19.874888+0000) 2024-04-27T03:57:20.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:20.055+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:20.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:20.055+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:20.364 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:20.055+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:20.365 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:20.056+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:20.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:20.087+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:20.088637+0000) 2024-04-27T03:57:20.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:20.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:20.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:20.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:20.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:20.260+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:20.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:20.260+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:20.666 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:20.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:20.666 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:20.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:20.666 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:20.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:20.666 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:20.494+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:20.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:20.809+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:20.809911+0000) 2024-04-27T03:57:20.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:20.876+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:20.877602+0000) 2024-04-27T03:57:21.308 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:21.007+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:21.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:21.007+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:21.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:21.007+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:21.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:21.009+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:21.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:21.090+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:21.091492+0000) 2024-04-27T03:57:21.309 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:21.306+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:21.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:21.306+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:21.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:21.306+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:21.666 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:21.307+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:21.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:21.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:21.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:21.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:21.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:21.472+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:21.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:21.472+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:21.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:21.811+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:21.812235+0000) 2024-04-27T03:57:21.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:21.879+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:21.879940+0000) 2024-04-27T03:57:22.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:22.093+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:22.094723+0000) 2024-04-27T03:57:22.178 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:21.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:22.178 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:21.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:22.178 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:21.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:22.178 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:21.980+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:22.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:22.268+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:22.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:22.268+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:22.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:22.268+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:22.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:22.268+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:22.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:22.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:22.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:22.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:22.815 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:22.497+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:22.816 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:22.498+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:23.077 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:22.813+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:22.814331+0000) 2024-04-27T03:57:23.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:22.881+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:22.882120+0000) 2024-04-27T03:57:23.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:22.988+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:23.079 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:22.988+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:23.079 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:22.988+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:23.079 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:22.990+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:23.364 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:57:23.260+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77028 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:57:23.365 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:23.255+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:23.256781+0000) 2024-04-27T03:57:23.365 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:23.255+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:23.256788+0000) 2024-04-27T03:57:23.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:23.098+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:23.098997+0000) 2024-04-27T03:57:23.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:23.255+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:23.256555+0000) 2024-04-27T03:57:23.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:23.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:23.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:23.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:23.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:23.285+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:23.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:23.286+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:23.666 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:23.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:23.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:23.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:23.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:23.547+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:23.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:23.547+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:23.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:23.816+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:23.817317+0000) 2024-04-27T03:57:23.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:23.884+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:23.885212+0000) 2024-04-27T03:57:24.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:24.101+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:24.101800+0000) 2024-04-27T03:57:24.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:24.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:24.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:24.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:24.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:24.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:24.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:24.027+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:24.508 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:24.317+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:24.508 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:24.317+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:24.509 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:24.317+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:24.509 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:24.317+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:24.822 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:24.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:24.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:24.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:24.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:24.506+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:24.823 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:24.507+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:25.077 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:24.821+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:24.822029+0000) 2024-04-27T03:57:25.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:24.886+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:24.887500+0000) 2024-04-27T03:57:25.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:25.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:25.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:25.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:25.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:25.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:25.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:25.065+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:25.364 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:25.105+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:25.106010+0000) 2024-04-27T03:57:25.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:25.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:25.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:25.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:25.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:25.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:25.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:25.295+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:25.672 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:25.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:25.672 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:25.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:25.672 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:25.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:25.672 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:25.535+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:25.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:25.825+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:25.826270+0000) 2024-04-27T03:57:25.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:25.890+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:25.891577+0000) 2024-04-27T03:57:26.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:26.109+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:26.110283+0000) 2024-04-27T03:57:26.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:26.280+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:26.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:26.280+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:26.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:26.280+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:26.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:26.281+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:26.365 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:26.056+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:26.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:26.057+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:26.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:26.057+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:26.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:26.057+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:26.672 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:26.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:26.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:26.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:26.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:26.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:26.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:26.487+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:26.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:26.829+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:26.830555+0000) 2024-04-27T03:57:26.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:26.893+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:26.894367+0000) 2024-04-27T03:57:27.365 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:27.090+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:27.365 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:27.090+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:27.365 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:27.090+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:27.365 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:27.091+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:27.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:27.112+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:27.113580+0000) 2024-04-27T03:57:27.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:27.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:27.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:27.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:27.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:27.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:27.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:27.303+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:27.672 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:27.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:27.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:27.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:27.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:27.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:27.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:27.529+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:27.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:27.832+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:27.833632+0000) 2024-04-27T03:57:27.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:27.896+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:27.897242+0000) 2024-04-27T03:57:28.365 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:28.264+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:28.265259+0000) 2024-04-27T03:57:28.365 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:57:28.268+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77068 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:57:28.365 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:28.113+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:28.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:28.113+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:28.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:28.113+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:28.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:28.113+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:28.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:28.264+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:28.265577+0000) 2024-04-27T03:57:28.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:28.117+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:28.117848+0000) 2024-04-27T03:57:28.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:28.264+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:28.265777+0000) 2024-04-27T03:57:28.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:28.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:28.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:28.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:28.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:28.286+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:28.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:28.287+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:28.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:28.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:28.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:28.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:28.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:28.549+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:28.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:28.551+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:28.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:28.835+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:28.836372+0000) 2024-04-27T03:57:28.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:28.899+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:28.900237+0000) 2024-04-27T03:57:29.328 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:29.119+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:29.119953+0000) 2024-04-27T03:57:29.329 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:29.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:29.329 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:29.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:29.329 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:29.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:29.329 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:29.075+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:29.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:29.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:29.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:29.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:29.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:29.326+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:29.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:29.327+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:29.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:29.508+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:29.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:29.508+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:29.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:29.508+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:29.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:29.510+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:29.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:29.839+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:29.840538+0000) 2024-04-27T03:57:29.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:29.901+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:29.902444+0000) 2024-04-27T03:57:30.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:30.121+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:30.122041+0000) 2024-04-27T03:57:30.365 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:30.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:30.365 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:30.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:30.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:30.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:30.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:30.064+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:30.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:30.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:30.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:30.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:30.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:30.504+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:30.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:30.505+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:30.682 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:30.370+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:30.682 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:30.370+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:30.682 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:30.370+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:30.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:30.370+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:30.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:30.842+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:30.842848+0000) 2024-04-27T03:57:30.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:30.903+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:30.904520+0000) 2024-04-27T03:57:31.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:31.123+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:31.124341+0000) 2024-04-27T03:57:31.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:31.057+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:31.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:31.057+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:31.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:31.057+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:31.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:31.058+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:31.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:31.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:31.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:31.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:31.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:31.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:31.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:31.526+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:31.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:31.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:31.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:31.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:31.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:31.388+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:31.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:31.389+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:31.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:31.844+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:31.844983+0000) 2024-04-27T03:57:31.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:31.905+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:31.906608+0000) 2024-04-27T03:57:32.365 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:32.125+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:32.126475+0000) 2024-04-27T03:57:32.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:32.020+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:32.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:32.020+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:32.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:32.020+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:32.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:32.020+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:32.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:32.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:32.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:32.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:32.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:32.526+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:32.684 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:32.527+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:32.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:32.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:32.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:32.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:32.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:32.436+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:32.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:32.437+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:32.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:32.846+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:32.847675+0000) 2024-04-27T03:57:32.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:32.907+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:32.908765+0000) 2024-04-27T03:57:33.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:33.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:33.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:33.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:33.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:33.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:33.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:33.023+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:33.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:33.129+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:33.130710+0000) 2024-04-27T03:57:33.546 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:57:33.277+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77108 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:57:33.546 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:33.272+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:33.273545+0000) 2024-04-27T03:57:33.547 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:33.273+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:33.274757+0000) 2024-04-27T03:57:33.547 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:33.273+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:33.274465+0000) 2024-04-27T03:57:33.547 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:33.484+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:33.547 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:33.484+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:33.547 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:33.484+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:33.548 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:33.485+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:33.852 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:33.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:33.852 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:33.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:33.852 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:33.544+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:33.852 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:33.545+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:34.134 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:33.850+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:33.850900+0000) 2024-04-27T03:57:34.134 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:33.910+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:33.910822+0000) 2024-04-27T03:57:34.134 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:34.036+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:34.134 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:34.036+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:34.134 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:34.036+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:34.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:34.037+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:34.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:34.132+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:34.132920+0000) 2024-04-27T03:57:34.854 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:34.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:34.854 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:34.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:34.854 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:34.530+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:34.854 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:34.531+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:34.855 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:34.498+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:34.855 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:34.498+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:34.855 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:34.498+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:34.855 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:34.498+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:35.136 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:34.852+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:34.853072+0000) 2024-04-27T03:57:35.137 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:34.912+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:34.912912+0000) 2024-04-27T03:57:35.137 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:35.014+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:35.137 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:35.014+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:35.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:35.014+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:35.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:35.015+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:35.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:35.134+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:35.135092+0000) 2024-04-27T03:57:35.856 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:35.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:35.856 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:35.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:35.856 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:35.493+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:35.856 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:35.495+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:35.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:35.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:35.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:35.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:35.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:35.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:35.857 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:35.488+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:36.138 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:35.854+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:35.855218+0000) 2024-04-27T03:57:36.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:35.914+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:35.915058+0000) 2024-04-27T03:57:36.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:35.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:36.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:35.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:36.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:35.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:36.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:35.975+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:36.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:36.136+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:36.137424+0000) 2024-04-27T03:57:36.860 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:36.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:36.860 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:36.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:36.861 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:36.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:36.861 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:36.526+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:36.861 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:36.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:36.862 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:36.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:36.862 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:36.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:36.862 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:36.510+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:37.142 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:36.858+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:36.859365+0000) 2024-04-27T03:57:37.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:36.916+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:36.917176+0000) 2024-04-27T03:57:37.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:37.005+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:37.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:37.005+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:37.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:37.005+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:37.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:37.006+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:37.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:37.140+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:37.140974+0000) 2024-04-27T03:57:37.863 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:37.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:37.864 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:37.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:37.864 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:37.496+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:37.864 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:37.498+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:37.864 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:37.563+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:37.864 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:37.563+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:37.864 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:37.563+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:37.865 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:37.563+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:38.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:37.861+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:37.862602+0000) 2024-04-27T03:57:38.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:37.918+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:37.919591+0000) 2024-04-27T03:57:38.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:38.008+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:38.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:38.008+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:38.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:38.008+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:38.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:38.008+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:38.407 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:57:38.286+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77148 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:57:38.408 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:38.282+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:38.283170+0000) 2024-04-27T03:57:38.408 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:38.282+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:38.282815+0000) 2024-04-27T03:57:38.408 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:38.142+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:38.143745+0000) 2024-04-27T03:57:38.409 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:38.283+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:38.283959+0000) 2024-04-27T03:57:38.690 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:38.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:38.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:38.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:38.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:38.535+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:38.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:38.536+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:38.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:38.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:38.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:38.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:38.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:38.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:38.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:38.539+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:38.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:38.866+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:38.866959+0000) 2024-04-27T03:57:38.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:38.922+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:38.922987+0000) 2024-04-27T03:57:39.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:39.146+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:39.147762+0000) 2024-04-27T03:57:39.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:39.029+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:39.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:39.029+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:39.367 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:39.029+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:39.367 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:39.029+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:39.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:39.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:39.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:39.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:39.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:39.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:39.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:39.543+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:39.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:39.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:39.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:39.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:39.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:39.549+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:39.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:39.549+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:39.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:39.868+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:39.869210+0000) 2024-04-27T03:57:39.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:39.924+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:39.925260+0000) 2024-04-27T03:57:40.252 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:40.149+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:40.150571+0000) 2024-04-27T03:57:40.252 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:39.987+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:40.252 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:39.987+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:40.252 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:39.987+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:40.253 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:39.988+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:40.872 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:40.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:40.872 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:40.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:40.873 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:40.583+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:40.873 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:40.585+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:40.873 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:40.532+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:40.873 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:40.532+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:40.873 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:40.532+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:40.874 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:40.533+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:41.154 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:40.870+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:40.871508+0000) 2024-04-27T03:57:41.154 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:40.926+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:40.927491+0000) 2024-04-27T03:57:41.154 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:40.992+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:41.155 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:40.992+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:41.155 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:40.992+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:41.155 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:40.994+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:41.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:41.152+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:41.153219+0000) 2024-04-27T03:57:41.874 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:41.613+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:41.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:41.613+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:41.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:41.613+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:41.875 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:41.615+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:41.875 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:41.486+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:41.876 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:41.486+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:41.876 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:41.486+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:41.876 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:41.486+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:42.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:41.872+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:41.873781+0000) 2024-04-27T03:57:42.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:41.928+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:41.929759+0000) 2024-04-27T03:57:42.157 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:41.998+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:42.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:41.998+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:42.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:41.998+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:42.158 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:42.000+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:42.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:42.155+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:42.155972+0000) 2024-04-27T03:57:42.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:42.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:42.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:42.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:42.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:42.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:42.878 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:42.599+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:42.878 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:42.527+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:42.878 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:42.527+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:42.878 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:42.527+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:42.878 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:42.528+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:43.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:42.875+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:42.876571+0000) 2024-04-27T03:57:43.159 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:42.931+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:42.931903+0000) 2024-04-27T03:57:43.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:42.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:43.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:42.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:43.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:42.952+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:43.160 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:42.953+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:43.415 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:57:43.295+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77189 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:57:43.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:43.292+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:43.292967+0000) 2024-04-27T03:57:43.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:43.292+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:43.293071+0000) 2024-04-27T03:57:43.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:43.157+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:43.158393+0000) 2024-04-27T03:57:43.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:43.291+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:43.292665+0000) 2024-04-27T03:57:43.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:43.521+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:43.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:43.521+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:43.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:43.521+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:43.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:43.521+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:43.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:43.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:43.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:43.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:43.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:43.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:43.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:43.626+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:43.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:43.878+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:43.879611+0000) 2024-04-27T03:57:43.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:43.933+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:43.934053+0000) 2024-04-27T03:57:43.953 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:43.943+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:43.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:43.943+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:43.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:43.943+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:43.954 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:43.943+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:44.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:44.160+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:44.161469+0000) 2024-04-27T03:57:44.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:44.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:44.885 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:44.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:44.885 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:44.537+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:44.885 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:44.538+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:44.885 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:44.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:44.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:44.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:44.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:44.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:44.886 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:44.623+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:45.165 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:44.883+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:44.883841+0000) 2024-04-27T03:57:45.165 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:44.935+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:44.936682+0000) 2024-04-27T03:57:45.166 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:44.984+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:45.166 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:44.984+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:45.166 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:44.984+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:45.166 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:44.985+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:45.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:45.163+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:45.164330+0000) 2024-04-27T03:57:45.889 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:45.563+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:45.889 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:45.563+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:45.889 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:45.563+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:45.889 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:45.564+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:45.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:45.649+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:45.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:45.650+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:45.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:45.650+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:45.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:45.650+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:46.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:45.887+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:45.888053+0000) 2024-04-27T03:57:46.168 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:45.939+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:45.940001+0000) 2024-04-27T03:57:46.168 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:45.983+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:46.168 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:45.983+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:46.168 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:45.983+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:46.169 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:45.984+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:46.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:46.165+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:46.166672+0000) 2024-04-27T03:57:46.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:46.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:46.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:46.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:46.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:46.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:46.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:46.620+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:46.892 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:46.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:46.893 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:46.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:46.893 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:46.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:46.893 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:46.579+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:47.170 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:46.890+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:46.890921+0000) 2024-04-27T03:57:47.171 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:46.942+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:46.942928+0000) 2024-04-27T03:57:47.171 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:46.984+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:47.171 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:46.984+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:47.171 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:46.984+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:47.171 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:46.985+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:47.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:47.168+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:47.169415+0000) 2024-04-27T03:57:47.894 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:47.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:47.894 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:47.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:47.894 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:47.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:47.894 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:47.619+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:47.895 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:47.531+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:47.895 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:47.531+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:47.895 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:47.531+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:47.895 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:47.531+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:48.173 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:47.892+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:47.893074+0000) 2024-04-27T03:57:48.173 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:47.944+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:47.945431+0000) 2024-04-27T03:57:48.173 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:47.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:48.173 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:47.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:48.174 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:47.990+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:48.174 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:47.990+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:48.424 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:57:48.304+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77229 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:57:48.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:48.299+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:48.300439+0000) 2024-04-27T03:57:48.425 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:48.301+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:48.301960+0000) 2024-04-27T03:57:48.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:48.170+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:48.171793+0000) 2024-04-27T03:57:48.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:48.301+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:48.302223+0000) 2024-04-27T03:57:48.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:48.635+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:48.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:48.635+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:48.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:48.635+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:48.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:48.636+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:48.706 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:48.547+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:48.706 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:48.547+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:48.706 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:48.547+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:48.706 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:48.548+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:48.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:48.896+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:48.897205+0000) 2024-04-27T03:57:48.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:48.948+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:48.949769+0000) 2024-04-27T03:57:48.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:48.953+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:48.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:48.953+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:48.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:48.953+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:48.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:48.954+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:49.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:49.173+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:49.174257+0000) 2024-04-27T03:57:49.632 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:49.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:49.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:49.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:49.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:49.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:49.633 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:49.499+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:49.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:49.631+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:49.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:49.631+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:49.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:49.631+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:49.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:49.631+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:50.152 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:49.899+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:49.900187+0000) 2024-04-27T03:57:50.153 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:49.933+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:50.153 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:49.933+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:50.153 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:49.933+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:50.153 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:49.934+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:50.153 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:49.951+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:49.952449+0000) 2024-04-27T03:57:50.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:50.175+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:50.176371+0000) 2024-04-27T03:57:50.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:50.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:50.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:50.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:50.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:50.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:50.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:50.609+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:50.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:50.545+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:50.907 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:50.545+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:50.907 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:50.545+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:50.907 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:50.545+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:51.179 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:50.904+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:50.905005+0000) 2024-04-27T03:57:51.179 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:50.956+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:50.956988+0000) 2024-04-27T03:57:51.180 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:50.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:51.180 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:50.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:51.180 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:50.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:51.180 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:50.975+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:51.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:51.177+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:51.178529+0000) 2024-04-27T03:57:51.910 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:51.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:51.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:51.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:51.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:51.518+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:51.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:51.519+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:51.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:51.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:51.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:51.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:51.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:51.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:51.912 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:51.629+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:52.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:51.908+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:51.909692+0000) 2024-04-27T03:57:52.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:51.961+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:51.961831+0000) 2024-04-27T03:57:52.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:52.004+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:52.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:52.004+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:52.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:52.004+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:52.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:52.004+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:52.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:52.180+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:52.181058+0000) 2024-04-27T03:57:52.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:52.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:52.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:52.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:52.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:52.528+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:52.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:52.529+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:52.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:52.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:52.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:52.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:52.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:52.598+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:52.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:52.599+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:53.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:52.913+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:52.914479+0000) 2024-04-27T03:57:53.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:52.965+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:52.966690+0000) 2024-04-27T03:57:53.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:52.975+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:53.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:52.975+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:53.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:52.975+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:53.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:52.976+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:53.437 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:57:53.313+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77269 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:57:53.438 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:53.309+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:53.309899+0000) 2024-04-27T03:57:53.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:53.182+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:53.183243+0000) 2024-04-27T03:57:53.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:53.308+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:53.309401+0000) 2024-04-27T03:57:53.439 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:53.310+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:53.311668+0000) 2024-04-27T03:57:53.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:53.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:53.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:53.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:53.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:53.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:53.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:53.628+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:53.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:53.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:53.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:53.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:53.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:53.539+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:53.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:53.539+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:53.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:53.916+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:53.916925+0000) 2024-04-27T03:57:54.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:54.184+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:54.185503+0000) 2024-04-27T03:57:54.254 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:53.970+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:53.970991+0000) 2024-04-27T03:57:54.254 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:53.989+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:54.254 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:53.989+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:54.254 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:53.989+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:54.254 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:53.991+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:54.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:54.668+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:54.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:54.668+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:54.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:54.668+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:54.923 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:54.669+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:54.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:54.551+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:54.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:54.551+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:54.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:54.551+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:54.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:54.551+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:55.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:54.921+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:54.922037+0000) 2024-04-27T03:57:55.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:54.975+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:54.976115+0000) 2024-04-27T03:57:55.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:55.040+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:55.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:55.040+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:55.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:55.040+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:55.183 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:55.042+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:55.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:55.187+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:55.188611+0000) 2024-04-27T03:57:55.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:55.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:55.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:55.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:55.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:55.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:55.928 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:55.623+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:55.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:55.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:55.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:55.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:55.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:55.575+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:55.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:55.575+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:56.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:55.924+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:55.926651+0000) 2024-04-27T03:57:56.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:55.979+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:55.980986+0000) 2024-04-27T03:57:56.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:56.047+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:56.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:56.047+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:56.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:56.047+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:56.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:56.048+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:56.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:56.189+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:56.190935+0000) 2024-04-27T03:57:56.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:56.665+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:56.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:56.665+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:56.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:56.665+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:56.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:56.666+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:56.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:56.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:56.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:56.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:56.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:56.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:56.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:56.579+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:57.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:56.927+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:56.929710+0000) 2024-04-27T03:57:57.181 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:56.983+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:56.985113+0000) 2024-04-27T03:57:57.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:57.073+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:57.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:57.073+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:57.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:57.073+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:57.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:57.075+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:57.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:57.194+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:57.196114+0000) 2024-04-27T03:57:57.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:57.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:57.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:57.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:57.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:57.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:57.933 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:57.648+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:57.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:57.584+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:57.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:57.584+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:57.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:57.584+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:57.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:57.585+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:58.202 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:57.930+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:57.932368+0000) 2024-04-27T03:57:58.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:57.985+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:57.987278+0000) 2024-04-27T03:57:58.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:58.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:58.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:58.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:58.202 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:58.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:58.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:58.082+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:58.471 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:57:58.323+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77309 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:57:58.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:58.316+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:58.318203+0000) 2024-04-27T03:57:58.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:58.318+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:58.320487+0000) 2024-04-27T03:57:58.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:58.199+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:58.201154+0000) 2024-04-27T03:57:58.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:58.317+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:58.319342+0000) 2024-04-27T03:57:58.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:58.649+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:58.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:58.649+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:58.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:58.649+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:58.936 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:58.650+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:58.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:58.584+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:58.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:58.584+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:58.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:58.584+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:58.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:58.584+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:59.207 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:58.933+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:58.935089+0000) 2024-04-27T03:57:59.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:58.988+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:58.990004+0000) 2024-04-27T03:57:59.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:59.110+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:59.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:59.110+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:59.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:59.110+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:57:59.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:59.111+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:59.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:59.204+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:59.206322+0000) 2024-04-27T03:57:59.938 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:59.648+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:59.938 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:59.648+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:59.939 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:59.648+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:57:59.939 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:59.649+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:57:59.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:59.607+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:59.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:59.607+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:59.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:59.607+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:57:59.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:57:59.607+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:00.209 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:57:59.935+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:59.937406+0000) 2024-04-27T03:58:00.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:57:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:57:59.990+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:57:59.992099+0000) 2024-04-27T03:58:00.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:00.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:00.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:00.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:00.211 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:00.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:00.211 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:00.125+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:00.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:00.206+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:00.208770+0000) 2024-04-27T03:58:00.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:00.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:00.941 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:00.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:00.941 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:00.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:00.941 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:00.647+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:00.941 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:00.572+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:00.941 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:00.572+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:00.942 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:00.572+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:00.942 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:00.573+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:01.213 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:00.938+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:00.939883+0000) 2024-04-27T03:58:01.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:00.994+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:00.996385+0000) 2024-04-27T03:58:01.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:01.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:01.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:01.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:01.214 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:01.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:01.214 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:01.129+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:01.214 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:01.210+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:01.212248+0000) 2024-04-27T03:58:01.856 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:01.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:01.856 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:01.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:01.856 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:01.568+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:01.856 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:01.568+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:01.856 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:01.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:01.857 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:01.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:01.857 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:01.606+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:01.857 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:01.607+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:02.119 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:01.940+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:01.942170+0000) 2024-04-27T03:58:02.120 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:01.999+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:02.001463+0000) 2024-04-27T03:58:02.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:02.213+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:02.215112+0000) 2024-04-27T03:58:02.382 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:02.117+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:02.382 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:02.117+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:02.382 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:02.117+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:02.383 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:02.118+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:02.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:02.571+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:02.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:02.571+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:02.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:02.571+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:02.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:02.572+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:02.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:02.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:02.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:02.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:02.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:02.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:02.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:02.635+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:03.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:02.942+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:02.944364+0000) 2024-04-27T03:58:03.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:03.001+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:03.003624+0000) 2024-04-27T03:58:03.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:03.089+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:03.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:03.089+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:03.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:03.089+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:03.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:03.090+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:03.471 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:58:03.332+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77349 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:58:03.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:03.328+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:03.330669+0000) 2024-04-27T03:58:03.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:03.218+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:03.220261+0000) 2024-04-27T03:58:03.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:03.328+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:03.330526+0000) 2024-04-27T03:58:03.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:03.328+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:03.330647+0000) 2024-04-27T03:58:03.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:03.556+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:03.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:03.556+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:03.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:03.556+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:03.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:03.556+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:03.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:03.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:03.948 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:03.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:03.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:03.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:03.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:03.649+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:04.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:03.944+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:03.946631+0000) 2024-04-27T03:58:04.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:04.004+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:04.006384+0000) 2024-04-27T03:58:04.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:04.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:04.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:04.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:04.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:04.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:04.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:04.130+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:04.609 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:04.220+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:04.222464+0000) 2024-04-27T03:58:04.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:04.674+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:04.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:04.674+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:04.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:04.674+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:04.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:04.675+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:04.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:04.606+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:04.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:04.606+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:04.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:04.606+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:04.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:04.607+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:05.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:04.947+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:04.949103+0000) 2024-04-27T03:58:05.182 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:05.006+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:05.008613+0000) 2024-04-27T03:58:05.183 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:05.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:05.183 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:05.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:05.183 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:05.124+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:05.183 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:05.125+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:05.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:05.222+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:05.224531+0000) 2024-04-27T03:58:05.740 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:05.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:05.741 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:05.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:05.741 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:05.684+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:05.741 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:05.685+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:05.741 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:05.641+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:05.741 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:05.641+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:05.741 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:05.641+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:05.741 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:05.641+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:06.013 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:05.949+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:05.951377+0000) 2024-04-27T03:58:06.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:06.224+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:06.226664+0000) 2024-04-27T03:58:06.369 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:06.011+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:06.012889+0000) 2024-04-27T03:58:06.369 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:06.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:06.369 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:06.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:06.369 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:06.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:06.369 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:06.150+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:06.954 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:06.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:06.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:06.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:06.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:06.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:06.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:06.665+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:06.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:06.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:06.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:06.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:06.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:06.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:06.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:06.719+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:07.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:06.952+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:06.953581+0000) 2024-04-27T03:58:07.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:07.014+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:07.015215+0000) 2024-04-27T03:58:07.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:07.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:07.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:07.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:07.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:07.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:07.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:07.188+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:07.661 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:07.227+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:07.229498+0000) 2024-04-27T03:58:07.957 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:07.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:07.957 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:07.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:07.957 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:07.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:07.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:07.722+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:07.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:07.660+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:07.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:07.660+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:07.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:07.660+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:07.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:07.660+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:08.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:07.955+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:07.956462+0000) 2024-04-27T03:58:08.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:08.016+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:08.017553+0000) 2024-04-27T03:58:08.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:08.167+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:08.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:08.167+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:08.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:08.167+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:08.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:08.167+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:08.681 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:58:08.342+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77389 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:58:08.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:08.339+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:08.339857+0000) 2024-04-27T03:58:08.682 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:08.232+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:08.232860+0000) 2024-04-27T03:58:08.682 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:08.338+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:08.339433+0000) 2024-04-27T03:58:08.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:08.338+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:08.338830+0000) 2024-04-27T03:58:08.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:08.679+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:08.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:08.679+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:08.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:08.679+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:08.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:08.680+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:08.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:08.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:08.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:08.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:08.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:08.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:08.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:08.693+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:09.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:08.958+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:08.959581+0000) 2024-04-27T03:58:09.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:09.019+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:09.020356+0000) 2024-04-27T03:58:09.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:09.163+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:09.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:09.163+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:09.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:09.163+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:09.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:09.163+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:09.640 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:09.235+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:09.236093+0000) 2024-04-27T03:58:09.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:09.639+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:09.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:09.639+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:09.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:09.639+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:09.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:09.640+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:09.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:09.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:09.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:09.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:09.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:09.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:09.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:09.710+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:10.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:09.961+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:09.961886+0000) 2024-04-27T03:58:10.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:10.021+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:10.022473+0000) 2024-04-27T03:58:10.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:10.193+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:10.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:10.193+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:10.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:10.193+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:10.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:10.193+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:10.522 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:10.238+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:10.239135+0000) 2024-04-27T03:58:10.965 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:10.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:10.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:10.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:10.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:10.601+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:10.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:10.602+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:10.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:10.670+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:10.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:10.670+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:10.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:10.670+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:10.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:10.670+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:11.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:10.964+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:10.964840+0000) 2024-04-27T03:58:11.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:11.024+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:11.025482+0000) 2024-04-27T03:58:11.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:11.204+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:11.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:11.204+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:11.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:11.204+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:11.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:11.205+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:11.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:11.240+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:11.241425+0000) 2024-04-27T03:58:11.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:11.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:11.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:11.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:11.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:11.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:11.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:11.710+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:11.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:11.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:11.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:11.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:11.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:11.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:11.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:11.573+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:12.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:11.968+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:11.968845+0000) 2024-04-27T03:58:12.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:12.026+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:12.027745+0000) 2024-04-27T03:58:12.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:12.220+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:12.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:12.220+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:12.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:12.220+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:12.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:12.221+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:12.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:12.244+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:12.245515+0000) 2024-04-27T03:58:12.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:12.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:12.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:12.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:12.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:12.622+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:12.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:12.622+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:12.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:12.731+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:12.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:12.731+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:12.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:12.731+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:12.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:12.731+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:13.243 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:12.971+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:12.972218+0000) 2024-04-27T03:58:13.244 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:13.029+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:13.030723+0000) 2024-04-27T03:58:13.642 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:58:13.351+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77429 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:58:13.642 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:13.347+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:13.348463+0000) 2024-04-27T03:58:13.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:13.248+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:13.248999+0000) 2024-04-27T03:58:13.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:13.347+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:13.348003+0000) 2024-04-27T03:58:13.643 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:13.242+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:13.643 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:13.242+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:13.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:13.242+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:13.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:13.243+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:13.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:13.347+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:13.348687+0000) 2024-04-27T03:58:13.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:13.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:13.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:13.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:13.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:13.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:13.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:13.642+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:13.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:13.687+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:13.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:13.687+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:13.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:13.687+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:13.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:13.688+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:14.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:13.973+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:13.974495+0000) 2024-04-27T03:58:14.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:14.032+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:14.033028+0000) 2024-04-27T03:58:14.678 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:14.250+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:14.251754+0000) 2024-04-27T03:58:14.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:14.255+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:14.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:14.255+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:14.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:14.255+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:14.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:14.256+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:14.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:14.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:14.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:14.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:14.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:14.681+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:14.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:14.682+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:14.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:14.676+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:14.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:14.676+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:14.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:14.676+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:14.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:14.677+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:15.255 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:14.976+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:14.977385+0000) 2024-04-27T03:58:15.255 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:15.035+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:15.036163+0000) 2024-04-27T03:58:15.572 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:15.253+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:15.254536+0000) 2024-04-27T03:58:15.572 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:15.290+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:15.572 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:15.290+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:15.573 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:15.290+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:15.573 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:15.291+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:15.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:15.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:15.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:15.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:15.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:15.722+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:15.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:15.723+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:15.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:15.663+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:15.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:15.663+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:15.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:15.663+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:15.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:15.664+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:16.256 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:15.978+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:15.979547+0000) 2024-04-27T03:58:16.256 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:16.037+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:16.038278+0000) 2024-04-27T03:58:16.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:16.256+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:16.257184+0000) 2024-04-27T03:58:16.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:16.690+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:16.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:16.690+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:16.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:16.690+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:16.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:16.327+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:16.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:16.327+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:16.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:16.327+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:16.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:16.328+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:16.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:16.691+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:16.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:16.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:16.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:16.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:16.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:16.712+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:16.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:16.713+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:17.256 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:16.983+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:16.984186+0000) 2024-04-27T03:58:17.256 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:17.039+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:17.040386+0000) 2024-04-27T03:58:17.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:17.259+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:17.259843+0000) 2024-04-27T03:58:17.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:17.668+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:17.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:17.668+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:17.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:17.668+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:17.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:17.350+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:17.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:17.350+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:17.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:17.350+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:17.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:17.352+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:17.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:17.669+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:17.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:17.680+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:17.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:17.680+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:17.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:17.680+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:17.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:17.681+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:18.256 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:17.985+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:17.986363+0000) 2024-04-27T03:58:18.256 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:18.041+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:18.042539+0000) 2024-04-27T03:58:18.666 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:58:18.361+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77469 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:58:18.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:18.358+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:18.359256+0000) 2024-04-27T03:58:18.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:18.261+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:18.262037+0000) 2024-04-27T03:58:18.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:18.356+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:18.357336+0000) 2024-04-27T03:58:18.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:18.665+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:18.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:18.665+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:18.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:18.665+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:18.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:18.356+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:18.356828+0000) 2024-04-27T03:58:18.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:18.391+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:18.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:18.391+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:18.669 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:18.391+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:18.669 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:18.391+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:18.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:18.665+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:18.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:18.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:18.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:18.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:18.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:18.711+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:18.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:18.712+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:19.256 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:18.987+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:18.988678+0000) 2024-04-27T03:58:19.256 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:19.046+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:19.047339+0000) 2024-04-27T03:58:19.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:19.263+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:19.264281+0000) 2024-04-27T03:58:19.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:19.647+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:19.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:19.647+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:19.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:19.647+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:19.650 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:19.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:19.650 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:19.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:19.650 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:19.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:19.650 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:19.369+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:19.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:19.746+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:19.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:19.746+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:19.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:19.746+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:19.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:19.747+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:19.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:19.648+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:20.256 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:19.989+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:19.990777+0000) 2024-04-27T03:58:20.257 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:20.048+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:20.049570+0000) 2024-04-27T03:58:20.632 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:20.266+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:20.266890+0000) 2024-04-27T03:58:20.651 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:20.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:20.651 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:20.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:20.651 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:20.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:20.651 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:20.359+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:20.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:20.790+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:20.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:20.790+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:20.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:20.790+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:20.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:20.791+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:20.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:20.672+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:20.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:20.672+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:20.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:20.672+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:20.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:20.672+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:21.256 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:20.994+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:20.995061+0000) 2024-04-27T03:58:21.257 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:21.051+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:21.051838+0000) 2024-04-27T03:58:21.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:21.333+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:21.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:21.333+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:21.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:21.333+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:21.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:21.334+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:21.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:21.269+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:21.270091+0000) 2024-04-27T03:58:21.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:21.695+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:21.697 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:21.695+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:21.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:21.695+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:21.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:21.695+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:21.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:21.831+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:21.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:21.831+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:21.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:21.831+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:21.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:21.832+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:22.256 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:21.997+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:21.998218+0000) 2024-04-27T03:58:22.257 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:22.055+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:22.056265+0000) 2024-04-27T03:58:22.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:22.272+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:22.273092+0000) 2024-04-27T03:58:22.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:22.689+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:22.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:22.357+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:22.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:22.357+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:22.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:22.357+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:22.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:22.358+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:22.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:22.689+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:22.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:22.689+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:22.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:22.690+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:22.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:22.880+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:22.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:22.880+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:22.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:22.880+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:22.973 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:22.881+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:23.257 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:22.999+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:23.000387+0000) 2024-04-27T03:58:23.257 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:23.058+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:23.058970+0000) 2024-04-27T03:58:23.650 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:58:23.369+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77509 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:58:23.651 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:23.365+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:23.366547+0000) 2024-04-27T03:58:23.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:23.276+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:23.277058+0000) 2024-04-27T03:58:23.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:23.366+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:23.367091+0000) 2024-04-27T03:58:23.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:23.648+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:23.652 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:23.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:23.652 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:23.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:23.652 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:23.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:23.652 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:23.353+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:23.652 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:23.365+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:23.366332+0000) 2024-04-27T03:58:23.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:23.648+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:23.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:23.648+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:23.921 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:23.649+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:24.184 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:24.061+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:24.062159+0000) 2024-04-27T03:58:24.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:23.919+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:24.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:23.919+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:24.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:23.919+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:24.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:23.920+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:24.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:24.002+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:24.003656+0000) 2024-04-27T03:58:24.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:24.279+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:24.280178+0000) 2024-04-27T03:58:24.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:24.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:24.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:24.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:24.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:24.369+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:24.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:24.370+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:24.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:24.876+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:24.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:24.876+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:24.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:24.876+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:24.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:24.877+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:24.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:24.684+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:24.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:24.684+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:24.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:24.684+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:24.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:24.685+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:25.257 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:25.005+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:25.006388+0000) 2024-04-27T03:58:25.257 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:25.065+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:25.066686+0000) 2024-04-27T03:58:25.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:25.283+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:25.284028+0000) 2024-04-27T03:58:25.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:25.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:25.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:25.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:25.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:25.365+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:25.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:25.365+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:25.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:25.856+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:25.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:25.856+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:25.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:25.856+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:25.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:25.857+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:25.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:25.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:25.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:25.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:25.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:25.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:25.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:25.735+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:26.287 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:26.008+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:26.009104+0000) 2024-04-27T03:58:26.287 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:26.068+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:26.069449+0000) 2024-04-27T03:58:26.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:26.285+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:26.286689+0000) 2024-04-27T03:58:26.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:26.322+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:26.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:26.322+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:26.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:26.322+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:26.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:26.322+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:27.012 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:26.843+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:27.012 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:26.843+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:27.012 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:26.843+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:27.012 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:26.843+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:27.013 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:26.726+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:27.013 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:26.726+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:27.013 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:26.726+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:27.013 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:26.726+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:27.291 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:27.010+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:27.011252+0000) 2024-04-27T03:58:27.291 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:27.071+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:27.072272+0000) 2024-04-27T03:58:27.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:27.289+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:27.290724+0000) 2024-04-27T03:58:27.702 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:27.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:27.702 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:27.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:27.702 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:27.344+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:27.702 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:27.345+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:27.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:27.799+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:27.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:27.799+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:27.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:27.799+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:27.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:27.799+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:27.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:27.700+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:27.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:27.700+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:27.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:27.700+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:27.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:27.700+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:28.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:28.013+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:28.014195+0000) 2024-04-27T03:58:28.295 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:28.073+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:28.074465+0000) 2024-04-27T03:58:28.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:58:28.379+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77549 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:58:28.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:28.374+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:28.375297+0000) 2024-04-27T03:58:28.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:28.293+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:28.294496+0000) 2024-04-27T03:58:28.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:28.374+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:28.375527+0000) 2024-04-27T03:58:28.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:28.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:28.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:28.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:28.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:28.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:28.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:28.298+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:28.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:28.374+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:28.375517+0000) 2024-04-27T03:58:29.018 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:28.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:29.018 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:28.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:29.018 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:28.838+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:29.018 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:28.839+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:29.019 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:28.720+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:29.019 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:28.720+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:29.019 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:28.720+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:29.019 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:28.721+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:29.297 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:29.016+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:29.017047+0000) 2024-04-27T03:58:29.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:29.078+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:29.079102+0000) 2024-04-27T03:58:29.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:29.296+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:29.297147+0000) 2024-04-27T03:58:29.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:29.327+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:29.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:29.327+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:29.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:29.327+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:29.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:29.328+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:30.020 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:29.856+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:30.020 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:29.856+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:30.020 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:29.856+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:30.020 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:29.856+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:30.021 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:29.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:30.021 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:29.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:30.021 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:29.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:30.021 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:29.725+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:30.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:30.018+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:30.019436+0000) 2024-04-27T03:58:30.300 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:30.083+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:30.084152+0000) 2024-04-27T03:58:30.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:30.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:30.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:30.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:30.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:30.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:30.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:30.330+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:30.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:30.298+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:30.299516+0000) 2024-04-27T03:58:30.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:30.688+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:30.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:30.688+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:30.691 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:30.688+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:30.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:30.689+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:30.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:30.847+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:30.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:30.847+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:30.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:30.847+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:30.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:30.848+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:31.288 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:31.020+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:31.021505+0000) 2024-04-27T03:58:31.288 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:31.088+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:31.089158+0000) 2024-04-27T03:58:31.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:31.286+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:31.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:31.286+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:31.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:31.286+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:31.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:31.287+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:31.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:31.301+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:31.302373+0000) 2024-04-27T03:58:31.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:31.681+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:31.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:31.681+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:31.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:31.681+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:31.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:31.682+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:31.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:31.807+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:31.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:31.807+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:31.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:31.807+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:31.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:31.808+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:32.305 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:32.093+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:32.094310+0000) 2024-04-27T03:58:32.305 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:32.022+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:32.023660+0000) 2024-04-27T03:58:32.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:32.303+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:32.304632+0000) 2024-04-27T03:58:32.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:32.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:32.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:32.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:32.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:32.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:32.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:32.331+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:32.859 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:32.663+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:32.860 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:32.663+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:32.860 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:32.663+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:32.860 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:32.664+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:33.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:33.098+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:33.099463+0000) 2024-04-27T03:58:33.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:32.858+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:33.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:32.858+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:33.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:32.858+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:33.186 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:32.859+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:33.186 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:33.025+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:33.026442+0000) 2024-04-27T03:58:33.471 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:58:33.389+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77589 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:58:33.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:33.384+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:33.384979+0000) 2024-04-27T03:58:33.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:33.307+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:33.307825+0000) 2024-04-27T03:58:33.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:33.384+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:33.385106+0000) 2024-04-27T03:58:33.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:33.350+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:33.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:33.350+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:33.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:33.350+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:33.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:33.351+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:33.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:33.384+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:33.385598+0000) 2024-04-27T03:58:33.786 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:33.690+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:33.786 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:33.690+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:33.786 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:33.690+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:33.787 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:33.691+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:34.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:33.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:34.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:33.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:34.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:33.893+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:34.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:33.894+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:34.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:34.028+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:34.029386+0000) 2024-04-27T03:58:34.372 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:34.311+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:34.312072+0000) 2024-04-27T03:58:34.372 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:34.103+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:34.104619+0000) 2024-04-27T03:58:34.372 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:34.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:34.372 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:34.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:34.372 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:34.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:34.373 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:34.331+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:35.033 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:34.912+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:35.033 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:34.912+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:35.033 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:34.912+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:35.033 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:34.913+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:35.034 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:34.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:35.034 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:34.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:35.034 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:34.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:35.034 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:34.734+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:35.315 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:35.031+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:35.032257+0000) 2024-04-27T03:58:35.316 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:35.108+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:35.109292+0000) 2024-04-27T03:58:35.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:35.313+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:35.314697+0000) 2024-04-27T03:58:35.647 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:35.372+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:35.647 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:35.372+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:35.647 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:35.372+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:35.647 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:35.373+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:35.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:35.698+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:35.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:35.698+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:35.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:35.698+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:35.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:35.699+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:35.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:35.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:35.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:35.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:35.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:35.885+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:35.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:35.887+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:36.317 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:36.034+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:36.035097+0000) 2024-04-27T03:58:36.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:36.110+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:36.111360+0000) 2024-04-27T03:58:36.666 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:36.316+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:36.316909+0000) 2024-04-27T03:58:36.666 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:36.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:36.666 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:36.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:36.666 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:36.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:36.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:36.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:36.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:36.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:36.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:36.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:36.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:36.409+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:36.922 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:36.665+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:37.186 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:37.112+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:37.113487+0000) 2024-04-27T03:58:37.186 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:36.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:37.186 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:36.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:37.186 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:36.921+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:37.186 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:36.921+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:37.187 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:37.037+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:37.038160+0000) 2024-04-27T03:58:37.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:37.318+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:37.319231+0000) 2024-04-27T03:58:37.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:37.378+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:37.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:37.378+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:37.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:37.378+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:37.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:37.378+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:37.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:37.888+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:37.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:37.888+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:37.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:37.888+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:37.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:37.889+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:37.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:37.679+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:37.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:37.679+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:37.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:37.679+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:37.973 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:37.679+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:38.322 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:38.039+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:38.040481+0000) 2024-04-27T03:58:38.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:38.117+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:38.117816+0000) 2024-04-27T03:58:38.686 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:58:38.397+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77629 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:58:38.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:38.394+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:38.395453+0000) 2024-04-27T03:58:38.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:38.320+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:38.321400+0000) 2024-04-27T03:58:38.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:38.394+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:38.394846+0000) 2024-04-27T03:58:38.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:38.684+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:38.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:38.684+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:38.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:38.684+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:38.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:38.394+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:38.395342+0000) 2024-04-27T03:58:38.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:38.400+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:38.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:38.400+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:38.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:38.400+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:38.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:38.401+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:38.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:38.685+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:38.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:38.909+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:38.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:38.909+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:38.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:38.909+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:38.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:38.911+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:39.324 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:39.042+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:39.042848+0000) 2024-04-27T03:58:39.324 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:39.121+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:39.122196+0000) 2024-04-27T03:58:39.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:39.322+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:39.323741+0000) 2024-04-27T03:58:39.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:39.402+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:39.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:39.402+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:39.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:39.402+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:39.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:39.403+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:40.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:39.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:40.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:39.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:40.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:39.734+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:40.046 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:39.734+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:40.047 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:39.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:40.047 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:39.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:40.047 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:39.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:40.047 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:39.931+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:40.327 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:40.044+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:40.045706+0000) 2024-04-27T03:58:40.327 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:40.124+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:40.124846+0000) 2024-04-27T03:58:40.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:40.325+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:40.326724+0000) 2024-04-27T03:58:40.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:40.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:40.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:40.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:40.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:40.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:40.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:40.364+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:40.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:40.692+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:40.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:40.692+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:40.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:40.692+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:40.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:40.693+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:41.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:41.127+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:41.128076+0000) 2024-04-27T03:58:41.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:40.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:41.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:40.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:41.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:40.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:41.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:40.946+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:41.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:41.049+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:41.050383+0000) 2024-04-27T03:58:41.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:41.329+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:41.330354+0000) 2024-04-27T03:58:41.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:41.378+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:41.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:41.378+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:41.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:41.378+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:41.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:41.379+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:41.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:41.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:41.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:41.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:41.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:41.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:41.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:41.713+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:41.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:41.916+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:41.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:41.916+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:41.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:41.916+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:41.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:41.917+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:42.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:42.053+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:42.054643+0000) 2024-04-27T03:58:42.333 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:42.130+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:42.130842+0000) 2024-04-27T03:58:42.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:42.331+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:42.332490+0000) 2024-04-27T03:58:42.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:42.386+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:42.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:42.386+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:42.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:42.386+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:42.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:42.387+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:43.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:42.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:43.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:42.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:43.058 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:42.945+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:43.058 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:42.947+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:43.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:42.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:43.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:42.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:43.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:42.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:43.058 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:42.725+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:43.335 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:43.056+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:43.056843+0000) 2024-04-27T03:58:43.335 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:43.134+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:43.135219+0000) 2024-04-27T03:58:43.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:58:43.406+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77670 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:58:43.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:43.403+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:43.404348+0000) 2024-04-27T03:58:43.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:43.334+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:43.334843+0000) 2024-04-27T03:58:43.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:43.402+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:43.403622+0000) 2024-04-27T03:58:43.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:43.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:43.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:43.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:43.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:43.366+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:43.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:43.367+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:43.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:43.402+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:43.402943+0000) 2024-04-27T03:58:44.061 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:43.739+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:44.062 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:43.739+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:44.062 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:43.739+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:44.062 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:43.740+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:44.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:43.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:44.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:43.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:44.062 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:43.947+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:44.063 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:43.948+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:44.063 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:44.060+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:44.061058+0000) 2024-04-27T03:58:44.337 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:44.136+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:44.137354+0000) 2024-04-27T03:58:44.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:44.336+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:44.337133+0000) 2024-04-27T03:58:44.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:44.414+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:44.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:44.414+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:44.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:44.414+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:44.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:44.415+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:45.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:44.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:45.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:44.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:45.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:44.972+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:45.064 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:44.972+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:45.064 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:44.765+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:45.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:44.765+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:45.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:44.765+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:45.065 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:44.765+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:45.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:45.062+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:45.063355+0000) 2024-04-27T03:58:45.342 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:45.138+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:45.139381+0000) 2024-04-27T03:58:45.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:45.340+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:45.341616+0000) 2024-04-27T03:58:45.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:45.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:45.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:45.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:45.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:45.445+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:45.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:45.446+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:46.066 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:45.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:46.066 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:45.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:46.066 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:45.931+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:46.067 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:45.932+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:46.067 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:45.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:46.067 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:45.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:46.067 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:45.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:46.067 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:45.786+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:46.347 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:46.064+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:46.065608+0000) 2024-04-27T03:58:46.347 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:46.141+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:46.142173+0000) 2024-04-27T03:58:46.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:46.345+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:46.346303+0000) 2024-04-27T03:58:46.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:46.427+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:46.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:46.427+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:46.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:46.427+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:46.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:46.427+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:47.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:46.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:47.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:46.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:47.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:46.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:47.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:46.931+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:46.780+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:46.780+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:46.780+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:47.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:46.780+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:47.349 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:47.068+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:47.068934+0000) 2024-04-27T03:58:47.350 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:47.143+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:47.144297+0000) 2024-04-27T03:58:47.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:47.348+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:47.349346+0000) 2024-04-27T03:58:47.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:47.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:47.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:47.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:47.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:47.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:47.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:47.380+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:48.073 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:47.928+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:48.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:47.928+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:48.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:47.928+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:48.074 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:47.929+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:48.074 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:47.796+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:48.074 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:47.796+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:48.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:47.796+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:48.075 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:47.796+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:48.354 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:48.072+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:48.073195+0000) 2024-04-27T03:58:48.355 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:48.145+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:48.146572+0000) 2024-04-27T03:58:48.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:58:48.415+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77710 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:58:48.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:48.412+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:48.412825+0000) 2024-04-27T03:58:48.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:48.353+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:48.354024+0000) 2024-04-27T03:58:48.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:48.411+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:48.412271+0000) 2024-04-27T03:58:48.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:48.410+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:48.411537+0000) 2024-04-27T03:58:48.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:48.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:48.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:48.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:48.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:48.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:48.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:48.422+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:49.077 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:48.936+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:49.077 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:48.936+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:49.077 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:48.936+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:49.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:48.936+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:49.078 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:48.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:49.078 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:48.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:49.078 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:48.846+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:49.078 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:48.847+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:49.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:49.075+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:49.076210+0000) 2024-04-27T03:58:49.357 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:49.150+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:49.151233+0000) 2024-04-27T03:58:49.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:49.355+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:49.356431+0000) 2024-04-27T03:58:49.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:49.414+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:49.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:49.414+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:49.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:49.414+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:49.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:49.415+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:50.079 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:49.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:50.079 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:49.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:50.079 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:49.920+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:50.080 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:49.921+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:50.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:49.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:50.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:49.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:50.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:49.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:50.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:49.830+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:50.359 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:50.077+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:50.078673+0000) 2024-04-27T03:58:50.360 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:50.152+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:50.153475+0000) 2024-04-27T03:58:50.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:50.358+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:50.358821+0000) 2024-04-27T03:58:50.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:50.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:50.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:50.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:50.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:50.450+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:50.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:50.450+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:51.082 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:50.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:51.082 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:50.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:51.082 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:50.852+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:51.083 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:50.852+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:51.083 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:50.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:51.083 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:50.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:51.084 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:50.938+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:51.084 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:50.939+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:51.362 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:51.080+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:51.081183+0000) 2024-04-27T03:58:51.363 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:51.154+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:51.155751+0000) 2024-04-27T03:58:51.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:51.361+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:51.361935+0000) 2024-04-27T03:58:51.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:51.404+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:51.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:51.404+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:51.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:51.404+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:51.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:51.405+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:52.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:51.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:52.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:51.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:52.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:51.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:52.085 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:51.892+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:52.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:51.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:52.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:51.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:52.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:51.984+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:52.085 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:51.984+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:52.366 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:52.083+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:52.083894+0000) 2024-04-27T03:58:52.367 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:52.159+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:52.160091+0000) 2024-04-27T03:58:52.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:52.365+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:52.366052+0000) 2024-04-27T03:58:52.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:52.396+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:52.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:52.396+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:52.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:52.396+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:52.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:52.396+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:53.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:52.979+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:53.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:52.980+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:53.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:52.980+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:53.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:52.980+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:53.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:52.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:53.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:52.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:53.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:52.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:53.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:52.871+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:53.369 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:53.086+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:53.087358+0000) 2024-04-27T03:58:53.369 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:53.163+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:53.164434+0000) 2024-04-27T03:58:53.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:58:53.423+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77750 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:58:53.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:53.420+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:53.421299+0000) 2024-04-27T03:58:53.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:53.367+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:53.368408+0000) 2024-04-27T03:58:53.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:53.419+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:53.420784+0000) 2024-04-27T03:58:53.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:53.368+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:53.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:53.368+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:53.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:53.368+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:53.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:53.369+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:53.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:53.420+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:53.420941+0000) 2024-04-27T03:58:54.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:53.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:54.079 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:53.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:54.079 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:53.992+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:54.079 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:53.993+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:54.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:53.830+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:54.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:53.830+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:54.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:53.830+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:54.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:53.830+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:54.341 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:54.088+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:54.089476+0000) 2024-04-27T03:58:54.341 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:54.165+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:54.166685+0000) 2024-04-27T03:58:54.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:54.370+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:54.370936+0000) 2024-04-27T03:58:54.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:54.339+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:54.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:54.339+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:54.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:54.339+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:54.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:54.340+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:55.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:54.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:55.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:54.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:55.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:54.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:55.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:54.865+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:55.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:55.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:55.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:55.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:55.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:55.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:55.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:55.027+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:55.370 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:55.090+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:55.091619+0000) 2024-04-27T03:58:55.370 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:55.167+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:55.168747+0000) 2024-04-27T03:58:55.370 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:55.368+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:55.371 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:55.368+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:55.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:55.373+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:55.374163+0000) 2024-04-27T03:58:55.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:55.368+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:55.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:55.370+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:56.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:55.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:56.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:55.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:56.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:55.874+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:56.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:55.875+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:56.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:56.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:56.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:56.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:56.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:56.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:56.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:56.055+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:56.342 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:56.095+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:56.096201+0000) 2024-04-27T03:58:56.342 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:56.172+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:56.173464+0000) 2024-04-27T03:58:56.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:56.341+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:56.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:56.341+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:56.343 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:56.341+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:56.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:56.341+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:56.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:56.375+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:56.376530+0000) 2024-04-27T03:58:57.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:56.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:57.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:56.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:57.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:56.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:57.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:56.865+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:57.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:57.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:57.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:57.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:57.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:57.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:57.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:57.007+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:57.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:57.099+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:57.100971+0000) 2024-04-27T03:58:57.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:57.174+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:57.176311+0000) 2024-04-27T03:58:57.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:57.378+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:57.380776+0000) 2024-04-27T03:58:57.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:57.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:57.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:57.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:57.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:57.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:57.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:57.373+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:58.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:58.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:58.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:58.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:58.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:58.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:58.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:58.055+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:58.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:57.879+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:58.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:57.879+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:58.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:57.879+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:58.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:57.880+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:58.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:58.102+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:58.104151+0000) 2024-04-27T03:58:58.374 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:58.177+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:58.179154+0000) 2024-04-27T03:58:58.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:58.428+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:58.430707+0000) 2024-04-27T03:58:58.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:58:58.433+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77790 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:58:58.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:58.383+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:58.385582+0000) 2024-04-27T03:58:58.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:58.429+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:58.431231+0000) 2024-04-27T03:58:58.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:58.385+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:58.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:58.385+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:58.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:58.385+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:58.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:58.386+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:58.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:58.427+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:58.429302+0000) 2024-04-27T03:58:59.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:59.071+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:59.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:59.071+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:59.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:59.071+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:58:59.076 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:59.073+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:59.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:58.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:59.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:58.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:59.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:58.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:58:59.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:58.890+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:58:59.374 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:58:59.106+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:59.108418+0000) 2024-04-27T03:58:59.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:59.179+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:59.181415+0000) 2024-04-27T03:58:59.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:59.387+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:58:59.388950+0000) 2024-04-27T03:58:59.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:59.378+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:59.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:59.378+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:59.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:59.378+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:58:59.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:58:59.378+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:00.087 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:59.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:00.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:59.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:00.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:59.848+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:00.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:58:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:58:59.848+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:00.360 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:00.184+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:00.186446+0000) 2024-04-27T03:59:00.361 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:00.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:00.361 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:00.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:00.361 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:00.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:00.361 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:00.092+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:00.361 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:00.110+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:00.112585+0000) 2024-04-27T03:59:00.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:00.389+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:00.391107+0000) 2024-04-27T03:59:00.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:00.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:00.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:00.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:00.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:00.358+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:00.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:00.359+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:01.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:00.828+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:01.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:00.828+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:01.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:00.828+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:01.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:00.828+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:01.365 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:01.113+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:01.114921+0000) 2024-04-27T03:59:01.366 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:01.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:01.366 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:01.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:01.366 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:01.123+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:01.366 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:01.125+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:01.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:01.189+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:01.191520+0000) 2024-04-27T03:59:01.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:01.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:01.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:01.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:01.367 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:01.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:01.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:01.364+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:01.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:01.393+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:01.395333+0000) 2024-04-27T03:59:02.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:01.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:02.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:01.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:02.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:01.829+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:02.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:01.830+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:02.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:02.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:02.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:02.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:02.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:02.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:02.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:02.091+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:02.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:02.115+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:02.117022+0000) 2024-04-27T03:59:02.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:02.194+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:02.196676+0000) 2024-04-27T03:59:02.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:02.319+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:02.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:02.319+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:02.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:02.319+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:02.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:02.320+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:02.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:02.395+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:02.397631+0000) 2024-04-27T03:59:03.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:02.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:03.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:02.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:03.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:02.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:03.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:02.835+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:03.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:03.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:03.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:03.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:03.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:03.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:03.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:03.087+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:03.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:03.119+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:03.121238+0000) 2024-04-27T03:59:03.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:03.199+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:03.201712+0000) 2024-04-27T03:59:03.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:03.271+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:03.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:03.271+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:03.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:03.271+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:03.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:03.272+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:03.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:59:03.442+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77829 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:59:03.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:03.438+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:03.440306+0000) 2024-04-27T03:59:03.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:03.438+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:03.440798+0000) 2024-04-27T03:59:03.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:03.399+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:03.400827+0000) 2024-04-27T03:59:03.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:03.439+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:03.441384+0000) 2024-04-27T03:59:04.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:03.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:04.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:03.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:04.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:03.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:04.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:03.841+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:04.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:04.114+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:04.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:04.114+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:04.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:04.114+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:04.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:04.116+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:04.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:04.121+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:04.123397+0000) 2024-04-27T03:59:04.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:04.205+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:04.206843+0000) 2024-04-27T03:59:04.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:04.321+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:04.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:04.322+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:04.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:04.322+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:04.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:04.322+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:04.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:04.403+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:04.405080+0000) 2024-04-27T03:59:05.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:04.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:05.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:04.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:05.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:04.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:05.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:04.842+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:05.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:05.207+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:05.208971+0000) 2024-04-27T03:59:05.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:05.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:05.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:05.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:05.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:05.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:05.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:05.274+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:05.376 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:05.106+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:05.376 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:05.106+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:05.376 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:05.106+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:05.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:05.107+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:05.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:05.125+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:05.127603+0000) 2024-04-27T03:59:05.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:05.405+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:05.407228+0000) 2024-04-27T03:59:06.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:05.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:06.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:05.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:06.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:05.883+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:06.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:05.884+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:06.366 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:06.210+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:06.212000+0000) 2024-04-27T03:59:06.367 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:06.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:06.367 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:06.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:06.367 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:06.258+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:06.367 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:06.259+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:06.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:06.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:06.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:06.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:06.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:06.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:06.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:06.109+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:06.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:06.127+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:06.129723+0000) 2024-04-27T03:59:06.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:06.410+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:06.411883+0000) 2024-04-27T03:59:07.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:06.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:07.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:06.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:07.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:06.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:07.089 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:06.835+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:07.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:07.079+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:07.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:07.079+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:07.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:07.079+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:07.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:07.080+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:07.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:07.130+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:07.131845+0000) 2024-04-27T03:59:07.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:07.213+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:07.215716+0000) 2024-04-27T03:59:07.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:07.298+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:07.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:07.298+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:07.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:07.298+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:07.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:07.299+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:07.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:07.412+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:07.414025+0000) 2024-04-27T03:59:07.843 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:07.839+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:07.843 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:07.839+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:07.843 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:07.839+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:07.844 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:07.840+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:08.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:08.074+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:08.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:08.074+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:08.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:08.074+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:08.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:08.074+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:08.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:08.133+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:08.133989+0000) 2024-04-27T03:59:08.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:08.216+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:08.217841+0000) 2024-04-27T03:59:08.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:08.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:08.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:08.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:08.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:08.264+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:08.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:08.264+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:08.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:08.445+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:08.447559+0000) 2024-04-27T03:59:08.722 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:59:08.451+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77869 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:59:08.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:08.445+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:08.447756+0000) 2024-04-27T03:59:08.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:08.416+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:08.417216+0000) 2024-04-27T03:59:08.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:08.447+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:08.449503+0000) 2024-04-27T03:59:09.088 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:09.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:09.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:09.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:09.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:09.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:09.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:09.046+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:09.089 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:08.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:09.089 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:08.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:09.089 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:08.864+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:09.090 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:08.864+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:09.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:09.135+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:09.136559+0000) 2024-04-27T03:59:09.375 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:09.220+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:09.221075+0000) 2024-04-27T03:59:09.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:09.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:09.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:09.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:09.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:09.261+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:09.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:09.262+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:09.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:09.418+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:09.419731+0000) 2024-04-27T03:59:10.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:09.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:10.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:09.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:10.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:09.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:10.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:09.870+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:10.140 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:10.025+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:10.140 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:10.025+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:10.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:10.025+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:10.141 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:10.025+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:10.424 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:10.138+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:10.139352+0000) 2024-04-27T03:59:10.425 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:10.222+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:10.223534+0000) 2024-04-27T03:59:10.425 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:10.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:10.425 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:10.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:10.425 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:10.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:10.425 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:10.271+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:10.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:10.423+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:10.424200+0000) 2024-04-27T03:59:11.089 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:10.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:11.089 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:10.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:11.089 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:10.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:11.089 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:10.828+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:11.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:11.051+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:11.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:11.051+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:11.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:11.051+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:11.090 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:11.052+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:11.375 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:11.141+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:11.142637+0000) 2024-04-27T03:59:11.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:11.224+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:11.225680+0000) 2024-04-27T03:59:11.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:11.278+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:11.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:11.278+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:11.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:11.278+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:11.376 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:11.280+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:11.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:11.426+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:11.427248+0000) 2024-04-27T03:59:12.146 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:12.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:12.146 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:12.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:12.146 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:12.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:12.147 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:12.046+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:12.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:11.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:12.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:11.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:12.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:11.869+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:12.147 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:11.870+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:12.430 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:12.144+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:12.145622+0000) 2024-04-27T03:59:12.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:12.229+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:12.230139+0000) 2024-04-27T03:59:12.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:12.298+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:12.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:12.298+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:12.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:12.298+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:12.431 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:12.298+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:12.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:12.429+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:12.430219+0000) 2024-04-27T03:59:13.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:13.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:13.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:13.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:13.150 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:13.076+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:13.151 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:13.077+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:13.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:12.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:13.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:12.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:13.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:12.866+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:13.151 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:12.866+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:13.433 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:13.149+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:13.149847+0000) 2024-04-27T03:59:13.433 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:13.231+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:13.232408+0000) 2024-04-27T03:59:13.433 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:13.315+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:13.433 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:13.315+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:13.434 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:13.315+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:13.434 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:13.316+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:13.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:59:13.462+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77909 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:59:13.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:13.457+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:13.458450+0000) 2024-04-27T03:59:13.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:13.457+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:13.457824+0000) 2024-04-27T03:59:13.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:13.431+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:13.432619+0000) 2024-04-27T03:59:13.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:13.457+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:13.458416+0000) 2024-04-27T03:59:14.153 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:14.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:14.153 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:14.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:14.153 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:14.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:14.153 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:14.054+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:14.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:13.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:14.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:13.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:14.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:13.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:14.154 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:13.892+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:14.436 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:14.151+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:14.152626+0000) 2024-04-27T03:59:14.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:14.233+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:14.234675+0000) 2024-04-27T03:59:14.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:14.359+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:14.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:14.359+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:14.436 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:14.359+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:14.437 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:14.359+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:14.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:14.434+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:14.435611+0000) 2024-04-27T03:59:15.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:15.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:15.157 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:15.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:15.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:15.016+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:15.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:15.017+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:15.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:14.933+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:15.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:14.933+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:15.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:14.933+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:15.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:14.933+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:15.440 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:15.156+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:15.156826+0000) 2024-04-27T03:59:15.440 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:15.236+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:15.237311+0000) 2024-04-27T03:59:15.440 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:15.385+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:15.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:15.385+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:15.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:15.385+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:15.441 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:15.386+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:15.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:15.439+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:15.439840+0000) 2024-04-27T03:59:16.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:16.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:16.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:16.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:16.160 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:16.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:16.160 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:16.008+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:16.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:15.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:16.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:15.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:16.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:15.953+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:16.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:15.953+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:16.442 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:16.158+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:16.159076+0000) 2024-04-27T03:59:16.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:16.239+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:16.240482+0000) 2024-04-27T03:59:16.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:16.372+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:16.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:16.372+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:16.443 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:16.372+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:16.444 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:16.373+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:16.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:16.441+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:16.442568+0000) 2024-04-27T03:59:17.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:16.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:17.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:16.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:17.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:16.936+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:17.162 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:16.937+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:17.163 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:17.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:17.163 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:17.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:17.163 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:17.050+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:17.163 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:17.051+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:17.445 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:17.160+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:17.161669+0000) 2024-04-27T03:59:17.445 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:17.244+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:17.245083+0000) 2024-04-27T03:59:17.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:17.351+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:17.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:17.351+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:17.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:17.351+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:17.446 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:17.352+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:17.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:17.444+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:17.445155+0000) 2024-04-27T03:59:18.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:18.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:18.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:18.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:18.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:18.022+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:18.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:18.023+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:18.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:17.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:18.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:17.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:18.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:17.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:18.168 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:17.961+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:18.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:18.165+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:18.166180+0000) 2024-04-27T03:59:18.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:18.248+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:18.249407+0000) 2024-04-27T03:59:18.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:18.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:18.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:18.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:18.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:18.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:18.449 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:18.396+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:18.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:59:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:59:18.471+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77949 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:59:18.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:18.467+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:18.467893+0000) 2024-04-27T03:59:18.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:18.467+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:18.468370+0000) 2024-04-27T03:59:18.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:18.446+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:18.447538+0000) 2024-04-27T03:59:18.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:18.466+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:18.467388+0000) 2024-04-27T03:59:19.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:19.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:19.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:19.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:19.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:19.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:19.171 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:19.054+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:19.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:18.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:19.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:18.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:19.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:18.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:19.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:18.991+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:19.450 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:19.170+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:19.170930+0000) 2024-04-27T03:59:19.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:19.252+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:19.253719+0000) 2024-04-27T03:59:19.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:19.390+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:19.450 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:19.390+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:19.451 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:19.390+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:19.451 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:19.391+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:19.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:19.449+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:19.449842+0000) 2024-04-27T03:59:20.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:20.042+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:20.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:20.042+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:20.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:20.042+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:20.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:20.043+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:20.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:19.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:20.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:19.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:20.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:19.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:20.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:19.969+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:20.454 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:20.172+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:20.173616+0000) 2024-04-27T03:59:20.454 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:20.257+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:20.258413+0000) 2024-04-27T03:59:20.454 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:20.368+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:20.455 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:20.368+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:20.455 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:20.368+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:20.455 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:20.368+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:20.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:20.453+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:20.453825+0000) 2024-04-27T03:59:21.176 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:21.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:21.176 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:21.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:21.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:21.053+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:21.177 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:21.054+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:21.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:20.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:21.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:20.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:21.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:20.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:21.178 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:20.944+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:21.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:21.175+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:21.176085+0000) 2024-04-27T03:59:21.459 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:21.261+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:21.262716+0000) 2024-04-27T03:59:21.459 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:21.359+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:21.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:21.359+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:21.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:21.359+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:21.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:21.360+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:21.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:21.457+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:21.458460+0000) 2024-04-27T03:59:22.027 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:21.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:22.028 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:21.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:22.028 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:21.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:22.028 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:21.960+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:22.332 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:22.266+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:22.267018+0000) 2024-04-27T03:59:22.332 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:22.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:22.332 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:22.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:22.332 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:22.026+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:22.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:22.027+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:22.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:22.179+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:22.180341+0000) 2024-04-27T03:59:22.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:22.460+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:22.461049+0000) 2024-04-27T03:59:22.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:22.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:22.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:22.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:22.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:22.330+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:22.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:22.331+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:23.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:22.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:23.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:22.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:23.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:22.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:23.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:22.951+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:23.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:23.043+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:23.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:23.043+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:23.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:23.043+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:23.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:23.044+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:23.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:23.181+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:23.182501+0000) 2024-04-27T03:59:23.466 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:23.269+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:23.270193+0000) 2024-04-27T03:59:23.466 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:23.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:23.466 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:23.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:23.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:23.307+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:23.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:23.308+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:23.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:23.475+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:23.476313+0000) 2024-04-27T03:59:23.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:59:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:59:23.480+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 77989 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:59:23.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:23.465+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:23.465843+0000) 2024-04-27T03:59:23.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:23.475+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:23.476691+0000) 2024-04-27T03:59:23.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:23.476+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:23.477360+0000) 2024-04-27T03:59:24.185 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:24.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:24.186 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:24.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:24.186 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:24.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:24.186 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:24.067+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:24.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:23.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:24.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:23.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:24.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:23.915+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:24.187 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:23.915+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:24.468 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:24.184+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:24.185165+0000) 2024-04-27T03:59:24.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:24.271+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:24.272425+0000) 2024-04-27T03:59:24.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:24.328+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:24.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:24.328+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:24.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:24.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:24.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:24.330+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:24.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:24.467+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:24.468073+0000) 2024-04-27T03:59:25.083 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:24.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:25.083 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:24.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:25.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:24.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:25.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:24.956+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:25.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:25.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:25.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:25.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:25.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:25.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:25.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:25.083+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:25.378 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:25.188+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:25.189140+0000) 2024-04-27T03:59:25.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:25.273+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:25.274546+0000) 2024-04-27T03:59:25.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:25.311+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:25.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:25.311+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:25.378 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:25.311+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:25.379 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:25.312+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:25.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:25.470+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:25.470825+0000) 2024-04-27T03:59:26.190 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:26.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:26.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:26.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:26.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:26.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:26.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:26.084+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:26.191 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:25.926+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:26.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:25.926+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:26.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:25.926+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:26.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:25.926+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:26.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:26.191+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:26.192208+0000) 2024-04-27T03:59:26.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:26.276+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:26.277574+0000) 2024-04-27T03:59:26.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:26.304+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:26.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:26.304+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:26.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:26.304+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:26.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:26.305+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:26.865 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:26.474+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:26.475056+0000) 2024-04-27T03:59:27.190 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:27.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:27.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:27.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:27.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:27.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:27.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:27.110+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:27.191 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:26.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:27.191 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:26.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:27.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:26.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:27.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:26.919+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:27.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:27.195+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:27.196461+0000) 2024-04-27T03:59:27.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:27.278+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:27.279741+0000) 2024-04-27T03:59:27.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:27.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:27.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:27.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:27.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:27.293+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:27.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:27.294+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:27.865 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:27.477+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:27.478253+0000) 2024-04-27T03:59:28.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:27.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:28.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:27.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:28.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:27.923+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:28.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:27.923+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:28.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:28.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:28.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:28.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:28.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:28.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:28.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:28.157+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:28.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:28.199+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:28.200684+0000) 2024-04-27T03:59:28.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:28.281+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:28.281882+0000) 2024-04-27T03:59:28.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:28.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:28.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:28.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:28.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:28.297+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:28.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:28.298+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:28.865 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:59:28.489+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78029 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:59:28.866 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:28.485+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:28.486751+0000) 2024-04-27T03:59:28.866 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:28.479+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:28.480641+0000) 2024-04-27T03:59:28.866 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:28.486+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:28.487688+0000) 2024-04-27T03:59:28.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:28.485+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:28.486181+0000) 2024-04-27T03:59:29.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:29.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:29.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:29.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:29.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:29.108+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:29.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:29.109+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:29.191 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:28.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:29.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:28.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:29.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:28.921+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:29.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:28.921+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:29.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:29.202+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:29.202843+0000) 2024-04-27T03:59:29.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:29.283+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:29.284045+0000) 2024-04-27T03:59:29.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:29.323+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:29.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:29.323+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:29.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:29.323+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:29.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:29.325+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:29.868 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:29.483+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:29.483965+0000) 2024-04-27T03:59:30.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:29.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:30.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:29.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:30.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:29.880+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:30.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:29.880+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:30.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:30.100+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:30.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:30.100+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:30.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:30.100+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:30.160 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:30.102+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:30.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:30.204+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:30.204982+0000) 2024-04-27T03:59:30.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:30.279+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:30.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:30.279+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:30.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:30.279+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:30.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:30.280+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:30.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:30.287+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:30.288324+0000) 2024-04-27T03:59:30.871 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:30.485+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:30.486261+0000) 2024-04-27T03:59:31.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:31.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:31.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:31.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:31.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:31.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:31.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:31.090+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:31.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:30.928+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:31.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:30.928+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:31.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:30.928+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:31.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:30.929+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:31.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:31.207+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:31.208095+0000) 2024-04-27T03:59:31.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:31.292+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:31.293049+0000) 2024-04-27T03:59:31.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:31.295+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:31.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:31.295+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:31.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:31.295+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:31.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:31.296+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:31.873 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:31.487+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:31.488417+0000) 2024-04-27T03:59:32.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:32.100+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:32.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:32.100+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:32.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:32.100+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:32.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:32.100+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:32.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:31.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:32.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:31.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:32.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:31.930+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:32.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:31.931+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:32.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:32.210+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:32.211246+0000) 2024-04-27T03:59:32.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:32.296+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:32.297441+0000) 2024-04-27T03:59:32.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:32.319+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:32.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:32.319+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:32.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:32.319+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:32.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:32.321+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:32.873 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:32.490+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:32.491443+0000) 2024-04-27T03:59:33.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:33.117+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:33.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:33.117+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:33.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:33.117+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:33.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:33.118+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:33.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:32.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:33.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:32.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:33.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:32.898+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:33.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:32.899+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:33.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:33.214+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:33.215501+0000) 2024-04-27T03:59:33.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:33.298+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:33.299607+0000) 2024-04-27T03:59:33.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:33.343+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:33.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:33.343+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:33.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:33.343+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:33.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:33.344+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:33.873 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:59:33.498+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78069 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:59:33.874 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:33.495+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:33.496252+0000) 2024-04-27T03:59:33.874 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:33.494+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:33.495680+0000) 2024-04-27T03:59:33.874 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:33.494+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:33.494878+0000) 2024-04-27T03:59:33.874 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:33.494+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:33.494910+0000) 2024-04-27T03:59:34.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:33.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:34.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:33.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:34.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:33.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:34.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:33.940+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:34.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:34.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:34.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:34.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:34.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:34.090+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:34.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:34.092+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:34.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:34.216+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:34.217628+0000) 2024-04-27T03:59:34.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:34.303+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:34.303893+0000) 2024-04-27T03:59:34.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:34.345+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:34.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:34.345+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:34.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:34.345+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:34.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:34.345+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:34.875 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:34.497+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:34.497998+0000) 2024-04-27T03:59:35.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:34.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:35.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:34.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:35.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:34.975+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:35.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:34.976+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:35.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:35.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:35.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:35.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:35.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:35.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:35.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:35.133+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:35.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:35.221+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:35.222187+0000) 2024-04-27T03:59:35.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:35.307+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:35.308174+0000) 2024-04-27T03:59:35.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:35.325+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:35.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:35.325+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:35.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:35.325+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:35.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:35.325+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:35.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:35.499+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:35.500340+0000) 2024-04-27T03:59:36.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:35.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:36.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:35.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:36.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:35.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:36.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:35.952+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:36.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:36.143+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:36.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:36.143+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:36.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:36.143+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:36.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:36.144+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:36.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:36.225+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:36.226390+0000) 2024-04-27T03:59:36.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:36.294+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:36.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:36.294+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:36.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:36.294+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:36.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:36.295+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:36.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:36.310+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:36.310842+0000) 2024-04-27T03:59:36.881 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:36.502+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:36.503681+0000) 2024-04-27T03:59:37.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:37.115+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:37.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:37.115+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:37.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:37.115+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:37.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:37.116+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:37.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:36.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:37.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:36.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:37.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:36.905+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:37.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:36.906+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:37.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:37.228+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:37.229360+0000) 2024-04-27T03:59:37.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:37.296+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:37.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:37.296+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:37.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:37.296+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:37.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:37.297+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:37.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:37.314+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:37.315202+0000) 2024-04-27T03:59:37.881 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:37.507+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:37.507962+0000) 2024-04-27T03:59:38.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:37.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:38.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:37.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:38.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:37.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:38.138 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:37.943+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:38.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:38.137+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:38.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:38.137+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:38.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:38.137+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:38.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:38.138+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:38.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:38.233+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:38.234158+0000) 2024-04-27T03:59:38.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:38.316+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:38.317467+0000) 2024-04-27T03:59:38.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:38.323+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:38.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:38.323+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:38.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:38.323+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:38.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:38.324+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:38.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:38.504+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:38.504881+0000) 2024-04-27T03:59:38.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:38.509+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:38.510201+0000) 2024-04-27T03:59:38.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:59:38.507+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78109 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:59:38.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:38.503+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:38.504065+0000) 2024-04-27T03:59:38.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:38.504+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:38.505014+0000) 2024-04-27T03:59:39.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:38.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:39.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:38.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:39.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:38.894+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:39.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:38.894+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:39.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:39.281+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:39.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:39.281+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:39.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:39.281+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:39.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:39.282+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:39.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:39.318+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:39.319725+0000) 2024-04-27T03:59:39.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:39.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:39.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:39.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:39.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:39.164+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:39.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:39.165+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:39.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:39.236+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:39.237270+0000) 2024-04-27T03:59:39.884 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:39.513+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:39.514086+0000) 2024-04-27T03:59:40.158 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:39.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:40.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:39.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:40.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:39.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:40.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:39.919+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:40.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:40.202+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:40.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:40.203+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:40.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:40.203+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:40.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:40.203+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:40.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:40.239+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:40.240226+0000) 2024-04-27T03:59:40.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:40.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:40.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:40.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:40.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:40.310+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:40.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:40.311+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:40.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:40.322+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:40.322999+0000) 2024-04-27T03:59:40.886 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:40.515+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:40.516292+0000) 2024-04-27T03:59:41.155 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:40.959+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:41.155 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:40.959+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:41.155 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:40.959+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:41.156 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:40.959+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:41.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:41.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:41.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:41.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:41.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:41.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:41.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:41.155+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:41.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:41.241+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:41.242361+0000) 2024-04-27T03:59:41.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:41.324+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:41.325277+0000) 2024-04-27T03:59:41.418 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:41.327+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:41.418 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:41.327+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:41.418 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:41.327+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:41.418 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:41.329+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:41.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:41.519+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:41.520591+0000) 2024-04-27T03:59:42.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:42.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:42.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:42.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:42.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:42.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:42.193 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:42.133+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:42.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:41.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:42.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:41.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:42.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:41.978+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:42.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:41.978+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:42.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:42.243+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:42.244560+0000) 2024-04-27T03:59:42.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:42.326+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:42.327534+0000) 2024-04-27T03:59:42.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:42.357+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:42.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:42.357+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:42.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:42.357+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:42.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:42.358+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:42.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:42.522+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:42.522904+0000) 2024-04-27T03:59:43.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:42.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:43.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:42.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:43.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:42.935+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:43.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:42.935+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:43.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:43.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:43.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:43.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:43.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:43.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:43.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:43.139+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:43.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:43.246+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:43.247667+0000) 2024-04-27T03:59:43.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:43.329+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:43.330391+0000) 2024-04-27T03:59:43.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:43.343+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:43.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:43.343+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:43.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:43.343+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:43.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:43.343+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:43.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:43.513+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:43.513954+0000) 2024-04-27T03:59:43.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:43.512+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:43.512839+0000) 2024-04-27T03:59:43.722 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:59:43.517+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78149 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:59:43.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:43.512+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:43.513116+0000) 2024-04-27T03:59:43.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:43.527+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:43.527816+0000) 2024-04-27T03:59:44.175 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:43.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:44.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:43.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:44.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:43.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:44.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:43.918+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:44.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:44.331+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:44.332533+0000) 2024-04-27T03:59:44.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:44.341+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:44.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:44.341+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:44.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:44.341+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:44.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:44.343+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:44.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:44.174+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:44.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:44.174+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:44.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:44.174+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:44.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:44.175+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:44.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:44.248+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:44.249769+0000) 2024-04-27T03:59:44.891 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:44.532+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:44.532956+0000) 2024-04-27T03:59:45.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:44.967+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:45.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:44.967+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:45.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:44.967+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:45.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:44.968+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:45.193 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:45.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:45.193 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:45.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:45.193 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:45.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:45.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:45.139+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:45.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:45.251+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:45.251842+0000) 2024-04-27T03:59:45.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:45.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:45.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:45.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:45.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:45.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:45.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:45.330+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:45.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:45.336+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:45.336820+0000) 2024-04-27T03:59:45.893 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:45.534+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:45.535104+0000) 2024-04-27T03:59:46.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:45.920+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:46.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:45.920+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:46.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:45.920+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:46.172 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:45.921+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:46.459 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:46.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:46.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:46.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:46.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:46.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:46.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:46.172+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:46.460 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:46.253+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:46.254478+0000) 2024-04-27T03:59:46.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:46.299+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:46.460 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:46.299+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:46.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:46.299+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:46.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:46.300+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:46.461 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:46.340+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:46.341194+0000) 2024-04-27T03:59:46.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:46.537+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:46.538175+0000) 2024-04-27T03:59:47.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:46.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:47.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:46.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:47.176 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:46.946+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:47.177 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:46.947+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:47.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:47.174+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:47.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:47.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:47.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:47.175+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:47.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:47.176+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:47.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:47.256+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:47.256848+0000) 2024-04-27T03:59:47.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:47.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:47.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:47.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:47.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:47.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:47.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:47.330+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:47.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:47.345+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:47.345881+0000) 2024-04-27T03:59:47.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:47.539+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:47.540611+0000) 2024-04-27T03:59:48.193 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:48.131+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:48.193 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:48.131+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:48.193 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:48.131+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:48.193 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:48.131+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:48.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:47.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:48.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:47.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:48.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:47.995+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:48.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:47.995+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:48.474 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:48.259+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:48.260133+0000) 2024-04-27T03:59:48.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:48.320+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:48.474 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:48.320+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:48.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:48.320+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:48.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:48.321+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:48.475 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:48.347+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:48.348527+0000) 2024-04-27T03:59:48.894 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:59:48.526+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78189 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:59:48.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:48.522+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:48.523018+0000) 2024-04-27T03:59:48.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:48.521+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:48.521988+0000) 2024-04-27T03:59:48.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:48.522+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:48.523118+0000) 2024-04-27T03:59:48.896 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:48.542+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:48.543021+0000) 2024-04-27T03:59:49.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:49.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:49.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:49.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:49.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:49.009+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:49.167 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:49.009+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:49.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:49.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:49.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:49.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:49.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:49.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:49.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:49.310+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:49.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:49.352+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:49.352887+0000) 2024-04-27T03:59:49.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:49.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:49.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:49.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:49.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:49.165+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:49.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:49.166+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:49.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:49.261+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:49.262273+0000) 2024-04-27T03:59:49.899 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:49.546+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:49.547267+0000) 2024-04-27T03:59:50.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:50.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:50.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:50.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:50.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:50.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:50.159 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:50.142+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:50.159 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:50.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:50.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:50.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:50.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:50.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:50.160 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:50.056+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:50.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:50.264+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:50.265248+0000) 2024-04-27T03:59:50.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:50.346+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:50.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:50.346+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:50.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:50.346+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:50.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:50.347+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:50.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:50.354+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:50.354924+0000) 2024-04-27T03:59:50.903 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:50.548+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:50.549545+0000) 2024-04-27T03:59:51.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:51.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:51.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:51.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:51.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:51.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:51.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:51.099+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:51.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:51.134+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:51.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:51.134+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:51.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:51.134+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:51.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:51.135+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:51.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:51.266+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:51.267500+0000) 2024-04-27T03:59:51.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:51.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:51.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:51.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:51.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:51.316+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:51.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:51.316+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:51.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:51.357+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:51.358081+0000) 2024-04-27T03:59:51.904 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:51.551+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:51.551842+0000) 2024-04-27T03:59:52.193 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:52.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:52.193 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:52.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:52.193 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:52.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:52.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:52.148+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:52.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:52.144+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:52.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:52.144+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:52.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:52.144+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:52.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:52.144+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:52.447 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:52.268+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:52.269610+0000) 2024-04-27T03:59:52.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:52.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:52.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:52.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:52.447 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:52.270+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:52.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:52.271+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:52.448 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:52.359+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:52.360743+0000) 2024-04-27T03:59:52.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:52.553+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:52.554557+0000) 2024-04-27T03:59:53.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:53.141+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:53.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:53.141+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:53.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:53.141+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:53.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:53.141+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:53.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:53.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:53.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:53.193+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:53.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:53.194+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:53.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:53.195+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:53.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:53.271+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:53.272343+0000) 2024-04-27T03:59:53.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:53.312+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:53.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:53.312+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:53.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:53.312+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:53.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:53.313+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:53.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:53.362+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:53.362933+0000) 2024-04-27T03:59:53.904 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:59:53.539+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78229 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:59:53.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:53.531+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:53.532267+0000) 2024-04-27T03:59:53.905 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:53.530+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:53.531736+0000) 2024-04-27T03:59:53.905 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:53.532+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:53.532844+0000) 2024-04-27T03:59:53.906 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:53.558+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:53.558830+0000) 2024-04-27T03:59:54.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:54.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:54.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:54.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:54.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:54.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:54.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:54.149+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:54.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:54.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:54.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:54.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:54.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:54.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:54.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:54.108+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:54.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:54.273+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:54.274626+0000) 2024-04-27T03:59:54.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:54.283+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:54.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:54.283+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:54.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:54.283+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:54.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:54.284+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:54.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:54.365+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:54.366124+0000) 2024-04-27T03:59:54.907 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:54.560+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:54.561081+0000) 2024-04-27T03:59:55.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:55.107+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:55.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:55.107+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:55.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:55.107+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:55.158 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:55.109+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:55.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:55.276+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:55.277737+0000) 2024-04-27T03:59:55.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:55.156+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:55.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:55.157+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:55.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:55.157+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:55.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:55.157+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:55.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:55.263+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:55.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:55.263+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:55.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:55.263+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:55.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:55.264+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:55.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:55.367+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:55.368242+0000) 2024-04-27T03:59:55.910 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:55.563+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:55.564246+0000) 2024-04-27T03:59:56.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:56.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:56.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:56.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:56.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:56.146+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:56.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:56.146+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:56.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:56.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:56.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:56.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:56.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:56.132+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:56.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:56.133+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:56.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:56.279+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:56.279826+0000) 2024-04-27T03:59:56.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:56.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:56.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:56.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:56.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:56.280+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:56.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:56.281+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:56.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:56.369+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:56.370348+0000) 2024-04-27T03:59:56.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:56.566+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:56.566940+0000) 2024-04-27T03:59:57.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:57.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:57.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:57.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:57.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:57.173+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:57.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:57.174+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:57.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:57.173+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:57.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:57.173+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:57.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:57.173+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:57.472 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:57.174+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:57.473 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:57.281+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:57.282111+0000) 2024-04-27T03:59:57.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:57.285+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:57.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:57.285+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:57.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:57.285+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:57.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:57.285+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:57.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:57.372+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:57.373520+0000) 2024-04-27T03:59:57.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:57.569+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:57.569811+0000) 2024-04-27T03:59:58.193 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:58.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:58.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:58.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:58.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:58.153+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:58.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:58.154+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:58.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:58.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:58.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:58.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:58.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:58.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:58.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:58.133+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:58.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:58.283+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:58.284273+0000) 2024-04-27T03:59:58.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:58.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:58.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:58.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:58.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:58.256+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:58.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:58.257+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:58.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:58.374+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:58.375826+0000) 2024-04-27T03:59:58.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:58.543+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:58.544979+0000) 2024-04-27T03:59:58.912 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 03:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T03:59:58.547+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78269 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T03:59:58.912 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:58.543+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:58.545104+0000) 2024-04-27T03:59:58.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:58.543+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:58.545462+0000) 2024-04-27T03:59:58.913 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:58.570+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:58.572099+0000) 2024-04-27T03:59:59.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:59.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:59.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:59.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:59.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:59.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T03:59:59.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:59.106+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:59.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:59.154+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:59.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:59.154+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:59.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:59.154+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T03:59:59.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:59.154+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:59.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 03:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T03:59:59.286+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:59.288557+0000) 2024-04-27T03:59:59.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:59.228+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:59.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:59.228+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:59.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:59.228+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T03:59:59.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:59.230+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T03:59:59.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 03:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T03:59:59.376+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:59.377990+0000) 2024-04-27T03:59:59.914 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 03:59:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T03:59:59.572+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T02:59:59.574315+0000) 2024-04-27T04:00:00.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:00.114+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:00.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:00.114+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:00.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:00.114+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:00.192 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:00.114+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:00.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:00.288+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:00.290751+0000) 2024-04-27T04:00:00.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:00.189+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:00.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:00.189+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:00.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:00.189+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:00.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:00.190+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:00.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:00.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:00.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:00.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:00.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:00.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:00.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:00.209+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:00.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:00.378+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:00.380212+0000) 2024-04-27T04:00:00.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:00.574+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:00.576521+0000) 2024-04-27T04:00:01.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:01.100+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:01.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:01.100+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:01.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:01.100+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:01.182 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:01.102+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:01.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:01.293+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:01.294975+0000) 2024-04-27T04:00:01.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:01.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:01.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:01.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:01.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:01.179+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:01.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:01.180+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:01.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:01.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:01.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:01.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:01.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:01.226+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:01.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:01.227+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:01.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:01.380+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:01.382451+0000) 2024-04-27T04:00:01.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:01.577+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:01.579376+0000) 2024-04-27T04:00:02.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:02.097+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:02.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:02.097+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:02.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:02.097+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:02.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:02.099+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:02.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:02.295+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:02.297063+0000) 2024-04-27T04:00:02.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:02.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:02.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:02.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:02.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:02.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:02.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:02.215+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:02.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:02.228+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:02.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:02.228+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:02.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:02.228+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:02.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:02.228+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:02.473 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:02.382+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:02.384591+0000) 2024-04-27T04:00:02.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:02.581+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:02.582917+0000) 2024-04-27T04:00:03.178 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:03.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:03.178 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:03.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:03.178 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:03.148+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:03.179 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:03.149+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:03.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:03.298+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:03.299786+0000) 2024-04-27T04:00:03.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:03.262+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:03.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:03.262+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:03.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:03.262+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:03.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:03.263+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:03.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:03.387+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:03.388908+0000) 2024-04-27T04:00:03.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:03.176+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:03.472 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:03.176+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:03.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:03.176+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:03.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:03.176+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:03.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:00:03.556+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78311 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:00:03.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:03.553+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:03.555620+0000) 2024-04-27T04:00:03.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:03.553+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:03.554892+0000) 2024-04-27T04:00:03.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:03.583+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:03.585596+0000) 2024-04-27T04:00:03.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:03.552+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:03.554312+0000) 2024-04-27T04:00:04.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:04.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:04.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:04.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:04.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:04.120+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:04.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:04.120+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:04.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:04.136+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:04.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:04.136+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:04.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:04.136+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:04.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:04.136+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:04.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:04.301+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:04.303029+0000) 2024-04-27T04:00:04.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:04.223+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:04.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:04.223+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:04.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:04.223+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:04.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:04.225+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:04.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:04.389+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:04.391047+0000) 2024-04-27T04:00:04.922 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:04.588+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:04.590690+0000) 2024-04-27T04:00:05.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:05.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:05.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:05.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:05.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:05.156+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:05.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:05.157+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:05.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:05.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:05.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:05.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:05.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:05.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:05.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:05.111+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:05.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:05.305+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:05.307224+0000) 2024-04-27T04:00:05.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:05.246+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:05.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:05.246+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:05.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:05.246+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:05.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:05.247+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:05.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:05.391+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:05.393206+0000) 2024-04-27T04:00:05.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:05.591+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:05.592821+0000) 2024-04-27T04:00:06.192 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:06.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:06.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:06.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:06.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:06.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:06.193 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:06.124+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:06.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:06.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:06.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:06.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:06.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:06.190+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:06.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:06.191+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:06.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:06.309+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:06.311417+0000) 2024-04-27T04:00:06.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:06.229+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:06.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:06.229+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:06.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:06.229+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:06.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:06.230+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:06.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:06.395+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:06.397530+0000) 2024-04-27T04:00:06.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:06.593+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:06.595074+0000) 2024-04-27T04:00:07.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:07.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:07.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:07.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:07.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:07.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:07.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:07.083+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:07.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:07.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:07.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:07.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:07.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:07.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:07.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:07.173+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:07.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:07.314+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:07.316575+0000) 2024-04-27T04:00:07.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:07.214+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:07.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:07.214+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:07.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:07.214+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:07.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:07.215+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:07.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:07.397+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:07.399777+0000) 2024-04-27T04:00:07.926 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:07.595+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:07.597309+0000) 2024-04-27T04:00:08.180 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:08.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:08.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:08.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:08.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:08.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:08.181 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:08.155+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:08.181 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:08.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:08.181 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:08.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:08.181 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:08.069+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:08.182 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:08.069+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:08.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:08.317+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:08.318879+0000) 2024-04-27T04:00:08.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:08.178+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:08.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:08.178+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:08.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:08.178+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:08.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:08.179+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:08.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:08.401+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:08.403040+0000) 2024-04-27T04:00:08.926 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:00:08.566+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78351 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:00:08.927 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:08.562+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:08.563843+0000) 2024-04-27T04:00:08.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:08.560+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:08.562685+0000) 2024-04-27T04:00:08.927 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:08.562+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:08.563982+0000) 2024-04-27T04:00:08.927 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:08.599+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:08.601602+0000) 2024-04-27T04:00:09.190 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:09.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:09.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:09.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:09.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:09.147+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:09.191 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:09.148+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:09.191 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:09.071+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:09.191 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:09.071+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:09.191 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:09.071+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:09.191 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:09.072+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:09.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:09.322+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:09.323143+0000) 2024-04-27T04:00:09.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:09.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:09.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:09.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:09.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:09.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:09.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:09.189+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:09.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:09.403+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:09.405334+0000) 2024-04-27T04:00:09.929 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:09.602+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:09.603759+0000) 2024-04-27T04:00:10.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:10.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:10.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:10.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:10.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:10.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:10.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:10.083+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:10.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:10.149+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:10.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:10.149+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:10.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:10.149+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:10.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:10.150+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:10.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:10.323+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:10.325286+0000) 2024-04-27T04:00:10.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:10.225+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:10.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:10.225+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:10.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:10.225+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:10.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:10.226+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:10.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:10.407+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:10.407930+0000) 2024-04-27T04:00:10.878 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:10.607+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:10.608156+0000) 2024-04-27T04:00:11.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:11.035+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:11.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:11.035+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:11.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:11.035+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:11.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:11.036+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:11.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:11.140+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:11.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:11.140+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:11.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:11.140+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:11.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:11.141+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:11.396 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:11.326+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:11.327463+0000) 2024-04-27T04:00:11.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:11.274+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:11.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:11.274+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:11.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:11.274+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:11.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:11.275+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:11.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:11.409+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:11.410050+0000) 2024-04-27T04:00:11.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:11.609+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:11.610488+0000) 2024-04-27T04:00:12.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:12.096+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:12.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:12.096+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:12.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:12.096+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:12.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:12.097+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:12.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:11.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:12.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:11.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:12.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:11.998+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:12.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:11.998+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:12.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:12.329+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:12.329822+0000) 2024-04-27T04:00:12.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:12.303+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:12.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:12.303+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:12.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:12.303+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:12.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:12.304+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:12.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:12.411+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:12.412002+0000) 2024-04-27T04:00:12.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:12.611+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:12.612706+0000) 2024-04-27T04:00:13.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:13.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:13.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:13.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:13.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:13.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:13.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:13.062+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:13.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:13.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:13.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:13.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:13.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:13.027+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:13.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:13.027+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:13.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:13.331+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:13.331956+0000) 2024-04-27T04:00:13.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:13.311+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:13.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:13.311+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:13.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:13.311+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:13.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:13.311+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:13.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:13.415+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:13.416343+0000) 2024-04-27T04:00:13.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:00:13.576+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78391 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:00:13.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:13.572+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:13.573307+0000) 2024-04-27T04:00:13.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:13.572+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:13.573108+0000) 2024-04-27T04:00:13.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:13.614+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:13.614845+0000) 2024-04-27T04:00:13.934 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:13.572+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:13.573130+0000) 2024-04-27T04:00:14.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:14.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:14.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:14.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:14.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:14.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:14.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:14.060+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:14.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:14.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:14.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:14.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:14.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:14.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:14.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:14.009+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:14.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:14.333+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:14.334210+0000) 2024-04-27T04:00:14.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:14.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:14.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:14.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:14.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:14.313+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:14.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:14.313+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:14.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:14.417+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:14.418514+0000) 2024-04-27T04:00:14.936 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:14.616+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:14.616944+0000) 2024-04-27T04:00:15.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:15.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:15.195 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:15.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:15.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:15.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:15.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:15.046+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:15.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:15.069+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:15.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:15.069+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:15.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:15.069+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:15.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:15.069+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:15.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:15.337+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:15.338390+0000) 2024-04-27T04:00:15.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:15.290+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:15.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:15.290+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:15.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:15.290+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:15.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:15.291+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:15.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:15.419+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:15.420609+0000) 2024-04-27T04:00:15.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:15.618+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:15.619094+0000) 2024-04-27T04:00:16.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:16.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:16.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:16.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:16.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:16.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:16.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:16.061+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:16.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:16.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:16.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:16.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:16.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:16.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:16.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:16.036+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:16.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:16.340+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:16.341125+0000) 2024-04-27T04:00:16.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:16.324+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:16.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:16.324+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:16.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:16.324+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:16.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:16.325+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:16.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:16.422+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:16.423663+0000) 2024-04-27T04:00:16.940 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:16.620+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:16.621176+0000) 2024-04-27T04:00:17.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:17.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:17.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:17.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:17.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:17.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:17.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:17.106+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:17.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:17.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:17.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:17.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:17.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:17.005+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:17.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:17.005+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:17.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:17.342+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:17.343427+0000) 2024-04-27T04:00:17.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:17.345+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:17.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:17.345+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:17.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:17.345+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:17.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:17.346+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:17.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:17.425+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:17.425917+0000) 2024-04-27T04:00:17.940 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:17.622+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:17.623355+0000) 2024-04-27T04:00:18.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:18.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:18.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:18.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:18.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:18.111+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:18.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:18.111+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:18.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:18.003+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:18.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:18.003+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:18.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:18.003+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:18.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:18.003+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:18.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:18.344+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:18.345684+0000) 2024-04-27T04:00:18.498 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:18.325+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:18.498 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:18.325+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:18.498 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:18.325+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:18.499 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:18.326+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:18.499 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:18.427+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:18.428035+0000) 2024-04-27T04:00:18.940 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:00:18.585+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78431 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:00:18.941 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:18.582+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:18.583272+0000) 2024-04-27T04:00:18.941 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:18.582+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:18.582965+0000) 2024-04-27T04:00:18.942 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:18.624+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:18.625503+0000) 2024-04-27T04:00:18.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:18.581+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:18.582754+0000) 2024-04-27T04:00:19.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:19.016+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:19.196 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:19.016+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:19.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:19.016+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:19.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:19.016+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:19.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:19.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:19.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:19.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:19.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:19.141+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:19.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:19.141+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:19.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:19.347+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:19.347800+0000) 2024-04-27T04:00:19.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:19.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:19.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:19.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:19.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:19.352+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:19.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:19.353+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:19.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:19.429+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:19.430154+0000) 2024-04-27T04:00:19.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:19.629+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:19.630484+0000) 2024-04-27T04:00:20.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:20.117+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:20.196 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:20.117+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:20.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:20.117+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:20.197 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:20.118+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:20.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:19.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:20.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:19.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:20.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:19.973+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:20.197 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:19.974+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:20.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:20.349+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:20.350650+0000) 2024-04-27T04:00:20.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:20.353+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:20.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:20.353+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:20.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:20.353+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:20.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:20.353+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:20.472 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:20.431+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:20.432251+0000) 2024-04-27T04:00:20.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:20.634+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:20.635396+0000) 2024-04-27T04:00:21.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:21.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:21.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:21.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:21.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:21.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:21.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:21.016+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:21.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:21.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:21.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:21.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:21.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:21.136+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:21.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:21.137+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:21.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:21.354+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:21.355299+0000) 2024-04-27T04:00:21.639 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:21.353+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:21.639 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:21.353+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:21.639 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:21.353+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:21.639 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:21.354+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:21.639 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:21.436+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:21.436821+0000) 2024-04-27T04:00:21.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:21.637+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:21.638328+0000) 2024-04-27T04:00:22.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:21.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:22.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:21.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:22.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:21.971+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:22.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:21.972+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:22.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:22.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:22.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:22.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:22.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:22.172+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:22.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:22.173+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:22.643 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:22.359+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:22.360032+0000) 2024-04-27T04:00:22.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:22.337+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:22.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:22.337+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:22.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:22.337+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:22.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:22.338+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:22.644 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:22.438+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:22.439518+0000) 2024-04-27T04:00:22.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:22.642+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:22.643327+0000) 2024-04-27T04:00:23.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:22.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:23.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:22.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:23.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:22.963+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:23.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:22.964+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:23.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:23.130+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:23.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:23.130+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:23.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:23.130+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:23.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:23.132+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:23.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:23.361+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:23.362203+0000) 2024-04-27T04:00:23.588 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:23.342+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:23.588 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:23.342+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:23.588 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:23.342+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:23.588 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:23.343+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:23.588 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:23.440+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:23.441666+0000) 2024-04-27T04:00:23.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:23.591+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:23.592256+0000) 2024-04-27T04:00:23.950 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:00:23.594+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78471 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:00:23.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:23.590+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:23.591253+0000) 2024-04-27T04:00:23.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:23.591+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:23.592489+0000) 2024-04-27T04:00:23.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:23.644+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:23.645757+0000) 2024-04-27T04:00:24.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:24.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:24.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:24.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:24.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:24.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:24.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:24.090+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:24.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:23.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:24.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:23.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:24.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:23.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:24.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:23.962+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:24.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:24.363+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:24.364411+0000) 2024-04-27T04:00:24.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:24.326+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:24.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:24.326+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:24.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:24.326+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:24.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:24.327+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:24.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:24.444+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:24.444948+0000) 2024-04-27T04:00:24.954 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:24.647+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:24.647822+0000) 2024-04-27T04:00:25.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:25.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:25.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:25.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:25.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:25.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:25.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:25.008+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:25.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:25.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:25.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:25.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:25.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:25.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:25.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:25.066+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:25.650 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:25.365+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:25.366578+0000) 2024-04-27T04:00:25.650 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:25.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:25.651 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:25.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:25.651 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:25.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:25.651 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:25.305+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:25.651 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:25.448+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:25.449550+0000) 2024-04-27T04:00:25.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:25.649+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:25.650249+0000) 2024-04-27T04:00:26.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:26.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:26.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:26.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:26.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:26.060+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:26.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:26.060+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:26.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:25.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:26.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:25.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:26.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:25.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:26.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:25.982+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:26.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:26.370+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:26.370819+0000) 2024-04-27T04:00:26.655 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:26.260+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:26.655 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:26.260+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:26.656 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:26.260+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:26.656 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:26.261+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:26.656 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:26.453+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:26.454293+0000) 2024-04-27T04:00:26.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:26.653+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:26.654508+0000) 2024-04-27T04:00:27.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:26.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:27.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:26.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:27.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:26.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:27.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:27.000+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:27.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:27.015+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:27.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:27.015+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:27.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:27.015+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:27.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:27.016+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:27.657 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:27.374+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:27.375621+0000) 2024-04-27T04:00:27.657 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:27.257+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:27.657 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:27.257+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:27.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:27.257+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:27.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:27.258+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:27.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:27.457+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:27.458681+0000) 2024-04-27T04:00:27.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:27.656+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:27.657334+0000) 2024-04-27T04:00:28.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:28.042+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:28.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:28.042+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:28.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:28.042+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:28.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:28.042+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:28.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:28.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:28.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:28.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:28.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:28.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:28.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:28.029+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:28.596 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:28.379+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:28.380183+0000) 2024-04-27T04:00:28.597 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:28.303+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:28.597 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:28.303+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:28.597 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:28.303+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:28.598 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:28.304+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:28.598 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:28.460+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:28.460823+0000) 2024-04-27T04:00:28.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:28.599+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:28.600271+0000) 2024-04-27T04:00:28.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:28.600+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:28.601022+0000) 2024-04-27T04:00:28.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:00:28.604+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78512 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:00:28.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:28.599+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:28.600056+0000) 2024-04-27T04:00:28.960 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:28.659+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:28.660439+0000) 2024-04-27T04:00:29.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:29.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:29.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:29.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:29.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:29.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:29.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:29.002+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:29.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:29.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:29.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:29.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:29.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:29.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:29.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:29.039+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:29.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:29.382+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:29.383227+0000) 2024-04-27T04:00:29.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:29.350+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:29.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:29.350+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:29.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:29.350+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:29.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:29.352+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:29.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:29.462+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:29.463519+0000) 2024-04-27T04:00:29.961 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:29.662+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:29.663344+0000) 2024-04-27T04:00:30.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:29.991+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:30.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:29.991+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:30.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:29.991+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:30.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:29.992+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:30.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:30.071+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:30.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:30.071+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:30.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:30.071+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:30.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:30.071+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:30.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:30.384+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:30.385490+0000) 2024-04-27T04:00:30.666 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:30.402+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:30.666 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:30.402+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:30.666 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:30.402+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:30.666 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:30.402+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:30.666 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:30.467+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:30.467979+0000) 2024-04-27T04:00:30.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:30.664+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:30.665733+0000) 2024-04-27T04:00:31.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:31.103+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:31.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:31.103+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:31.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:31.103+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:31.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:31.104+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:31.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:31.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:31.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:31.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:31.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:31.040+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:31.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:31.041+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:31.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:31.386+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:31.387741+0000) 2024-04-27T04:00:31.669 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:31.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:31.669 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:31.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:31.669 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:31.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:31.669 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:31.389+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:31.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:31.469+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:31.470252+0000) 2024-04-27T04:00:31.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:31.668+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:31.668921+0000) 2024-04-27T04:00:32.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:31.995+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:32.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:31.995+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:32.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:31.995+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:32.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:31.996+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:32.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:32.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:32.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:32.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:32.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:32.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:32.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:32.080+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:32.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:32.391+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:32.392016+0000) 2024-04-27T04:00:32.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:32.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:32.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:32.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:32.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:32.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:32.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:32.404+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:32.674 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:32.472+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:32.473242+0000) 2024-04-27T04:00:32.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:32.672+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:32.673212+0000) 2024-04-27T04:00:33.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:33.034+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:33.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:33.034+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:33.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:33.034+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:33.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:33.035+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:33.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:33.034+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:33.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:33.034+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:33.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:33.034+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:33.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:33.035+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:33.606 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:33.393+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:33.394137+0000) 2024-04-27T04:00:33.607 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:33.356+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:33.607 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:33.356+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:33.607 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:33.356+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:33.607 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:33.357+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:33.607 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:33.474+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:33.475526+0000) 2024-04-27T04:00:33.963 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:00:33.613+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78552 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:00:33.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:33.612+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:33.613701+0000) 2024-04-27T04:00:33.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:33.609+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:33.610196+0000) 2024-04-27T04:00:33.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:33.674+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:33.675748+0000) 2024-04-27T04:00:33.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:33.608+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:33.609557+0000) 2024-04-27T04:00:34.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:34.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:34.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:34.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:34.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:34.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:34.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:34.012+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:34.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:34.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:34.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:34.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:34.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:34.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:34.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:34.006+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:34.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:34.396+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:34.397198+0000) 2024-04-27T04:00:34.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:34.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:34.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:34.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:34.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:34.403+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:34.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:34.404+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:34.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:34.476+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:34.477682+0000) 2024-04-27T04:00:34.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:34.679+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:34.680372+0000) 2024-04-27T04:00:34.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:34.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:34.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:34.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:34.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:34.957+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:35.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:34.957+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:35.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:34.999+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:35.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:34.999+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:35.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:34.999+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:35.222 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:35.000+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:35.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:35.399+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:35.399979+0000) 2024-04-27T04:00:35.683 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:35.390+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:35.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:35.390+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:35.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:35.390+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:35.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:35.390+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:35.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:35.480+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:35.480894+0000) 2024-04-27T04:00:35.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:35.682+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:35.683388+0000) 2024-04-27T04:00:35.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:35.920+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:35.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:35.920+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:35.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:35.920+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:35.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:35.921+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:36.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:35.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:36.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:35.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:36.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:35.973+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:36.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:35.975+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:36.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:36.401+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:36.402076+0000) 2024-04-27T04:00:36.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:36.376+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:36.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:36.377+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:36.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:36.377+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:36.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:36.377+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:36.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:36.482+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:36.482978+0000) 2024-04-27T04:00:36.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:36.686+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:36.687571+0000) 2024-04-27T04:00:36.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:36.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:36.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:36.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:36.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:36.918+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:36.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:36.919+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:37.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:36.969+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:37.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:36.969+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:37.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:36.969+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:37.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:36.971+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:37.690 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:37.403+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:37.404367+0000) 2024-04-27T04:00:37.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:37.368+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:37.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:37.368+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:37.691 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:37.368+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:37.691 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:37.368+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:37.691 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:37.484+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:37.485229+0000) 2024-04-27T04:00:37.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:37.689+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:37.690121+0000) 2024-04-27T04:00:37.945 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:37.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:37.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:37.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:37.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:37.944+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:38.198 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:37.945+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:38.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:38.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:38.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:38.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:38.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:38.012+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:38.198 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:38.013+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:38.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:38.406+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:38.407464+0000) 2024-04-27T04:00:38.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:38.381+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:38.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:38.381+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:38.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:38.381+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:38.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:38.382+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:38.744 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:00:38.623+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78593 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:00:38.745 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:38.619+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:38.619975+0000) 2024-04-27T04:00:38.745 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:38.486+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:38.487525+0000) 2024-04-27T04:00:38.745 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:38.619+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:38.619843+0000) 2024-04-27T04:00:38.746 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:38.618+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:38.619412+0000) 2024-04-27T04:00:38.746 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:38.693+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:38.694376+0000) 2024-04-27T04:00:39.021 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:38.949+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:39.021 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:38.949+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:39.021 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:38.949+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:39.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:38.950+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:39.272 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:39.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:39.272 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:39.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:39.272 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:39.020+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:39.272 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:39.021+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:39.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:39.409+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:39.410472+0000) 2024-04-27T04:00:39.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:39.354+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:39.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:39.354+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:39.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:39.354+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:39.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:39.355+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:39.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:39.491+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:39.491902+0000) 2024-04-27T04:00:39.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:39.697+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:39.698608+0000) 2024-04-27T04:00:40.272 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:39.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:40.272 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:39.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:40.273 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:39.981+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:40.273 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:39.982+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:40.273 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:40.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:40.273 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:40.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:40.273 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:40.002+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:40.273 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:40.004+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:40.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:40.413+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:40.414655+0000) 2024-04-27T04:00:40.703 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:40.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:40.703 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:40.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:40.703 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:40.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:40.703 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:40.398+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:40.703 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:40.493+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:40.494015+0000) 2024-04-27T04:00:40.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:40.701+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:40.702587+0000) 2024-04-27T04:00:41.272 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:41.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:41.273 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:41.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:41.273 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:41.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:41.273 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:41.042+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:41.273 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:40.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:41.273 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:40.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:41.273 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:40.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:41.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:40.981+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:41.598 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:41.417+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:41.417950+0000) 2024-04-27T04:00:41.598 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:41.361+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:41.598 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:41.361+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:41.598 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:41.361+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:41.599 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:41.363+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:41.599 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:41.495+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:41.496157+0000) 2024-04-27T04:00:41.954 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:41.705+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:41.705905+0000) 2024-04-27T04:00:42.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:42.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:42.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:42.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:42.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:42.065+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:42.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:42.067+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:42.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:41.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:42.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:41.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:42.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:41.960+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:42.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:41.961+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:42.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:42.419+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:42.420044+0000) 2024-04-27T04:00:42.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:42.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:42.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:42.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:42.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:42.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:42.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:42.349+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:42.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:42.497+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:42.498300+0000) 2024-04-27T04:00:42.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:42.709+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:42.710191+0000) 2024-04-27T04:00:43.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:43.064+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:43.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:43.064+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:43.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:43.064+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:43.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:43.064+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:43.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:42.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:43.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:42.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:43.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:42.969+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:43.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:42.969+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:43.625 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:43.421+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:43.422187+0000) 2024-04-27T04:00:43.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:43.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:43.625 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:43.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:43.626 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:43.329+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:43.626 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:43.331+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:43.626 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:43.501+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:43.502592+0000) 2024-04-27T04:00:43.963 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:00:43.632+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78633 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:00:43.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:43.627+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:43.628464+0000) 2024-04-27T04:00:43.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:43.627+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:43.628311+0000) 2024-04-27T04:00:43.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:43.627+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:43.628284+0000) 2024-04-27T04:00:43.964 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:43.711+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:43.712680+0000) 2024-04-27T04:00:44.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:44.068+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:44.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:44.068+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:44.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:44.068+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:44.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:44.069+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:44.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:43.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:44.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:43.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:44.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:43.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:44.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:43.988+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:44.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:44.423+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:44.424375+0000) 2024-04-27T04:00:44.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:44.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:44.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:44.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:44.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:44.314+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:44.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:44.314+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:44.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:44.504+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:44.504846+0000) 2024-04-27T04:00:44.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:44.716+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:44.716900+0000) 2024-04-27T04:00:45.270 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:44.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:45.270 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:44.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:45.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:44.988+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:45.271 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:44.989+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:45.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:45.083+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:45.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:45.083+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:45.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:45.083+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:45.271 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:45.085+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:45.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:45.425+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:45.426540+0000) 2024-04-27T04:00:45.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:45.269+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:45.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:45.269+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:45.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:45.269+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:45.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:45.270+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:45.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:45.506+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:45.506906+0000) 2024-04-27T04:00:45.986 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:45.720+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:45.721190+0000) 2024-04-27T04:00:46.273 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:46.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:46.273 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:46.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:46.273 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:46.022+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:46.273 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:46.023+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:46.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:46.228+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:46.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:46.228+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:46.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:46.228+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:46.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:46.229+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:46.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:46.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:46.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:46.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:46.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:46.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:46.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:46.075+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:46.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:46.508+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:46.508983+0000) 2024-04-27T04:00:46.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:46.430+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:46.431548+0000) 2024-04-27T04:00:46.988 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:46.722+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:46.723557+0000) 2024-04-27T04:00:47.273 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:47.042+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:47.273 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:47.042+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:47.273 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:47.042+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:47.273 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:47.042+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:47.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:47.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:47.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:47.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:47.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:47.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:47.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:47.033+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:47.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:47.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:47.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:47.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:47.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:47.218+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:47.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:47.219+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:47.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:47.510+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:47.511131+0000) 2024-04-27T04:00:47.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:47.435+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:47.436541+0000) 2024-04-27T04:00:47.988 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:47.725+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:47.725915+0000) 2024-04-27T04:00:48.273 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:48.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:48.273 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:48.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:48.273 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:48.085+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:48.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:48.086+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:48.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:48.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:48.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:48.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:48.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:48.030+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:48.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:48.030+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:48.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:48.176+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:48.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:48.176+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:48.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:48.176+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:48.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:48.176+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:48.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:48.438+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:48.438820+0000) 2024-04-27T04:00:48.635 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:48.512+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:48.513246+0000) 2024-04-27T04:00:48.971 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:00:48.642+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78673 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:00:48.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:48.637+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:48.638607+0000) 2024-04-27T04:00:48.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:48.637+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:48.638749+0000) 2024-04-27T04:00:48.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:48.638+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:48.639282+0000) 2024-04-27T04:00:48.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:48.727+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:48.728176+0000) 2024-04-27T04:00:49.273 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:49.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:49.273 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:49.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:49.273 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:49.208+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:49.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:49.209+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:49.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:48.990+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:49.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:48.990+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:49.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:48.990+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:49.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:48.990+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:49.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:49.119+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:49.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:49.119+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:49.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:49.119+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:49.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:49.120+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:49.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:49.440+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:49.440994+0000) 2024-04-27T04:00:49.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:49.514+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:49.515434+0000) 2024-04-27T04:00:49.990 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:49.732+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:49.732935+0000) 2024-04-27T04:00:50.273 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:50.117+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:50.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:50.117+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:50.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:50.117+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:50.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:50.118+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:50.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:50.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:50.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:50.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:50.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:50.222+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:50.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:50.222+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:50.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:50.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:50.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:50.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:50.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:50.025+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:50.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:50.025+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:50.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:50.444+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:50.445708+0000) 2024-04-27T04:00:50.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:50.517+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:50.518302+0000) 2024-04-27T04:00:50.994 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:50.734+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:50.735197+0000) 2024-04-27T04:00:51.273 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:51.155+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:51.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:51.155+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:51.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:51.155+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:51.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:51.155+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:51.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:51.219+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:51.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:51.219+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:51.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:51.219+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:51.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:51.220+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:51.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:51.066+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:51.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:51.066+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:51.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:51.066+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:51.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:51.066+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:51.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:51.447+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:51.448406+0000) 2024-04-27T04:00:51.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:51.519+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:51.520414+0000) 2024-04-27T04:00:51.995 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:51.736+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:51.737384+0000) 2024-04-27T04:00:52.273 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:52.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:52.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:52.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:52.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:52.188+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:52.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:52.189+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:52.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:52.196+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:52.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:52.196+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:52.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:52.196+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:52.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:52.196+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:52.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:52.067+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:52.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:52.067+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:52.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:52.067+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:52.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:52.068+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:52.525 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:52.449+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:52.450600+0000) 2024-04-27T04:00:52.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:52.524+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:52.525219+0000) 2024-04-27T04:00:52.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:52.738+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:52.739601+0000) 2024-04-27T04:00:53.273 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:53.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:53.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:53.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:53.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:53.186+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:53.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:53.187+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:53.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:53.176+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:53.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:53.176+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:53.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:53.176+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:53.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:53.177+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:53.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:53.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:53.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:53.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:53.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:53.108+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:53.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:53.109+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:53.529 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:53.452+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:53.453784+0000) 2024-04-27T04:00:53.963 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:00:53.651+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78713 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:00:53.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:53.647+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:53.648714+0000) 2024-04-27T04:00:53.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:53.528+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:53.529619+0000) 2024-04-27T04:00:53.964 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:53.646+0000 7f49280c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:53.647096+0000) 2024-04-27T04:00:53.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:53.646+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:53.647492+0000) 2024-04-27T04:00:53.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:53.740+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:53.741771+0000) 2024-04-27T04:00:54.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:54.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:54.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:54.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:54.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:54.180+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:54.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:54.181+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:54.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:54.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:54.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:54.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:54.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:54.113+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:54.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:54.114+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:54.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:54.149+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:54.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:54.149+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:54.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:54.149+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:54.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:54.150+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:54.532 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:54.455+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:54.456452+0000) 2024-04-27T04:00:54.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:54.743+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:54.744076+0000) 2024-04-27T04:00:54.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:54.531+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:54.532419+0000) 2024-04-27T04:00:55.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:55.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:55.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:55.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:55.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:55.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:55.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:55.171+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:55.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:55.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:55.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:55.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:55.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:55.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:55.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:55.072+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:55.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:55.104+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:55.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:55.104+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:55.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:55.104+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:55.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:55.104+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:55.535 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:55.457+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:55.458536+0000) 2024-04-27T04:00:55.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:55.745+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:55.746357+0000) 2024-04-27T04:00:55.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:55.534+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:55.535272+0000) 2024-04-27T04:00:56.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:56.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:56.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:56.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:56.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:56.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:56.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:56.177+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:56.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:56.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:56.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:56.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:56.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:56.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:56.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:56.030+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:56.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:56.088+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:56.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:56.088+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:56.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:56.088+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:56.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:56.088+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:56.537 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:56.460+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:56.461337+0000) 2024-04-27T04:00:56.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:56.748+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:56.749311+0000) 2024-04-27T04:00:56.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:56.536+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:56.537560+0000) 2024-04-27T04:00:57.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:57.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:57.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:57.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:57.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:57.138+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:57.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:57.139+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:57.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:57.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:57.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:57.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:57.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:57.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:57.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:57.037+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:57.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:57.067+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:57.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:57.067+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:57.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:57.067+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:57.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:57.067+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:57.541 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:57.463+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:57.464055+0000) 2024-04-27T04:00:57.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:57.540+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:57.541127+0000) 2024-04-27T04:00:57.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:57.751+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:57.752087+0000) 2024-04-27T04:00:58.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:58.174+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:58.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:58.174+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:58.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:58.174+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:58.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:58.175+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:58.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:58.044+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:58.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:58.044+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:58.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:58.044+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:58.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:58.045+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:58.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:58.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:58.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:58.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:58.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:58.083+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:58.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:58.084+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:58.543 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:58.465+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:58.466370+0000) 2024-04-27T04:00:58.970 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:00:58.660+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78753 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:00:58.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:58.656+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:58.657697+0000) 2024-04-27T04:00:58.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:58.542+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:58.543385+0000) 2024-04-27T04:00:58.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:58.655+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:58.656122+0000) 2024-04-27T04:00:58.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:58.655+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:58.656647+0000) 2024-04-27T04:00:58.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:58.755+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:58.756180+0000) 2024-04-27T04:00:59.274 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:59.173+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:59.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:59.173+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:59.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:59.173+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:00:59.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:59.174+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:59.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:59.098+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:59.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:59.098+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:59.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:59.098+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:00:59.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:59.098+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:59.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:59.054+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:59.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:59.054+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:59.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:59.054+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:00:59.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:59.054+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:00:59.546 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:00:59.467+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:59.468662+0000) 2024-04-27T04:00:59.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:00:59.545+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:59.546625+0000) 2024-04-27T04:00:59.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:00:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:00:59.760+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:00:59.761223+0000) 2024-04-27T04:01:00.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:00.103+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:00.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:00.103+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:00.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:00.103+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:00.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:00.104+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:00.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:00.079+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:00.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:00.079+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:00.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:00.079+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:00.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:00.080+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:00.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:00.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:00.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:00.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:00.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:00.154+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:00.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:00.155+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:00.548 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:00.469+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:00.471406+0000) 2024-04-27T04:01:00.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:00.764+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:00.766362+0000) 2024-04-27T04:01:00.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:00.547+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:00.548841+0000) 2024-04-27T04:01:01.274 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:01.056+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:01.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:01.056+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:01.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:01.056+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:01.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:01.057+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:01.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:01.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:01.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:01.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:01.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:01.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:01.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:01.043+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:01.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:01.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:01.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:01.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:01.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:01.176+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:01.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:01.177+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:01.551 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:01.471+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:01.473583+0000) 2024-04-27T04:01:01.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:01.769+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:01.771450+0000) 2024-04-27T04:01:01.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:01.549+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:01.551171+0000) 2024-04-27T04:01:02.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:02.207+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:02.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:02.207+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:02.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:02.207+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:02.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:02.208+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:02.275 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:02.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:02.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:02.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:02.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:02.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:02.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:02.025+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:02.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:02.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:02.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:02.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:02.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:02.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:02.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:02.090+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:02.553 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:02.474+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:02.475824+0000) 2024-04-27T04:01:02.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:02.551+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:02.553444+0000) 2024-04-27T04:01:02.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:02.774+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:02.776539+0000) 2024-04-27T04:01:03.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:03.226+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:03.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:03.226+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:03.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:03.226+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:03.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:03.227+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:03.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:03.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:03.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:03.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:03.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:03.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:03.276 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:03.067+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:03.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:03.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:03.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:03.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:03.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:03.056+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:03.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:03.057+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:03.555 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:03.477+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:03.479043+0000) 2024-04-27T04:01:03.964 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:01:03.669+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78793 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:01:03.965 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:03.663+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:03.665468+0000) 2024-04-27T04:01:03.965 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:03.554+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:03.555888+0000) 2024-04-27T04:01:03.965 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:03.664+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:03.666785+0000) 2024-04-27T04:01:03.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:03.664+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:03.666221+0000) 2024-04-27T04:01:03.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:03.776+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:03.778776+0000) 2024-04-27T04:01:04.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:04.071+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:04.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:04.071+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:04.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:04.071+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:04.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:04.071+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:04.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:04.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:04.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:04.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:04.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:04.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:04.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:04.063+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:04.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:04.217+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:04.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:04.217+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:04.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:04.217+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:04.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:04.218+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:04.783 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:04.479+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:04.481678+0000) 2024-04-27T04:01:04.784 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:04.558+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:04.560666+0000) 2024-04-27T04:01:05.100 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:04.782+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:04.783825+0000) 2024-04-27T04:01:05.101 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:05.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:05.101 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:05.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:05.101 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:05.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:05.101 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:05.027+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:05.387 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:05.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:05.387 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:05.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:05.387 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:05.111+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:05.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:05.111+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:05.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:05.224+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:05.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:05.224+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:05.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:05.224+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:05.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:05.225+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:05.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:05.483+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:05.484832+0000) 2024-04-27T04:01:05.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:05.563+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:05.564917+0000) 2024-04-27T04:01:06.077 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:05.787+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:05.788827+0000) 2024-04-27T04:01:06.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:06.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:06.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:06.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:06.078 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:06.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:06.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:06.177+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:06.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:06.177+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:06.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:06.177+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:06.369 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:06.178+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:06.369 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:06.076+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:06.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:06.130+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:06.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:06.130+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:06.370 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:06.130+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:06.370 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:06.130+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:06.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:06.487+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:06.489776+0000) 2024-04-27T04:01:06.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:06.565+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:06.567049+0000) 2024-04-27T04:01:07.061 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:06.792+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:06.793913+0000) 2024-04-27T04:01:07.387 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:07.155+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:07.387 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:07.155+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:07.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:07.155+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:07.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:07.156+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:07.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:07.153+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:07.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:07.153+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:07.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:07.153+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:07.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:07.154+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:07.389 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:07.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:07.389 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:07.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:07.389 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:07.059+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:07.389 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:07.061+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:07.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:07.493+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:07.494823+0000) 2024-04-27T04:01:07.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:07.567+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:07.569389+0000) 2024-04-27T04:01:08.057 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:07.797+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:07.799066+0000) 2024-04-27T04:01:08.387 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:08.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:08.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:08.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:08.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:08.109+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:08.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:08.109+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:08.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:08.197+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:08.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:08.197+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:08.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:08.197+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:08.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:08.197+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:08.390 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:08.055+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:08.390 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:08.055+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:08.390 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:08.055+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:08.390 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:08.056+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:08.672 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:08.498+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:08.499823+0000) 2024-04-27T04:01:08.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:08.570+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:08.572617+0000) 2024-04-27T04:01:08.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:08.675+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:08.676946+0000) 2024-04-27T04:01:08.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:08.802+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:08.804187+0000) 2024-04-27T04:01:08.971 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:01:08.678+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78833 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:01:08.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:08.673+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:08.675581+0000) 2024-04-27T04:01:08.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:08.674+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:08.675922+0000) 2024-04-27T04:01:09.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:09.079+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:09.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:09.079+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:09.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:09.079+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:09.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:09.080+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:09.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:09.210+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:09.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:09.210+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:09.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:09.210+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:09.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:09.211+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:09.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:09.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:09.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:09.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:09.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:09.013+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:09.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:09.014+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:09.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:09.502+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:09.504794+0000) 2024-04-27T04:01:09.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:09.575+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:09.577073+0000) 2024-04-27T04:01:09.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:09.806+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:09.808425+0000) 2024-04-27T04:01:10.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:10.070+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:10.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:10.070+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:10.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:10.070+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:10.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:10.071+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:10.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:10.197+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:10.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:10.197+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:10.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:10.197+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:10.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:10.198+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:10.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:10.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:10.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:10.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:10.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:10.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:10.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:10.002+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:10.579 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:10.505+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:10.506864+0000) 2024-04-27T04:01:10.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:10.577+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:10.579454+0000) 2024-04-27T04:01:10.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:10.809+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:10.811548+0000) 2024-04-27T04:01:11.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:11.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:11.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:11.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:11.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:11.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:11.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:11.041+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:11.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:10.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:11.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:10.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:11.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:10.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:11.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:10.978+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:11.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:11.191+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:11.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:11.191+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:11.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:11.191+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:11.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:11.191+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:11.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:11.508+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:11.510120+0000) 2024-04-27T04:01:11.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:11.581+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:11.582292+0000) 2024-04-27T04:01:11.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:11.812+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:11.813879+0000) 2024-04-27T04:01:12.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:12.044+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:12.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:12.044+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:12.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:12.044+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:12.276 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:12.044+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:12.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:11.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:12.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:11.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:12.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:11.974+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:12.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:11.974+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:12.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:12.211+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:12.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:12.211+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:12.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:12.211+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:12.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:12.211+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:12.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:12.511+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:12.512419+0000) 2024-04-27T04:01:12.927 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:12.815+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:12.816188+0000) 2024-04-27T04:01:12.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:12.584+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:12.585753+0000) 2024-04-27T04:01:12.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:12.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:12.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:12.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:12.928 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:12.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:13.201 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:12.927+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:13.202 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:13.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:13.202 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:13.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:13.202 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:13.041+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:13.202 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:13.042+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:13.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:13.212+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:13.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:13.212+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:13.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:13.212+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:13.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:13.212+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:13.809 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:01:13.689+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78873 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:01:13.809 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:13.683+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:13.684676+0000) 2024-04-27T04:01:13.810 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:13.515+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:13.516727+0000) 2024-04-27T04:01:13.810 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:13.684+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:13.684931+0000) 2024-04-27T04:01:13.810 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:13.587+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:13.588261+0000) 2024-04-27T04:01:13.810 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:13.685+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:13.686343+0000) 2024-04-27T04:01:14.062 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:13.818+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:13.819221+0000) 2024-04-27T04:01:14.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:13.955+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:14.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:13.955+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:14.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:13.955+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:14.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:13.956+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:14.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:14.195+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:14.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:14.195+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:14.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:14.195+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:14.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:14.196+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:14.389 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:14.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:14.389 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:14.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:14.389 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:14.062+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:14.389 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:14.063+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:14.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:14.590+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:14.591275+0000) 2024-04-27T04:01:14.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:14.518+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:14.519427+0000) 2024-04-27T04:01:15.076 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:14.823+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:14.824287+0000) 2024-04-27T04:01:15.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:14.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:15.076 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:14.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:15.077 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:14.926+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:15.077 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:14.926+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:15.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:15.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:15.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:15.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:15.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:15.167+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:15.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:15.168+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:15.389 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:15.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:15.389 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:15.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:15.389 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:15.075+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:15.389 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:15.076+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:15.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:15.523+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:15.524329+0000) 2024-04-27T04:01:15.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:15.593+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:15.593905+0000) 2024-04-27T04:01:16.103 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:15.828+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:15.829310+0000) 2024-04-27T04:01:16.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:16.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:16.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:16.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:16.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:16.055+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:16.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:16.057+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:16.104 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:15.960+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:16.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:15.960+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:16.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:15.960+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:16.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:15.961+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:16.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:16.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:16.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:16.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:16.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:16.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:16.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:16.128+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:16.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:16.526+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:16.527233+0000) 2024-04-27T04:01:16.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:16.596+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:16.597111+0000) 2024-04-27T04:01:17.090 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:16.831+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:16.832101+0000) 2024-04-27T04:01:17.090 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:16.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:17.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:16.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:17.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:16.928+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:17.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:16.929+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:17.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:17.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:17.389 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:17.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:17.389 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:17.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:17.389 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:17.090+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:17.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:17.137+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:17.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:17.137+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:17.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:17.137+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:17.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:17.138+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:17.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:17.531+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:17.532361+0000) 2024-04-27T04:01:17.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:17.599+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:17.600200+0000) 2024-04-27T04:01:18.103 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:17.833+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:17.834477+0000) 2024-04-27T04:01:18.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:18.066+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:18.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:18.066+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:18.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:18.066+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:18.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:18.068+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:18.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:17.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:18.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:17.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:18.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:17.950+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:18.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:17.952+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:18.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:18.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:18.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:18.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:18.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:18.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:18.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:18.128+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:18.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:18.533+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:18.534620+0000) 2024-04-27T04:01:18.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:18.602+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:18.603270+0000) 2024-04-27T04:01:18.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:18.695+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:18.696709+0000) 2024-04-27T04:01:18.971 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:01:18.699+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78913 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:01:18.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:18.695+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:18.696541+0000) 2024-04-27T04:01:18.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:18.694+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:18.695609+0000) 2024-04-27T04:01:18.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:18.836+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:18.837466+0000) 2024-04-27T04:01:19.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:19.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:19.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:19.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:19.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:19.127+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:19.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:19.128+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:19.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:18.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:19.277 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:18.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:19.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:18.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:19.278 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:18.977+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:19.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:19.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:19.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:19.105+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:19.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:19.106+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:19.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:19.107+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:19.840 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:19.535+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:19.536791+0000) 2024-04-27T04:01:19.840 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:19.606+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:19.607493+0000) 2024-04-27T04:01:20.104 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:19.839+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:19.840303+0000) 2024-04-27T04:01:20.104 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:19.935+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:20.104 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:19.935+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:20.104 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:19.935+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:20.104 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:19.937+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:20.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:20.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:20.389 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:20.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:20.389 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:20.129+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:20.389 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:20.130+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:20.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:20.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:20.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:20.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:20.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:20.133+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:20.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:20.134+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:20.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:20.538+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:20.538962+0000) 2024-04-27T04:01:20.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:20.609+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:20.610298+0000) 2024-04-27T04:01:21.104 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:20.842+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:20.843336+0000) 2024-04-27T04:01:21.104 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:20.939+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:21.104 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:20.939+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:21.104 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:20.939+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:21.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:20.940+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:21.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:21.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:21.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:21.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:21.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:21.124+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:21.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:21.125+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:21.390 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:21.133+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:21.390 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:21.133+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:21.390 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:21.133+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:21.391 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:21.134+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:21.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:21.613+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:21.614592+0000) 2024-04-27T04:01:21.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:21.542+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:21.543174+0000) 2024-04-27T04:01:22.099 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:21.845+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:21.846267+0000) 2024-04-27T04:01:22.100 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:21.944+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:22.100 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:21.944+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:22.100 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:21.944+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:22.100 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:21.944+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:22.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:22.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:22.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:22.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:22.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:22.099+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:22.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:22.099+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:22.390 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:22.122+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:22.390 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:22.122+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:22.390 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:22.122+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:22.390 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:22.123+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:22.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:22.545+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:22.546219+0000) 2024-04-27T04:01:22.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:22.615+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:22.616731+0000) 2024-04-27T04:01:23.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:23.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:23.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:23.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:23.105 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:23.082+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:23.105 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:23.083+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:23.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:22.942+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:23.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:22.942+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:23.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:22.942+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:23.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:22.942+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:23.106 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:22.848+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:22.848917+0000) 2024-04-27T04:01:23.106 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:23.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:23.106 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:23.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:23.106 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:23.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:23.106 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:23.072+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:23.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:23.547+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:23.548342+0000) 2024-04-27T04:01:23.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:23.618+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:23.618996+0000) 2024-04-27T04:01:23.965 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:01:23.708+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78953 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:01:23.965 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:23.704+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:23.705479+0000) 2024-04-27T04:01:23.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:23.705+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:23.706039+0000) 2024-04-27T04:01:23.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:23.850+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:23.851081+0000) 2024-04-27T04:01:23.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:23.704+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:23.704927+0000) 2024-04-27T04:01:24.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:24.070+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:24.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:24.070+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:24.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:24.070+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:24.221 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:24.071+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:24.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:23.975+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:24.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:23.975+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:24.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:23.975+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:24.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:23.976+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:24.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:24.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:24.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:24.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:24.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:24.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:24.223 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:24.090+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:24.855 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:24.550+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:24.551199+0000) 2024-04-27T04:01:24.855 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:24.620+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:24.621149+0000) 2024-04-27T04:01:25.136 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:25.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:25.136 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:25.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:25.136 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:25.046+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:25.136 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:25.047+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:25.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:24.854+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:24.855339+0000) 2024-04-27T04:01:25.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:25.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:25.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:25.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:25.137 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:25.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:25.137 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:24.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:25.137 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:24.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:25.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:24.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:25.138 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:24.941+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:25.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:25.136+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:25.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:25.622+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:25.623352+0000) 2024-04-27T04:01:25.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:25.552+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:25.553393+0000) 2024-04-27T04:01:26.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:25.858+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:25.859437+0000) 2024-04-27T04:01:26.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:26.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:26.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:26.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:26.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:26.089+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:26.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:26.089+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:26.184 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:25.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:26.184 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:25.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:26.184 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:25.968+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:26.184 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:25.968+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:26.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:26.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:26.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:26.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:26.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:26.182+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:26.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:26.183+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:26.862 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:26.555+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:26.556279+0000) 2024-04-27T04:01:26.862 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:26.625+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:26.626436+0000) 2024-04-27T04:01:27.205 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:26.861+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:26.862322+0000) 2024-04-27T04:01:27.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:27.077+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:27.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:27.077+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:27.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:27.077+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:27.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:27.078+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:27.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:26.958+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:27.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:26.958+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:27.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:26.958+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:27.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:26.959+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:27.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:27.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:27.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:27.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:27.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:27.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:27.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:27.219+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:27.865 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:27.558+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:27.558937+0000) 2024-04-27T04:01:27.865 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:27.627+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:27.628672+0000) 2024-04-27T04:01:28.203 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:27.864+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:27.865273+0000) 2024-04-27T04:01:28.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:28.098+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:28.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:28.098+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:28.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:28.098+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:28.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:28.099+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:28.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:27.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:28.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:27.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:28.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:27.996+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:28.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:27.997+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:28.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:28.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:28.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:28.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:28.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:28.218+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:28.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:28.219+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:28.837 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:01:28.718+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 78993 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:01:28.837 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:28.631+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:28.631912+0000) 2024-04-27T04:01:28.838 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:28.712+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:28.713423+0000) 2024-04-27T04:01:28.838 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:28.713+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:28.713883+0000) 2024-04-27T04:01:28.838 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:28.560+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:28.561090+0000) 2024-04-27T04:01:28.839 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:28.713+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:28.714449+0000) 2024-04-27T04:01:29.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:28.866+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:28.867771+0000) 2024-04-27T04:01:29.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:28.969+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:29.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:28.969+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:29.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:28.969+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:29.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:28.970+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:29.106 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:29.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:29.106 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:29.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:29.106 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:29.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:29.106 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:29.054+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:29.389 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:29.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:29.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:29.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:29.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:29.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:29.390 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:29.254+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:29.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:29.562+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:29.563199+0000) 2024-04-27T04:01:29.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:29.633+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:29.633992+0000) 2024-04-27T04:01:30.165 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:29.871+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:29.872016+0000) 2024-04-27T04:01:30.165 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:30.029+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:30.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:30.029+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:30.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:30.029+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:30.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:30.030+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:30.166 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:29.944+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:30.166 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:29.944+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:30.167 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:29.944+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:30.167 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:29.945+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:30.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:30.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:30.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:30.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:30.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:30.262+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:30.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:30.262+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:30.874 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:30.566+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:30.567385+0000) 2024-04-27T04:01:30.874 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:30.635+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:30.636036+0000) 2024-04-27T04:01:31.203 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:30.873+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:30.874241+0000) 2024-04-27T04:01:31.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:30.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:31.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:30.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:31.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:30.977+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:31.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:30.978+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:31.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:31.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:31.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:31.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:31.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:31.001+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:31.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:31.002+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:31.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:31.238+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:31.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:31.238+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:31.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:31.238+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:31.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:31.239+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:31.877 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:31.568+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:31.569474+0000) 2024-04-27T04:01:31.877 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:31.637+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:31.638099+0000) 2024-04-27T04:01:32.203 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:31.876+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:31.877194+0000) 2024-04-27T04:01:32.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:32.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:32.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:32.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:32.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:32.052+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:32.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:32.052+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:32.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:31.992+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:32.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:31.992+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:32.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:31.992+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:32.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:31.992+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:32.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:32.283+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:32.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:32.283+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:32.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:32.283+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:32.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:32.284+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:32.880 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:32.570+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:32.571754+0000) 2024-04-27T04:01:32.880 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:32.639+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:32.640349+0000) 2024-04-27T04:01:33.203 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:32.879+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:32.880449+0000) 2024-04-27T04:01:33.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:33.015+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:33.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:33.015+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:33.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:33.015+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:33.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:33.016+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:33.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:32.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:33.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:32.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:33.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:32.979+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:33.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:32.980+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:33.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:33.234+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:33.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:33.234+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:33.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:33.234+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:33.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:33.234+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:33.844 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:01:33.726+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79033 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:01:33.845 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:33.723+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:33.724629+0000) 2024-04-27T04:01:33.845 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:33.573+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:33.574065+0000) 2024-04-27T04:01:33.846 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:33.722+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:33.723569+0000) 2024-04-27T04:01:33.846 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:33.641+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:33.642609+0000) 2024-04-27T04:01:33.846 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:33.722+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:33.723387+0000) 2024-04-27T04:01:34.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:33.881+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:33.882649+0000) 2024-04-27T04:01:34.105 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:34.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:34.106 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:34.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:34.106 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:34.010+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:34.106 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:34.011+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:34.106 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:34.014+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:34.106 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:34.014+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:34.106 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:34.014+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:34.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:34.015+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:34.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:34.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:34.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:34.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:34.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:34.248+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:34.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:34.249+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:34.645 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:34.575+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:34.576214+0000) 2024-04-27T04:01:34.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:34.644+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:34.645413+0000) 2024-04-27T04:01:34.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:34.885+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:34.885869+0000) 2024-04-27T04:01:35.390 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:35.048+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:35.390 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:35.048+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:35.391 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:35.048+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:35.391 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:35.049+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:35.391 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:35.238+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:35.391 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:35.238+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:35.391 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:35.238+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:35.391 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:35.239+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:35.392 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:35.038+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:35.392 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:35.038+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:35.392 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:35.038+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:35.392 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:35.039+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:35.649 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:35.579+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:35.580426+0000) 2024-04-27T04:01:35.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:35.887+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:35.887999+0000) 2024-04-27T04:01:35.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:35.648+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:35.649547+0000) 2024-04-27T04:01:36.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:36.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:36.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:36.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:36.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:36.054+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:36.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:36.054+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:36.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:36.029+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:36.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:36.029+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:36.369 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:36.029+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:36.369 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:36.030+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:36.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:36.231+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:36.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:36.231+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:36.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:36.231+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:36.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:36.232+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:36.651 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:36.581+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:36.582519+0000) 2024-04-27T04:01:36.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:36.651+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:36.651827+0000) 2024-04-27T04:01:36.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:36.889+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:36.890072+0000) 2024-04-27T04:01:37.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:37.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:37.279 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:37.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:37.279 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:37.036+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:37.279 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:37.037+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:37.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:37.226+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:37.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:37.226+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:37.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:37.226+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:37.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:37.227+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:37.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:37.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:37.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:37.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:37.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:37.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:37.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:37.023+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:37.892 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:37.584+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:37.585520+0000) 2024-04-27T04:01:37.892 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:37.654+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:37.654822+0000) 2024-04-27T04:01:38.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:38.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:38.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:38.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:38.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:38.007+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:38.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:38.008+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:38.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:37.891+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:37.892202+0000) 2024-04-27T04:01:38.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:38.183+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:38.185 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:38.183+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:38.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:38.067+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:38.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:38.067+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:38.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:38.067+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:38.186 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:38.068+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:38.473 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:38.183+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:38.474 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:38.183+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:38.855 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:01:38.736+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79073 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:01:38.855 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:38.587+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:38.588352+0000) 2024-04-27T04:01:38.855 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:38.731+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:38.732544+0000) 2024-04-27T04:01:38.856 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:38.731+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:38.732512+0000) 2024-04-27T04:01:38.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:38.656+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:38.656920+0000) 2024-04-27T04:01:38.856 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:38.732+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:38.733235+0000) 2024-04-27T04:01:39.106 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:38.894+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:38.895066+0000) 2024-04-27T04:01:39.106 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:39.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:39.107 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:39.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:39.107 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:39.024+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:39.107 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:39.025+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:39.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:39.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:39.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:39.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:39.107 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:39.026+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:39.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:39.027+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:39.391 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:39.137+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:39.391 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:39.137+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:39.391 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:39.137+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:39.391 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:39.137+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:39.659 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:39.590+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:39.591201+0000) 2024-04-27T04:01:39.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:39.658+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:39.659633+0000) 2024-04-27T04:01:39.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:39.896+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:39.897611+0000) 2024-04-27T04:01:40.279 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:40.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:40.279 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:40.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:40.279 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:40.000+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:40.279 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:40.001+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:40.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:40.070+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:40.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:40.070+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:40.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:40.070+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:40.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:40.071+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:40.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:40.115+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:40.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:40.115+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:40.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:40.115+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:40.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:40.116+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:40.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:40.593+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:40.594218+0000) 2024-04-27T04:01:40.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:40.661+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:40.662597+0000) 2024-04-27T04:01:41.204 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:41.032+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:41.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:41.032+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:41.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:41.032+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:41.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:41.033+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:41.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:41.085+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:41.205 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:41.085+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:41.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:41.085+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:41.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:41.086+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:41.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:40.899+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:40.899970+0000) 2024-04-27T04:01:41.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:41.136+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:41.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:41.136+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:41.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:41.136+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:41.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:41.137+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:41.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:41.596+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:41.597584+0000) 2024-04-27T04:01:41.904 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:41.664+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:41.664859+0000) 2024-04-27T04:01:42.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:42.058+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:42.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:42.058+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:42.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:42.058+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:42.206 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:42.059+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:42.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:41.903+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:41.904264+0000) 2024-04-27T04:01:42.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:42.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:42.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:42.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:42.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:42.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:42.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:42.106+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:42.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:42.130+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:42.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:42.130+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:42.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:42.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:42.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:42.131+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:42.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:42.601+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:42.601893+0000) 2024-04-27T04:01:42.907 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:42.666+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:42.667162+0000) 2024-04-27T04:01:43.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:43.106+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:43.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:43.106+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:43.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:43.106+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:43.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:43.107+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:43.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:42.905+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:42.906372+0000) 2024-04-27T04:01:43.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:43.059+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:43.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:43.059+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:43.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:43.059+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:43.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:43.059+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:43.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:43.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:43.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:43.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:43.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:43.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:43.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:43.093+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:43.866 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:01:43.745+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79113 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:01:43.867 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:43.741+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:43.742481+0000) 2024-04-27T04:01:43.867 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:43.603+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:43.604034+0000) 2024-04-27T04:01:43.867 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:43.741+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:43.742321+0000) 2024-04-27T04:01:43.868 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:43.668+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:43.669404+0000) 2024-04-27T04:01:43.868 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:43.741+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:43.742223+0000) 2024-04-27T04:01:44.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:44.059+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:44.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:44.059+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:44.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:44.059+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:44.205 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:44.061+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:44.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:43.909+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:43.909843+0000) 2024-04-27T04:01:44.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:44.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:44.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:44.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:44.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:44.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:44.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:44.058+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:44.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:44.076+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:44.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:44.076+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:44.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:44.076+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:44.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:44.076+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:44.743 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:44.605+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:44.606161+0000) 2024-04-27T04:01:44.744 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:44.670+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:44.671701+0000) 2024-04-27T04:01:45.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:44.911+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:44.912177+0000) 2024-04-27T04:01:45.391 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:45.096+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:45.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:45.096+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:45.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:45.096+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:45.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:45.097+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:45.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:45.088+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:45.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:45.088+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:45.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:45.088+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:45.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:45.089+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:45.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:45.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:45.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:45.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:45.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:45.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:45.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:45.082+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:45.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:45.607+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:45.608456+0000) 2024-04-27T04:01:45.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:45.914+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:45.915475+0000) 2024-04-27T04:01:45.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:45.673+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:45.673974+0000) 2024-04-27T04:01:46.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:46.137+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:46.367 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:46.137+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:46.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:46.137+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:46.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:46.138+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:46.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:46.073+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:46.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:46.073+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:46.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:46.073+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:46.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:46.074+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:46.369 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:46.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:46.369 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:46.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:46.369 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:46.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:46.369 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:46.063+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:46.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:46.611+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:46.612678+0000) 2024-04-27T04:01:46.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:46.917+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:46.917896+0000) 2024-04-27T04:01:46.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:46.677+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:46.678667+0000) 2024-04-27T04:01:47.391 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:47.109+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:47.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:47.109+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:47.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:47.109+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:47.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:47.110+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:47.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:47.177+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:47.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:47.177+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:47.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:47.177+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:47.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:47.179+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:47.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:47.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:47.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:47.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:47.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:47.062+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:47.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:47.064+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:47.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:47.616+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:47.617790+0000) 2024-04-27T04:01:47.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:47.919+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:47.920254+0000) 2024-04-27T04:01:47.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:47.681+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:47.682046+0000) 2024-04-27T04:01:48.391 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:48.094+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:48.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:48.094+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:48.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:48.094+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:48.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:48.095+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:48.392 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:48.106+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:48.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:48.106+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:48.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:48.106+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:48.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:48.107+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:48.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:48.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:48.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:48.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:48.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:48.135+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:48.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:48.137+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:48.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:48.619+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:48.619950+0000) 2024-04-27T04:01:48.970 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:01:48.756+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79153 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:01:48.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:48.749+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:48.750397+0000) 2024-04-27T04:01:48.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:48.750+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:48.751440+0000) 2024-04-27T04:01:48.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:48.923+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:48.924575+0000) 2024-04-27T04:01:48.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:48.684+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:48.685172+0000) 2024-04-27T04:01:48.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:48.750+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:48.751415+0000) 2024-04-27T04:01:49.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:49.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:49.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:49.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:49.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:49.170+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:49.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:49.171+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:49.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:49.115+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:49.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:49.115+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:49.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:49.115+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:49.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:49.115+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:49.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:49.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:49.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:49.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:49.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:49.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:49.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:49.151+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:49.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:49.621+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:49.622645+0000) 2024-04-27T04:01:49.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:49.686+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:49.687601+0000) 2024-04-27T04:01:49.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:49.926+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:49.927567+0000) 2024-04-27T04:01:50.392 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:50.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:50.392 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:50.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:50.392 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:50.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:50.392 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:50.163+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:50.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:50.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:50.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:50.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:50.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:50.090+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:50.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:50.091+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:50.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:50.200+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:50.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:50.200+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:50.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:50.200+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:50.394 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:50.201+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:50.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:50.624+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:50.625768+0000) 2024-04-27T04:01:50.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:50.690+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:50.691743+0000) 2024-04-27T04:01:50.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:50.929+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:50.929826+0000) 2024-04-27T04:01:51.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:51.198+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:51.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:51.198+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:51.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:51.198+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:51.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:51.199+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:51.392 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:51.115+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:51.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:51.115+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:51.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:51.115+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:51.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:51.116+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:51.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:51.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:51.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:51.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:51.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:51.135+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:51.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:51.135+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:51.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:51.627+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:51.628487+0000) 2024-04-27T04:01:51.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:51.693+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:51.694197+0000) 2024-04-27T04:01:51.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:51.931+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:51.932591+0000) 2024-04-27T04:01:52.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:52.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:52.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:52.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:52.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:52.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:52.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:52.169+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:52.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:52.214+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:52.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:52.214+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:52.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:52.214+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:52.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:52.215+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:52.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:52.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:52.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:52.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:52.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:52.165+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:52.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:52.166+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:52.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:52.630+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:52.631589+0000) 2024-04-27T04:01:52.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:52.935+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:52.935818+0000) 2024-04-27T04:01:52.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:52.698+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:52.699323+0000) 2024-04-27T04:01:53.392 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:53.213+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:53.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:53.213+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:53.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:53.213+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:53.393 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:53.214+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:53.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:53.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:53.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:53.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:53.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:53.132+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:53.394 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:53.133+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:53.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:53.187+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:53.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:53.187+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:53.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:53.187+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:53.394 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:53.187+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:53.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:53.633+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:53.633929+0000) 2024-04-27T04:01:53.967 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:01:53.765+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79194 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:01:53.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:53.760+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:53.761594+0000) 2024-04-27T04:01:53.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:53.761+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:53.762730+0000) 2024-04-27T04:01:53.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:53.937+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:53.938685+0000) 2024-04-27T04:01:53.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:53.703+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:53.704398+0000) 2024-04-27T04:01:53.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:53.760+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:53.761772+0000) 2024-04-27T04:01:54.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:54.153+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:54.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:54.153+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:54.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:54.153+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:54.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:54.153+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:54.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:54.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:54.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:54.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:54.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:54.162+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:54.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:54.163+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:54.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:54.247+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:54.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:54.247+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:54.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:54.247+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:54.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:54.247+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:54.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:54.708+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:54.709495+0000) 2024-04-27T04:01:54.942 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:54.635+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:54.636013+0000) 2024-04-27T04:01:55.206 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:54.941+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:54.941945+0000) 2024-04-27T04:01:55.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:55.163+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:55.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:55.163+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:55.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:55.163+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:55.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:55.163+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:55.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:55.158+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:55.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:55.158+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:55.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:55.158+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:55.208 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:55.158+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:55.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:55.270+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:55.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:55.270+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:55.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:55.270+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:55.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:55.271+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:55.946 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:55.712+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:55.713768+0000) 2024-04-27T04:01:55.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:55.637+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:55.638177+0000) 2024-04-27T04:01:56.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:56.163+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:56.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:56.163+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:56.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:56.163+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:56.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:56.163+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:56.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:55.945+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:55.946153+0000) 2024-04-27T04:01:56.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:56.192+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:56.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:56.192+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:56.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:56.192+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:56.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:56.193+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:56.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:56.295+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:56.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:56.295+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:56.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:56.295+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:56.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:56.295+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:56.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:56.640+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:56.640852+0000) 2024-04-27T04:01:56.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:56.715+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:56.715955+0000) 2024-04-27T04:01:57.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:57.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:57.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:57.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:57.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:57.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:57.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:57.161+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:57.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:56.949+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:56.950000+0000) 2024-04-27T04:01:57.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:57.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:57.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:57.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:57.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:57.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:57.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:57.169+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:57.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:57.316+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:57.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:57.316+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:57.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:57.316+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:57.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:57.317+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:57.951 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:57.643+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:57.644166+0000) 2024-04-27T04:01:57.952 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:57.720+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:57.721048+0000) 2024-04-27T04:01:58.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:58.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:58.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:58.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:58.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:58.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:58.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:58.175+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:58.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:57.951+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:57.952238+0000) 2024-04-27T04:01:58.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:58.142+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:58.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:58.142+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:58.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:58.142+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:58.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:58.143+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:58.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:58.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:58.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:58.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:58.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:58.367+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:58.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:58.368+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:58.954 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:01:58.774+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79233 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:01:58.954 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:58.770+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:58.770848+0000) 2024-04-27T04:01:58.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:58.646+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:58.647189+0000) 2024-04-27T04:01:58.955 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:58.770+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:58.770896+0000) 2024-04-27T04:01:58.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:58.722+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:58.723399+0000) 2024-04-27T04:01:58.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:58.769+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:58.770723+0000) 2024-04-27T04:01:59.206 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:59.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:59.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:59.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:59.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:59.161+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:01:59.207 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:59.162+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:59.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:58.953+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:58.954529+0000) 2024-04-27T04:01:59.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:59.163+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:59.207 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:59.163+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:59.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:59.163+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:01:59.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:59.163+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:59.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:59.341+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:59.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:59.341+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:59.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:59.341+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:01:59.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:59.342+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:01:59.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:01:59.649+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:59.650235+0000) 2024-04-27T04:01:59.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:01:59.724+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:59.725585+0000) 2024-04-27T04:02:00.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:00.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:00.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:00.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:00.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:00.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:00.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:00.152+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:00.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:01:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:01:59.957+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:01:59.958772+0000) 2024-04-27T04:02:00.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:00.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:00.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:00.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:00.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:00.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:00.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:00.169+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:00.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:00.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:00.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:00.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:00.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:00.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:00.654 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:00.324+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:00.962 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:00.653+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:00.654295+0000) 2024-04-27T04:02:00.963 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:00.727+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:00.728289+0000) 2024-04-27T04:02:01.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:00.962+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:00.963010+0000) 2024-04-27T04:02:01.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:01.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:01.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:01.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:01.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:01.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:01.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:01.206+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:01.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:01.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:01.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:01.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:01.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:01.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:01.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:01.157+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:01.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:01.317+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:01.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:01.317+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:01.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:01.317+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:01.658 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:01.318+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:01.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:01.657+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:01.658556+0000) 2024-04-27T04:02:01.967 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:01.730+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:01.732592+0000) 2024-04-27T04:02:02.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:01.965+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:01.967271+0000) 2024-04-27T04:02:02.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:02.177+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:02.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:02.177+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:02.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:02.177+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:02.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:02.178+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:02.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:02.199+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:02.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:02.199+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:02.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:02.199+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:02.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:02.199+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:02.660 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:02.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:02.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:02.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:02.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:02.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:02.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:02.345+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:02.661 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:02.659+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:02.660946+0000) 2024-04-27T04:02:02.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:02.735+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:02.736986+0000) 2024-04-27T04:02:03.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:02.969+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:02.971301+0000) 2024-04-27T04:02:03.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:03.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:03.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:03.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:03.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:03.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:03.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:03.205+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:03.282 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:03.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:03.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:03.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:03.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:03.175+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:03.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:03.176+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:03.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:03.346+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:03.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:03.347+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:03.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:03.347+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:03.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:03.347+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:03.967 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:02:03.783+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79274 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:02:03.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:03.778+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:03.779967+0000) 2024-04-27T04:02:03.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:03.661+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:03.663792+0000) 2024-04-27T04:02:03.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:03.779+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:03.780830+0000) 2024-04-27T04:02:03.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:03.740+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:03.742058+0000) 2024-04-27T04:02:03.968 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:03.780+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:03.782164+0000) 2024-04-27T04:02:04.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:04.155+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:04.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:04.155+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:04.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:04.155+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:04.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:04.157+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:04.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:03.973+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:03.975592+0000) 2024-04-27T04:02:04.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:04.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:04.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:04.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:04.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:04.214+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:04.222 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:04.215+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:04.541 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:04.384+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:04.542 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:04.384+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:04.542 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:04.384+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:04.542 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:04.385+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:04.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:04.665+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:04.666821+0000) 2024-04-27T04:02:04.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:04.743+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:04.745175+0000) 2024-04-27T04:02:05.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:04.978+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:04.980352+0000) 2024-04-27T04:02:05.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:05.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:05.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:05.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:05.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:05.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:05.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:05.226+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:05.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:05.107+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:05.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:05.107+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:05.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:05.107+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:05.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:05.109+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:05.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:05.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:05.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:05.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:05.672 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:05.420+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:05.672 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:05.421+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:05.672 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:05.669+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:05.671569+0000) 2024-04-27T04:02:05.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:05.746+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:05.747844+0000) 2024-04-27T04:02:06.238 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:05.982+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:05.984590+0000) 2024-04-27T04:02:06.238 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:06.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:06.239 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:06.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:06.239 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:06.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:06.239 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:06.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:06.239 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:06.082+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:06.674 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:06.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:06.674 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:06.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:06.674 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:06.397+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:06.674 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:06.398+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:06.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:06.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:06.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:06.236+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:06.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:06.237+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:06.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:06.748+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:06.749986+0000) 2024-04-27T04:02:06.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:06.672+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:06.674514+0000) 2024-04-27T04:02:07.255 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:06.987+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:06.989362+0000) 2024-04-27T04:02:07.255 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:07.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:07.255 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:07.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:07.255 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:07.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:07.255 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:07.099+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:07.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:07.372+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:07.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:07.372+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:07.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:07.372+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:07.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:07.373+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:07.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:07.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:07.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:07.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:07.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:07.253+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:07.680 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:07.254+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:07.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:07.677+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:07.679000+0000) 2024-04-27T04:02:07.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:07.751+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:07.753221+0000) 2024-04-27T04:02:08.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:07.989+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:07.991587+0000) 2024-04-27T04:02:08.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:08.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:08.280 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:08.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:08.281 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:08.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:08.281 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:08.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:08.281 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:08.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:08.281 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:08.092+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:08.281 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:08.093+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:08.680 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:08.279+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:08.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:08.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:08.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:08.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:08.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:08.353+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:08.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:08.354+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:08.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:08.788+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:08.790432+0000) 2024-04-27T04:02:08.971 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:02:08.792+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79313 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:02:08.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:08.679+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:08.681223+0000) 2024-04-27T04:02:08.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:08.788+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:08.790089+0000) 2024-04-27T04:02:08.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:08.754+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:08.756007+0000) 2024-04-27T04:02:08.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:08.787+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:08.789527+0000) 2024-04-27T04:02:09.248 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:08.991+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:08.993800+0000) 2024-04-27T04:02:09.248 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:09.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:09.248 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:09.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:09.249 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:09.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:09.249 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:09.097+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:09.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:09.327+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:09.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:09.327+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:09.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:09.327+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:09.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:09.327+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:09.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:09.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:09.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:09.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:09.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:09.246+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:09.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:09.247+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:09.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:09.683+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:09.685409+0000) 2024-04-27T04:02:09.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:09.756+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:09.758239+0000) 2024-04-27T04:02:10.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:09.995+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:09.997022+0000) 2024-04-27T04:02:10.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:10.116+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:10.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:10.116+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:10.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:10.116+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:10.280 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:10.117+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:10.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:10.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:10.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:10.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:10.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:10.312+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:10.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:10.313+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:10.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:10.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:10.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:10.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:10.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:10.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:10.687 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:10.278+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:10.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:10.686+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:10.687816+0000) 2024-04-27T04:02:10.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:10.760+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:10.762624+0000) 2024-04-27T04:02:11.256 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:10.997+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:10.999739+0000) 2024-04-27T04:02:11.256 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:11.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:11.256 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:11.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:11.256 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:11.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:11.256 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:11.157+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:11.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:11.330+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:11.692 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:11.330+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:11.692 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:11.330+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:11.692 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:11.331+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:11.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:11.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:11.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:11.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:11.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:11.254+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:11.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:11.255+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:11.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:11.690+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:11.692032+0000) 2024-04-27T04:02:11.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:11.762+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:11.764788+0000) 2024-04-27T04:02:12.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:12.000+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:12.001934+0000) 2024-04-27T04:02:12.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:12.178+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:12.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:12.178+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:12.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:12.178+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:12.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:12.179+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:12.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:12.299+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:12.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:12.299+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:12.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:12.299+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:12.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:12.301+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:12.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:12.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:12.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:12.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:12.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:12.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:12.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:12.294+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:12.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:12.693+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:12.694544+0000) 2024-04-27T04:02:12.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:12.765+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:12.767499+0000) 2024-04-27T04:02:13.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:13.004+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:13.005198+0000) 2024-04-27T04:02:13.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:13.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:13.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:13.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:13.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:13.200+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:13.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:13.200+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:13.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:13.319+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:13.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:13.319+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:13.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:13.319+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:13.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:13.319+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:13.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:13.332+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:13.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:13.332+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:13.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:13.332+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:13.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:13.332+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:13.968 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:02:13.802+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79354 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:02:13.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:13.696+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:13.698077+0000) 2024-04-27T04:02:13.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:13.798+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:13.798919+0000) 2024-04-27T04:02:13.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:13.769+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:13.770305+0000) 2024-04-27T04:02:13.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:13.798+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:13.799425+0000) 2024-04-27T04:02:13.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:13.798+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:13.799196+0000) 2024-04-27T04:02:14.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:14.006+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:14.007582+0000) 2024-04-27T04:02:14.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:14.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:14.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:14.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:14.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:14.160+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:14.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:14.161+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:14.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:14.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:14.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:14.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:14.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:14.321+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:14.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:14.321+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:14.701 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:14.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:14.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:14.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:14.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:14.320+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:14.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:14.321+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:14.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:14.771+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:14.772607+0000) 2024-04-27T04:02:14.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:14.700+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:14.701315+0000) 2024-04-27T04:02:15.282 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:15.008+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:15.009673+0000) 2024-04-27T04:02:15.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:15.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:15.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:15.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:15.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:15.150+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:15.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:15.151+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:15.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:15.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:15.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:15.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:15.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:15.354+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:15.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:15.355+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:15.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:15.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:15.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:15.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:15.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:15.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:15.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:15.344+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:15.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:15.702+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:15.703492+0000) 2024-04-27T04:02:15.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:15.776+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:15.776868+0000) 2024-04-27T04:02:16.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:16.011+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:16.011989+0000) 2024-04-27T04:02:16.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:16.125+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:16.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:16.125+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:16.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:16.125+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:16.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:16.126+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:16.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:16.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:16.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:16.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:16.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:16.334+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:16.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:16.334+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:16.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:16.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:16.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:16.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:16.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:16.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:16.708 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:16.344+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:16.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:16.778+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:16.779143+0000) 2024-04-27T04:02:16.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:16.705+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:16.705916+0000) 2024-04-27T04:02:17.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:17.014+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:17.014802+0000) 2024-04-27T04:02:17.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:17.174+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:17.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:17.174+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:17.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:17.174+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:17.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:17.174+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:17.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:17.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:17.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:17.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:17.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:17.349+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:17.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:17.350+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:17.708 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:17.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:17.708 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:17.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:17.709 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:17.366+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:17.709 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:17.367+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:17.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:17.780+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:17.781393+0000) 2024-04-27T04:02:17.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:17.707+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:17.708360+0000) 2024-04-27T04:02:18.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:18.016+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:18.016995+0000) 2024-04-27T04:02:18.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:18.176+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:18.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:18.176+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:18.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:18.176+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:18.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:18.177+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:18.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:18.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:18.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:18.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:18.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:18.373+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:18.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:18.374+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:18.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:18.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:18.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:18.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:18.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:18.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:18.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:18.350+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:18.970 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:02:18.812+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79393 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:02:18.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:18.807+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:18.807944+0000) 2024-04-27T04:02:18.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:18.710+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:18.711588+0000) 2024-04-27T04:02:18.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:18.807+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:18.807942+0000) 2024-04-27T04:02:18.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:18.782+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:18.783547+0000) 2024-04-27T04:02:18.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:18.807+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:18.808313+0000) 2024-04-27T04:02:19.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:19.018+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:19.019341+0000) 2024-04-27T04:02:19.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:19.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:19.283 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:19.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:19.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:19.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:19.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:19.130+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:19.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:19.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:19.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:19.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:19.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:19.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:19.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:19.397+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:19.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:19.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:19.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:19.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:19.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:19.397+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:19.714 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:19.398+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:19.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:19.712+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:19.713705+0000) 2024-04-27T04:02:19.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:19.784+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:19.785634+0000) 2024-04-27T04:02:20.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:20.021+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:20.022457+0000) 2024-04-27T04:02:20.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:20.176+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:20.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:20.176+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:20.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:20.176+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:20.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:20.177+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:20.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:20.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:20.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:20.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:20.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:20.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:20.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:20.415+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:20.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:20.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:20.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:20.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:20.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:20.391+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:20.718 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:20.392+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:20.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:20.717+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:20.717858+0000) 2024-04-27T04:02:20.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:20.789+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:20.790309+0000) 2024-04-27T04:02:21.283 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:21.024+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:21.025393+0000) 2024-04-27T04:02:21.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:21.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:21.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:21.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:21.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:21.146+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:21.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:21.146+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:21.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:21.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:21.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:21.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:21.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:21.381+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:21.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:21.382+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:21.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:21.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:21.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:21.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:21.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:21.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:21.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:21.460+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:22.028 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:21.720+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:21.721670+0000) 2024-04-27T04:02:22.028 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:21.792+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:21.793255+0000) 2024-04-27T04:02:22.284 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:22.027+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:22.028729+0000) 2024-04-27T04:02:22.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:22.111+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:22.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:22.111+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:22.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:22.111+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:22.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:22.112+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:22.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:22.423+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:22.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:22.423+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:22.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:22.423+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:22.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:22.424+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:22.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:22.403+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:22.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:22.403+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:22.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:22.403+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:22.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:22.403+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:23.032 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:22.724+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:22.724823+0000) 2024-04-27T04:02:23.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:22.795+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:22.795906+0000) 2024-04-27T04:02:23.284 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:23.032+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:23.033051+0000) 2024-04-27T04:02:23.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:23.127+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:23.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:23.127+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:23.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:23.127+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:23.284 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:23.128+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:23.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:23.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:23.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:23.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:23.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:23.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:23.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:23.455+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:23.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:23.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:23.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:23.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:23.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:23.410+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:23.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:23.410+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:24.035 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:02:23.820+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79434 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:02:24.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:23.726+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:23.727490+0000) 2024-04-27T04:02:24.036 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:23.816+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:23.817035+0000) 2024-04-27T04:02:24.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:23.817+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:23.818447+0000) 2024-04-27T04:02:24.037 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:23.799+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:23.800359+0000) 2024-04-27T04:02:24.037 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:23.817+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:23.817997+0000) 2024-04-27T04:02:24.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:24.035+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:24.035909+0000) 2024-04-27T04:02:24.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:24.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:24.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:24.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:24.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:24.129+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:24.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:24.130+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:24.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:24.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:24.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:24.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:24.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:24.445+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:24.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:24.446+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:24.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:24.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:24.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:24.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:24.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:24.446+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:24.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:24.446+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:25.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:24.801+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:24.802518+0000) 2024-04-27T04:02:25.038 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:24.729+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:24.730510+0000) 2024-04-27T04:02:25.395 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:25.038+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:25.038885+0000) 2024-04-27T04:02:25.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:25.116+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:25.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:25.116+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:25.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:25.116+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:25.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:25.117+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:25.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:25.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:25.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:25.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:25.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:25.401+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:25.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:25.402+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:25.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:25.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:25.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:25.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:25.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:25.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:25.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:25.477+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:26.041 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:25.732+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:25.733320+0000) 2024-04-27T04:02:26.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:25.804+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:25.805359+0000) 2024-04-27T04:02:26.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:26.040+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:26.041740+0000) 2024-04-27T04:02:26.367 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:26.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:26.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:26.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:26.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:26.084+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:26.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:26.085+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:26.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:26.408+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:26.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:26.408+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:26.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:26.408+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:26.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:26.409+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:26.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:26.457+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:26.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:26.457+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:26.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:26.457+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:26.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:26.458+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:27.044 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:26.734+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:26.735622+0000) 2024-04-27T04:02:27.044 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:26.808+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:26.808923+0000) 2024-04-27T04:02:27.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:27.044+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:27.044978+0000) 2024-04-27T04:02:27.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:27.101+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:27.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:27.101+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:27.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:27.101+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:27.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:27.102+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:27.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:27.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:27.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:27.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:27.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:27.450+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:27.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:27.451+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:27.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:27.434+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:27.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:27.434+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:27.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:27.434+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:27.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:27.435+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:28.047 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:27.737+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:27.738760+0000) 2024-04-27T04:02:28.047 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:27.810+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:27.811066+0000) 2024-04-27T04:02:28.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:28.046+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:28.047448+0000) 2024-04-27T04:02:28.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:28.135+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:28.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:28.135+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:28.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:28.135+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:28.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:28.136+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:28.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:28.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:28.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:28.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:28.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:28.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:28.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:28.448+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:28.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:28.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:28.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:28.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:28.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:28.415+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:28.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:28.415+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:29.050 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:02:28.830+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79473 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:02:29.051 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:28.827+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:28.828177+0000) 2024-04-27T04:02:29.051 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:28.812+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:28.813759+0000) 2024-04-27T04:02:29.051 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:28.825+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:28.826044+0000) 2024-04-27T04:02:29.052 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:28.740+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:28.741603+0000) 2024-04-27T04:02:29.052 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:28.825+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:28.826354+0000) 2024-04-27T04:02:29.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:29.050+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:29.051049+0000) 2024-04-27T04:02:29.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:29.123+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:29.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:29.123+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:29.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:29.123+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:29.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:29.123+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:29.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:29.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:29.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:29.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:29.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:29.463+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:29.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:29.464+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:29.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:29.461+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:29.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:29.461+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:29.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:29.461+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:29.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:29.461+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:30.053 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:29.744+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:29.744845+0000) 2024-04-27T04:02:30.054 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:29.816+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:29.816973+0000) 2024-04-27T04:02:30.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:30.053+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:30.054110+0000) 2024-04-27T04:02:30.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:30.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:30.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:30.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:30.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:30.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:30.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:30.144+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:30.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:30.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:30.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:30.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:30.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:30.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:30.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:30.415+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:30.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:30.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:30.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:30.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:30.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:30.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:30.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:30.479+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:31.056 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:30.820+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:30.821717+0000) 2024-04-27T04:02:31.057 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:30.746+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:30.747067+0000) 2024-04-27T04:02:31.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:31.056+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:31.057244+0000) 2024-04-27T04:02:31.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:31.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:31.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:31.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:31.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:31.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:31.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:31.143+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:31.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:31.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:31.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:31.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:31.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:31.451+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:31.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:31.451+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:31.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:31.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:31.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:31.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:31.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:31.500+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:31.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:31.501+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:32.059 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:31.748+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:31.749185+0000) 2024-04-27T04:02:32.059 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:31.823+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:31.824029+0000) 2024-04-27T04:02:32.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:32.058+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:32.059561+0000) 2024-04-27T04:02:32.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:32.181+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:32.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:32.181+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:32.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:32.181+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:32.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:32.182+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:32.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:32.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:32.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:32.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:32.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:32.483+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:32.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:32.483+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:32.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:32.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:32.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:32.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:32.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:32.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:32.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:32.526+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:33.063 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:32.752+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:32.753440+0000) 2024-04-27T04:02:33.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:32.825+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:32.826701+0000) 2024-04-27T04:02:33.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:33.062+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:33.063771+0000) 2024-04-27T04:02:33.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:33.220+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:33.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:33.220+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:33.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:33.220+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:33.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:33.221+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:33.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:33.519+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:33.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:33.519+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:33.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:33.519+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:33.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:33.519+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:33.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:33.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:33.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:33.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:33.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:33.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:33.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:33.529+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:34.067 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:02:33.840+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79514 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:02:34.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:33.754+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:33.755609+0000) 2024-04-27T04:02:34.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:33.833+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:33.834767+0000) 2024-04-27T04:02:34.068 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:33.836+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:33.837239+0000) 2024-04-27T04:02:34.069 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:33.835+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:33.836011+0000) 2024-04-27T04:02:34.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:34.067+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:34.068025+0000) 2024-04-27T04:02:34.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:34.246+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:34.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:34.246+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:34.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:34.246+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:34.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:34.247+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:34.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:34.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:34.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:34.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:34.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:34.548+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:34.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:34.549+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:34.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:34.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:34.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:34.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:34.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:34.499+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:34.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:34.500+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:35.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:34.759+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:34.759841+0000) 2024-04-27T04:02:35.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:35.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:35.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:35.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:35.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:35.273+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:35.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:35.275+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:35.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:35.069+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:35.070583+0000) 2024-04-27T04:02:35.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:35.541+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:35.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:35.541+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:35.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:35.541+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:35.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:35.542+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:35.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:35.596+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:35.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:35.596+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:35.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:35.596+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:35.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:35.598+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:36.073 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:35.761+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:35.761957+0000) 2024-04-27T04:02:36.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:36.072+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:36.073519+0000) 2024-04-27T04:02:36.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:36.234+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:36.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:36.234+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:36.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:36.234+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:36.369 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:36.235+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:36.648 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:36.511+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:36.648 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:36.511+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:36.648 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:36.511+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:36.648 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:36.512+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:36.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:36.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:36.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:36.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:36.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:36.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:36.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:36.648+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:36.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:36.763+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:36.764007+0000) 2024-04-27T04:02:37.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:37.074+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:37.075712+0000) 2024-04-27T04:02:37.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:37.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:37.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:37.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:37.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:37.247+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:37.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:37.248+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:37.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:37.600+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:37.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:37.600+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:37.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:37.600+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:37.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:37.601+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:37.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:37.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:37.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:37.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:37.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:37.467+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:37.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:37.467+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:38.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:37.765+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:37.766111+0000) 2024-04-27T04:02:38.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:38.077+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:38.077992+0000) 2024-04-27T04:02:38.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:38.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:38.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:38.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:38.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:38.211+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:38.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:38.212+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:38.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:38.484+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:38.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:38.484+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:38.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:38.484+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:38.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:38.484+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:38.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:38.610+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:38.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:38.610+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:38.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:38.610+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:38.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:38.612+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:39.080 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:02:38.850+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79553 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:02:39.080 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:38.770+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:38.771131+0000) 2024-04-27T04:02:39.080 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:38.846+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:38.847417+0000) 2024-04-27T04:02:39.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:38.846+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:38.846934+0000) 2024-04-27T04:02:39.081 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:38.845+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:38.846463+0000) 2024-04-27T04:02:39.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:39.079+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:39.080327+0000) 2024-04-27T04:02:39.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:39.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:39.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:39.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:39.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:39.239+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:39.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:39.240+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:39.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:39.610+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:39.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:39.610+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:39.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:39.610+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:39.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:39.611+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:39.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:39.506+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:39.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:39.506+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:39.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:39.506+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:39.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:39.507+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:40.083 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:39.775+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:39.776265+0000) 2024-04-27T04:02:40.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:40.082+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:40.083717+0000) 2024-04-27T04:02:40.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:40.235+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:40.397 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:40.235+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:40.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:40.235+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:40.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:40.236+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:40.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:40.580+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:40.581693+0000) 2024-04-27T04:02:40.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:40.466+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:40.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:40.466+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:40.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:40.466+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:40.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:40.467+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:40.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:40.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:40.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:40.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:40.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:40.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:40.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:40.583+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:41.086 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:40.780+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:40.780878+0000) 2024-04-27T04:02:41.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:41.085+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:41.086722+0000) 2024-04-27T04:02:41.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:41.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:41.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:41.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:41.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:41.202+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:41.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:41.203+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:41.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:41.585+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:41.586660+0000) 2024-04-27T04:02:41.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:41.469+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:41.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:41.469+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:41.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:41.469+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:41.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:41.469+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:41.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:41.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:41.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:41.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:41.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:41.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:41.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:41.609+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:42.089 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:41.784+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:41.785093+0000) 2024-04-27T04:02:42.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:42.089+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:42.090100+0000) 2024-04-27T04:02:42.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:42.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:42.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:42.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:42.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:42.156+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:42.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:42.157+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:42.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:42.602+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:42.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:42.602+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:42.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:42.602+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:42.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:42.604+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:42.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:42.498+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:42.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:42.498+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:42.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:42.498+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:42.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:42.498+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:42.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:42.589+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:42.589884+0000) 2024-04-27T04:02:43.092 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:42.786+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:42.787195+0000) 2024-04-27T04:02:43.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:43.091+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:43.092453+0000) 2024-04-27T04:02:43.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:43.164+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:43.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:43.164+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:43.399 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:43.164+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:43.399 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:43.165+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:43.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:43.591+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:43.592775+0000) 2024-04-27T04:02:43.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:43.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:43.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:43.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:43.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:43.488+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:43.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:43.489+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:43.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:43.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:43.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:43.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:43.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:43.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.662773+0000 (oldest deadline 2024-04-27T03:52:31.662773+0000) 2024-04-27T04:02:43.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:43.635+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:44.095 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:02:43.859+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79594 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:02:44.096 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:43.855+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:43.856348+0000) 2024-04-27T04:02:44.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:43.788+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:43.789339+0000) 2024-04-27T04:02:44.096 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:43.854+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:43.855735+0000) 2024-04-27T04:02:44.096 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:43.855+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:43.856540+0000) 2024-04-27T04:02:44.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:44.095+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:44.095823+0000) 2024-04-27T04:02:44.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:44.151+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:44.399 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:44.151+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:44.399 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:44.151+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:44.399 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:44.152+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:44.672 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:44.594+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:44.594913+0000) 2024-04-27T04:02:44.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:44.505+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:44.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:44.505+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:44.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:44.505+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:44.673 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:44.505+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:44.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:44.672+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:44.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:44.792+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:44.793658+0000) 2024-04-27T04:02:45.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:45.097+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:45.098461+0000) 2024-04-27T04:02:45.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:45.193+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:45.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:45.193+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:45.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:45.193+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:45.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:45.194+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:45.678 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:45.598+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:45.599172+0000) 2024-04-27T04:02:45.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:45.553+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:45.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:45.553+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:45.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:45.553+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:09.070433+0000 (oldest deadline 2024-04-27T03:52:29.070433+0000) 2024-04-27T04:02:45.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:45.553+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:45.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:45.678+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:45.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:45.795+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:45.795840+0000) 2024-04-27T04:02:46.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:46.099+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:46.100690+0000) 2024-04-27T04:02:46.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:46.183+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:46.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:46.183+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:46.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:46.183+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T03:52:11.666296+0000 (oldest deadline 2024-04-27T03:52:31.666296+0000) 2024-04-27T04:02:46.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:46.185+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:46.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:46.524+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:46.705 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:46.602+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:46.603496+0000) 2024-04-27T04:02:46.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:46.704+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:46.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:46.797+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:46.797962+0000) 2024-04-27T04:02:47.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:47.102+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:47.102999+0000) 2024-04-27T04:02:47.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:47.193+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:47.669 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:47.488+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:47.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:47.607+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:47.608614+0000) 2024-04-27T04:02:47.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:47.669+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:47.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:47.799+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:47.800270+0000) 2024-04-27T04:02:48.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:48.104+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:48.105317+0000) 2024-04-27T04:02:48.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:48.234+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:48.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:48.646+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:48.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:48.521+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:48.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:48.612+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:48.613718+0000) 2024-04-27T04:02:49.107 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:48.863+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:48.864763+0000) 2024-04-27T04:02:49.108 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:02:48.868+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79633 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:02:49.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:48.802+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:48.802803+0000) 2024-04-27T04:02:49.108 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:48.863+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:48.864530+0000) 2024-04-27T04:02:49.108 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:48.863+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:48.864084+0000) 2024-04-27T04:02:49.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:49.196+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:49.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:49.107+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:49.107994+0000) 2024-04-27T04:02:49.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:49.626+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:49.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:49.615+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:49.615978+0000) 2024-04-27T04:02:49.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:49.477+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:50.112 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:49.806+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:49.807082+0000) 2024-04-27T04:02:50.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:50.112+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:50.113343+0000) 2024-04-27T04:02:50.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:50.153+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:50.658 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:50.456+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:50.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:50.617+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:50.618216+0000) 2024-04-27T04:02:50.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:50.657+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:50.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:50.808+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:50.809193+0000) 2024-04-27T04:02:51.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:51.203+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:51.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:51.117+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:51.118361+0000) 2024-04-27T04:02:51.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:51.620+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:51.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:51.621+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:51.622237+0000) 2024-04-27T04:02:51.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:51.478+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:52.114 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:51.810+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:51.811474+0000) 2024-04-27T04:02:52.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:52.211+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:52.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:52.120+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:52.121338+0000) 2024-04-27T04:02:52.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:52.486+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:52.668 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:52.624+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:52.625280+0000) 2024-04-27T04:02:52.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:52.667+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:52.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:52.813+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:52.814545+0000) 2024-04-27T04:02:53.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:53.230+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:53.399 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:53.125+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:53.126462+0000) 2024-04-27T04:02:53.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:53.643+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:53.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:53.467+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:53.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:53.627+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:53.627945+0000) 2024-04-27T04:02:53.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:53.872+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:53.873780+0000) 2024-04-27T04:02:53.970 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:02:53.877+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79674 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:02:53.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:53.815+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:53.816721+0000) 2024-04-27T04:02:53.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:53.873+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:53.874371+0000) 2024-04-27T04:02:53.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:53.874+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:53.874900+0000) 2024-04-27T04:02:54.274 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:54.127+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:54.128626+0000) 2024-04-27T04:02:54.630 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:54.274+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:54.631 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:54.476+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:54.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:54.630+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:54.631168+0000) 2024-04-27T04:02:54.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:54.634+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:54.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:54.818+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:54.819426+0000) 2024-04-27T04:02:55.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:55.320+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:55.399 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:55.132+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:55.133739+0000) 2024-04-27T04:02:55.651 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:55.633+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:55.634333+0000) 2024-04-27T04:02:55.651 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:55.481+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:55.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:55.651+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:55.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:55.823+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:55.824442+0000) 2024-04-27T04:02:56.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:56.138+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:56.138826+0000) 2024-04-27T04:02:56.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:56.342+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:56.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:56.434+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:56.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:56.645+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:56.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:56.827+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:56.828722+0000) 2024-04-27T04:02:56.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:56.635+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:56.636404+0000) 2024-04-27T04:02:57.382 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:57.143+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:57.144028+0000) 2024-04-27T04:02:57.638 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:57.472+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:57.639 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:57.382+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:57.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:57.687+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:57.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:57.830+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:57.830838+0000) 2024-04-27T04:02:57.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:57.638+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:57.639386+0000) 2024-04-27T04:02:58.399 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:58.148+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:58.149198+0000) 2024-04-27T04:02:58.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:58.489+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:58.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:58.413+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:58.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:58.643+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:58.644339+0000) 2024-04-27T04:02:58.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:58.882+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:58.883304+0000) 2024-04-27T04:02:58.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:58.882+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:58.882994+0000) 2024-04-27T04:02:58.971 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:02:58.886+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79713 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:02:58.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:58.669+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:58.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:58.834+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:58.835061+0000) 2024-04-27T04:02:58.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:58.882+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:58.883314+0000) 2024-04-27T04:02:59.287 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:59.153+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:59.154397+0000) 2024-04-27T04:02:59.646 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:59.415+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:59.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:02:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:02:59.459+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:59.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:02:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:02:59.645+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:59.646514+0000) 2024-04-27T04:02:59.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:59.679+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:02:59.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:02:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:02:59.838+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:02:59.839316+0000) 2024-04-27T04:03:00.376 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:00.155+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:00.156488+0000) 2024-04-27T04:03:00.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:00.375+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:00.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:00.425+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:00.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:00.650+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:00.651475+0000) 2024-04-27T04:03:00.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:00.648+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:00.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:00.841+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:00.841924+0000) 2024-04-27T04:03:01.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:01.157+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:01.158729+0000) 2024-04-27T04:03:01.653 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:01.627+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:01.653 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:01.394+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:01.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:01.393+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:01.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:01.843+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:01.844135+0000) 2024-04-27T04:03:01.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:01.652+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:01.653733+0000) 2024-04-27T04:03:02.399 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:02.160+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:02.161557+0000) 2024-04-27T04:03:02.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:02.584+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:02.657 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:02.399+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:02.657 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:02.435+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:02.909 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:02.845+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:02.846400+0000) 2024-04-27T04:03:02.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:02.656+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:02.656975+0000) 2024-04-27T04:03:03.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:03.386+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:03.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:03.164+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:03.164811+0000) 2024-04-27T04:03:03.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:03.659+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:03.661357+0000) 2024-04-27T04:03:03.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:03.414+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:03.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:03.538+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:04.115 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:03:03.893+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79756 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:03:04.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:03.846+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:03.848614+0000) 2024-04-27T04:03:04.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:03.890+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:03.891832+0000) 2024-04-27T04:03:04.116 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:03.889+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:03.891254+0000) 2024-04-27T04:03:04.117 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:03.890+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:03.892232+0000) 2024-04-27T04:03:04.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:04.347+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:04.394 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:04.165+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:04.167121+0000) 2024-04-27T04:03:04.394 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:04.392+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:04.662 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:04.511+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:04.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:04.851+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:04.852889+0000) 2024-04-27T04:03:04.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:04.661+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:04.663610+0000) 2024-04-27T04:03:05.370 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:05.348+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:05.371 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:05.167+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:05.169770+0000) 2024-04-27T04:03:05.371 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:05.369+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:05.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:05.507+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:05.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:05.853+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:05.855541+0000) 2024-04-27T04:03:05.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:05.664+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:05.666639+0000) 2024-04-27T04:03:06.288 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:06.172+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:06.174042+0000) 2024-04-27T04:03:06.544 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:06.334+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:06.545 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:06.355+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:06.858 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:06.543+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:06.858 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:06.666+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:06.668770+0000) 2024-04-27T04:03:07.116 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:06.856+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:06.858637+0000) 2024-04-27T04:03:07.378 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:07.176+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:07.178341+0000) 2024-04-27T04:03:07.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:07.377+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:07.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:07.525+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:07.671 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:07.388+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:07.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:07.859+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:07.860826+0000) 2024-04-27T04:03:07.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:07.669+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:07.671510+0000) 2024-04-27T04:03:08.387 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:08.180+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:08.182634+0000) 2024-04-27T04:03:08.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:08.434+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:08.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:08.434+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:08.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:08.434+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:08.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:08.435+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:08.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:08.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:08.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:08.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:08.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:08.541+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:08.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:08.542+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:08.676 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:08.386+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:08.676 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:08.386+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:08.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:08.386+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:08.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:08.387+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:08.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:08.898+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:08.899825+0000) 2024-04-27T04:03:08.971 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:03:08.902+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79795 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:03:08.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:08.861+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:08.862951+0000) 2024-04-27T04:03:08.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:08.900+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:08.901972+0000) 2024-04-27T04:03:08.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:08.673+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:08.674962+0000) 2024-04-27T04:03:08.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:08.898+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:08.900339+0000) 2024-04-27T04:03:09.289 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:09.183+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:09.185419+0000) 2024-04-27T04:03:09.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:09.368+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:09.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:09.368+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:09.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:09.368+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:09.678 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:09.369+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:09.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:09.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:09.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:09.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:09.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:09.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:09.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:09.515+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:09.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:09.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:09.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:09.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:09.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:09.394+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:09.679 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:09.394+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:09.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:09.863+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:09.865006+0000) 2024-04-27T04:03:09.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:09.675+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:09.677769+0000) 2024-04-27T04:03:10.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:10.185+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:10.187509+0000) 2024-04-27T04:03:10.400 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:10.332+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:10.400 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:10.332+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:10.401 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:10.332+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:10.401 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:10.333+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:10.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:10.431+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:10.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:10.431+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:10.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:10.431+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:10.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:10.431+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:10.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:10.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:10.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:10.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:10.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:10.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:10.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:10.471+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:10.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:10.865+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:10.867157+0000) 2024-04-27T04:03:10.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:10.679+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:10.681531+0000) 2024-04-27T04:03:11.400 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:11.188+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:11.190599+0000) 2024-04-27T04:03:11.400 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:11.349+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:11.401 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:11.349+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:11.401 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:11.349+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:11.401 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:11.350+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:11.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:11.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:11.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:11.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:11.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:11.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:11.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:11.510+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:11.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:11.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:11.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:11.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:11.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:11.399+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:11.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:11.400+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:11.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:11.868+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:11.869801+0000) 2024-04-27T04:03:11.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:11.683+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:11.685733+0000) 2024-04-27T04:03:12.400 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:12.336+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:12.400 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:12.336+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:12.401 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:12.336+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:12.401 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:12.337+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:12.401 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:12.190+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:12.192769+0000) 2024-04-27T04:03:12.401 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:12.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:12.401 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:12.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:12.401 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:12.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:12.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:12.356+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:12.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:12.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:12.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:12.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:12.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:12.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:12.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:12.465+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:12.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:12.870+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:12.871922+0000) 2024-04-27T04:03:12.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:12.687+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:12.688926+0000) 2024-04-27T04:03:13.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:13.193+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:13.195136+0000) 2024-04-27T04:03:13.388 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:13.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:13.389 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:13.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:13.389 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:13.363+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:13.389 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:13.364+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:13.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:13.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:13.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:13.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:13.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:13.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:13.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:13.445+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:13.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:13.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:13.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:13.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:13.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:13.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:13.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:13.387+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:13.970 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:03:13.911+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79836 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:03:13.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:13.906+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:13.908793+0000) 2024-04-27T04:03:13.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:13.874+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:13.876172+0000) 2024-04-27T04:03:13.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:13.907+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:13.908817+0000) 2024-04-27T04:03:13.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:13.689+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:13.691697+0000) 2024-04-27T04:03:13.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:13.907+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:13.908868+0000) 2024-04-27T04:03:14.224 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:14.196+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:14.197877+0000) 2024-04-27T04:03:14.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:14.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:14.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:14.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:14.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:14.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:14.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:14.431+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:14.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:14.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:14.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:14.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:14.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:14.387+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:14.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:14.387+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:14.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:14.371+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:14.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:14.371+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:14.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:14.371+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:14.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:14.371+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:14.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:14.879+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:14.881288+0000) 2024-04-27T04:03:14.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:14.693+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:14.694142+0000) 2024-04-27T04:03:15.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:15.199+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:15.200012+0000) 2024-04-27T04:03:15.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:15.362+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:15.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:15.362+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:15.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:15.362+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:15.396 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:15.363+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:15.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:15.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:15.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:15.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:15.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:15.416+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:15.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:15.416+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:15.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:15.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:15.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:15.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:15.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:15.395+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:15.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:15.396+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:15.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:15.882+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:15.883702+0000) 2024-04-27T04:03:15.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:15.696+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:15.697553+0000) 2024-04-27T04:03:16.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:16.201+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:16.202208+0000) 2024-04-27T04:03:16.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:16.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:16.368 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:16.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:16.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:16.350+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:16.369 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:16.350+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:16.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:16.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:16.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:16.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:16.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:16.456+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:16.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:16.457+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:16.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:16.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:16.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:16.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:16.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:16.407+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:16.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:16.408+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:16.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:16.886+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:16.887089+0000) 2024-04-27T04:03:16.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:16.700+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:16.700993+0000) 2024-04-27T04:03:17.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:17.203+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:17.204414+0000) 2024-04-27T04:03:17.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:17.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:17.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:17.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:17.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:17.344+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:17.374 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:17.344+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:17.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:17.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:17.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:17.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:17.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:17.430+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:17.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:17.431+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:17.704 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:17.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:17.704 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:17.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:17.705 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:17.373+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:17.705 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:17.374+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:17.705 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:17.703+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:17.703857+0000) 2024-04-27T04:03:17.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:17.889+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:17.889970+0000) 2024-04-27T04:03:18.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:18.205+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:18.206708+0000) 2024-04-27T04:03:18.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:18.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:18.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:18.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:18.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:18.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:18.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:18.325+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:18.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:18.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:18.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:18.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:18.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:18.449+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:18.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:18.450+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:18.707 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:18.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:18.707 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:18.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:18.707 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:18.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:18.707 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:18.398+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:18.970 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:03:18.922+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79875 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:03:18.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:18.916+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:18.917670+0000) 2024-04-27T04:03:18.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:18.892+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:18.893111+0000) 2024-04-27T04:03:18.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:18.917+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:18.917974+0000) 2024-04-27T04:03:18.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:18.705+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:18.706263+0000) 2024-04-27T04:03:18.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:18.917+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:18.918294+0000) 2024-04-27T04:03:19.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:19.208+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:19.209153+0000) 2024-04-27T04:03:19.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:19.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:19.708 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:19.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:19.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:19.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:19.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:19.453+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:19.709 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:19.323+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:19.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:19.323+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:19.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:19.323+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:19.710 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:19.324+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:19.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:19.431+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:19.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:19.431+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:19.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:19.431+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:19.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:19.432+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:19.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:19.895+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:19.895862+0000) 2024-04-27T04:03:19.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:19.707+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:19.708524+0000) 2024-04-27T04:03:20.401 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:20.210+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:20.211622+0000) 2024-04-27T04:03:20.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:20.349+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:20.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:20.349+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:20.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:20.349+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:20.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:20.350+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:20.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:20.419+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:20.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:20.419+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:20.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:20.419+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:20.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:20.420+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:20.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:20.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:20.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:20.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:20.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:20.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:20.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:20.442+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:20.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:20.898+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:20.898984+0000) 2024-04-27T04:03:20.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:20.710+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:20.711388+0000) 2024-04-27T04:03:21.397 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:21.214+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:21.214813+0000) 2024-04-27T04:03:21.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:21.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:21.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:21.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:21.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:21.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:21.398 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:21.338+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:21.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:21.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:21.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:21.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:21.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:21.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:21.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:21.444+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:21.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:21.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:21.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:21.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:21.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:21.397+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:21.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:21.398+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:21.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:21.900+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:21.901666+0000) 2024-04-27T04:03:21.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:21.713+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:21.714323+0000) 2024-04-27T04:03:22.379 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:22.216+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:22.217534+0000) 2024-04-27T04:03:22.380 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:22.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:22.380 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:22.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:22.380 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:22.356+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:22.380 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:22.357+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:22.380 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:22.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:22.380 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:22.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:22.381 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:22.379+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:22.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:22.380+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:22.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:22.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:22.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:22.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:22.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:22.436+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:22.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:22.438+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:22.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:22.904+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:22.905569+0000) 2024-04-27T04:03:22.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:22.716+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:22.717390+0000) 2024-04-27T04:03:23.370 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:23.219+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:23.219903+0000) 2024-04-27T04:03:23.371 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:23.370+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:23.371 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:23.370+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:23.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:23.370+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:23.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:23.371+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:23.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:23.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:23.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:23.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:23.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:23.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:23.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:23.487+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:23.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:23.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:23.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:23.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:23.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:23.383+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:23.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:23.384+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:23.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:23.928+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:23.928832+0000) 2024-04-27T04:03:23.970 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:03:23.931+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79916 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:03:23.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:23.906+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:23.907733+0000) 2024-04-27T04:03:23.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:23.926+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:23.927307+0000) 2024-04-27T04:03:23.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:23.718+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:23.719771+0000) 2024-04-27T04:03:23.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:23.929+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:23.929824+0000) 2024-04-27T04:03:24.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:24.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:24.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:24.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:24.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:24.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:24.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:24.468+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:24.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:24.374+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:24.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:24.374+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:24.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:24.374+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:24.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:24.374+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:24.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:24.223+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:24.224181+0000) 2024-04-27T04:03:24.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:24.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:24.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:24.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:24.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:24.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:24.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:24.344+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:25.118 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:24.911+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:24.912103+0000) 2024-04-27T04:03:25.118 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:24.722+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:24.722921+0000) 2024-04-27T04:03:25.393 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:25.226+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:25.227232+0000) 2024-04-27T04:03:25.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:25.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:25.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:25.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:25.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:25.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:25.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:25.476+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:25.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:25.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:25.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:25.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:25.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:25.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:25.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:25.422+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:25.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:25.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:25.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:25.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:25.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:25.393+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:25.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:25.394+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:26.118 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:25.724+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:25.725727+0000) 2024-04-27T04:03:26.118 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:25.913+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:25.914301+0000) 2024-04-27T04:03:26.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:26.230+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:26.230911+0000) 2024-04-27T04:03:26.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:26.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:26.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:26.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:26.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:26.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:26.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:26.516+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:26.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:26.438+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:26.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:26.438+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:26.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:26.438+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:26.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:26.440+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:26.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:26.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:26.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:26.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:26.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:26.406+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:26.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:26.406+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:27.118 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:26.728+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:26.728934+0000) 2024-04-27T04:03:27.119 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:26.917+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:26.918547+0000) 2024-04-27T04:03:27.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:27.234+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:27.235217+0000) 2024-04-27T04:03:27.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:27.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:27.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:27.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:27.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:27.440+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:27.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:27.441+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:27.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:27.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:27.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:27.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:27.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:27.517+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:27.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:27.519+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:27.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:27.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:27.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:27.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:27.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:27.423+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:27.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:27.424+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:28.118 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:27.730+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:27.731013+0000) 2024-04-27T04:03:28.119 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:27.922+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:27.922904+0000) 2024-04-27T04:03:28.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:28.237+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:28.238289+0000) 2024-04-27T04:03:28.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:28.439+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:28.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:28.439+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:28.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:28.439+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:28.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:28.439+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:28.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:28.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:28.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:28.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:28.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:28.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:28.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:28.446+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:28.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:28.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:28.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:28.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:28.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:28.479+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:28.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:28.480+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:29.059 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:03:28.940+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79955 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:03:29.059 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:28.937+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:28.938351+0000) 2024-04-27T04:03:29.059 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:28.925+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:28.926176+0000) 2024-04-27T04:03:29.060 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:28.936+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:28.937602+0000) 2024-04-27T04:03:29.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:28.732+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:28.733118+0000) 2024-04-27T04:03:29.060 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:28.936+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:28.937228+0000) 2024-04-27T04:03:29.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:29.240+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:29.241297+0000) 2024-04-27T04:03:29.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:29.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:29.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:29.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:29.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:29.445+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:29.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:29.446+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:29.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:29.441+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:29.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:29.441+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:29.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:29.441+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:29.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:29.442+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:29.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:29.502+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:29.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:29.502+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:29.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:29.502+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:29.725 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:29.503+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:30.119 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:29.928+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:29.928813+0000) 2024-04-27T04:03:30.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:29.734+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:29.735417+0000) 2024-04-27T04:03:30.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:30.242+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:30.243414+0000) 2024-04-27T04:03:30.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:30.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:30.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:30.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:30.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:30.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:30.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:30.471+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:30.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:30.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:30.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:30.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:30.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:30.474+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:30.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:30.475+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:30.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:30.471+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:30.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:30.471+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:30.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:30.471+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:30.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:30.472+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:31.119 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:30.932+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:30.933489+0000) 2024-04-27T04:03:31.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:30.736+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:30.737571+0000) 2024-04-27T04:03:31.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:31.246+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:31.247693+0000) 2024-04-27T04:03:31.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:31.473+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:31.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:31.473+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:31.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:31.473+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:31.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:31.474+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:31.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:31.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:31.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:31.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:31.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:31.500+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:31.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:31.501+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:31.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:31.502+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:31.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:31.502+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:31.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:31.502+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:31.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:31.503+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:32.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:31.741+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:31.741845+0000) 2024-04-27T04:03:32.119 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:31.934+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:31.935638+0000) 2024-04-27T04:03:32.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:32.249+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:32.249953+0000) 2024-04-27T04:03:32.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:32.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:32.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:32.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:32.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:32.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:32.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:32.474+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:32.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:32.468+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:32.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:32.468+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:32.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:32.468+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:32.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:32.468+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:32.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:32.508+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:32.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:32.508+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:32.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:32.508+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:32.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:32.510+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:33.119 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:32.938+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:32.938833+0000) 2024-04-27T04:03:33.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:32.743+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:32.744091+0000) 2024-04-27T04:03:33.403 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:33.253+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:33.253823+0000) 2024-04-27T04:03:33.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:33.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:33.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:33.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:33.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:33.533+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:33.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:33.534+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:33.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:33.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:33.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:33.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:33.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:33.464+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:33.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:33.465+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:33.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:33.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:33.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:33.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:33.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:33.430+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:33.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:33.431+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:34.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:03:33.950+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 79996 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:03:34.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:33.745+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:33.746197+0000) 2024-04-27T04:03:34.070 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:33.945+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:33.946622+0000) 2024-04-27T04:03:34.070 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:33.946+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:33.946910+0000) 2024-04-27T04:03:34.070 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:33.940+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:33.941500+0000) 2024-04-27T04:03:34.071 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:33.947+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:33.948302+0000) 2024-04-27T04:03:34.403 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:34.255+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:34.256728+0000) 2024-04-27T04:03:34.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:34.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:34.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:34.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:34.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:34.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:34.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:34.496+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:34.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:34.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:34.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:34.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:34.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:34.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:34.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:34.424+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:34.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:34.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:34.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:34.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:34.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:34.425+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:34.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:34.426+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:35.119 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:34.944+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:34.945051+0000) 2024-04-27T04:03:35.119 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:34.748+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:34.749242+0000) 2024-04-27T04:03:35.403 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:35.258+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:35.259795+0000) 2024-04-27T04:03:35.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:35.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:35.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:35.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:35.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:35.494+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:35.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:35.495+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:35.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:35.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:35.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:35.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:35.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:35.423+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:35.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:35.423+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:35.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:35.463+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:35.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:35.463+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:35.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:35.463+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:35.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:35.464+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:36.119 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:35.946+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:35.947151+0000) 2024-04-27T04:03:36.120 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:35.753+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:35.753968+0000) 2024-04-27T04:03:36.388 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:36.261+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:36.261960+0000) 2024-04-27T04:03:36.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:36.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:36.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:36.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:36.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:36.512+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:36.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:36.513+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:36.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:36.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:36.702 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:36.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:36.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:36.414+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:36.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:36.415+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:36.703 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:36.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:36.703 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:36.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:36.703 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:36.388+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:36.704 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:36.389+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:36.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:36.949+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:36.949815+0000) 2024-04-27T04:03:36.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:36.756+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:36.756823+0000) 2024-04-27T04:03:37.385 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:37.264+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:37.265278+0000) 2024-04-27T04:03:37.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:37.427+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:37.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:37.427+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:37.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:37.427+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:37.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:37.427+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:37.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:37.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:37.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:37.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:37.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:37.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:37.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:37.491+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:37.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:37.384+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:37.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:37.384+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:37.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:37.384+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:37.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:37.385+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:38.119 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:37.951+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:37.951903+0000) 2024-04-27T04:03:38.120 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:37.758+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:37.758878+0000) 2024-04-27T04:03:38.403 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:38.267+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:38.268034+0000) 2024-04-27T04:03:38.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:38.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:38.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:38.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:38.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:38.348+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:38.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:38.349+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:38.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:38.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:38.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:38.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:38.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:38.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:38.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:38.427+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:38.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:38.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:38.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:38.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:38.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:38.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:38.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:38.469+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:39.082 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:03:38.963+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80035 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:03:39.082 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:38.954+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:38.954991+0000) 2024-04-27T04:03:39.083 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:38.954+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:38.955753+0000) 2024-04-27T04:03:39.083 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:38.955+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:38.955912+0000) 2024-04-27T04:03:39.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:38.760+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:38.761600+0000) 2024-04-27T04:03:39.084 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:38.954+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:38.954832+0000) 2024-04-27T04:03:39.342 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:39.271+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:39.271941+0000) 2024-04-27T04:03:39.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:39.342+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:39.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:39.342+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:39.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:39.342+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:39.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:39.343+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:39.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:39.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:39.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:39.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:39.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:39.447+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:39.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:39.448+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:39.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:39.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:39.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:39.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:39.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:39.404+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:39.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:39.404+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:40.120 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:39.958+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:39.959422+0000) 2024-04-27T04:03:40.120 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:39.765+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:39.765960+0000) 2024-04-27T04:03:40.403 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:40.275+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:40.276775+0000) 2024-04-27T04:03:40.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:40.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:40.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:40.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:40.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:40.309+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:40.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:40.310+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:40.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:40.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:40.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:40.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:40.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:40.406+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:40.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:40.407+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:40.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:40.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:40.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:40.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:40.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:40.412+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:40.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:40.413+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:41.120 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:40.961+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:40.962391+0000) 2024-04-27T04:03:41.120 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:40.768+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:40.769119+0000) 2024-04-27T04:03:41.403 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:41.278+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:41.278982+0000) 2024-04-27T04:03:41.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:41.336+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:41.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:41.336+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:41.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:41.336+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:41.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:41.337+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:41.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:41.438+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:41.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:41.438+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:41.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:41.438+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:41.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:41.440+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:41.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:41.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:41.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:41.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:41.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:41.408+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:41.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:41.408+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:42.120 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:41.965+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:41.966663+0000) 2024-04-27T04:03:42.120 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:41.770+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:41.771259+0000) 2024-04-27T04:03:42.403 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:42.281+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:42.282211+0000) 2024-04-27T04:03:42.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:42.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:42.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:42.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:42.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:42.305+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:42.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:42.306+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:42.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:42.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:42.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:42.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:42.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:42.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:42.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:42.468+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:42.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:42.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:42.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:42.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:42.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:42.409+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:42.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:42.409+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:43.120 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:42.970+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:42.970946+0000) 2024-04-27T04:03:43.120 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:42.774+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:42.775584+0000) 2024-04-27T04:03:43.403 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:43.283+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:43.284345+0000) 2024-04-27T04:03:43.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:43.340+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:43.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:43.340+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:43.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:43.340+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:43.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:43.341+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:43.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:43.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:43.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:43.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:43.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:43.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:43.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:43.420+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:43.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:43.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:43.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:43.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:43.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:43.426+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:43.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:43.427+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:44.090 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:03:43.973+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80076 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:03:44.091 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:43.969+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:43.970433+0000) 2024-04-27T04:03:44.091 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:43.974+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:43.975243+0000) 2024-04-27T04:03:44.091 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:43.779+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:43.779971+0000) 2024-04-27T04:03:44.092 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:43.967+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:43.968325+0000) 2024-04-27T04:03:44.092 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:43.968+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:43.968802+0000) 2024-04-27T04:03:44.386 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:44.285+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:44.286464+0000) 2024-04-27T04:03:44.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:44.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:44.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:44.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:44.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:44.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:44.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:44.467+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:44.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:44.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:44.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:44.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:44.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:44.456+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:44.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:44.456+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:44.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:44.385+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:44.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:44.385+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:44.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:44.386+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:44.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:44.387+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:45.120 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:44.977+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:44.978018+0000) 2024-04-27T04:03:45.121 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:44.781+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:44.782117+0000) 2024-04-27T04:03:45.392 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:45.290+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:45.291519+0000) 2024-04-27T04:03:45.392 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:45.392+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:45.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:45.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:45.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:45.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:45.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:45.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:45.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:45.471+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:45.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:45.392+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:45.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:45.392+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:45.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:45.393+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:45.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:45.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:45.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:45.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:45.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:45.477+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:45.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:45.477+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:46.122 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:45.980+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:45.981761+0000) 2024-04-27T04:03:46.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:45.786+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:45.787151+0000) 2024-04-27T04:03:46.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:46.293+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:46.294737+0000) 2024-04-27T04:03:46.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:46.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:46.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:46.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:46.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:46.421+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:46.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:46.422+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:46.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:46.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:46.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:46.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:46.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:46.526+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:46.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:46.527+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:46.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:46.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:46.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:46.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:46.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:46.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:46.724 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:46.508+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:47.120 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:46.789+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:46.790716+0000) 2024-04-27T04:03:47.121 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:46.983+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:46.984451+0000) 2024-04-27T04:03:47.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:47.296+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:47.297464+0000) 2024-04-27T04:03:47.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:47.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:47.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:47.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:47.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:47.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:47.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:47.477+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:47.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:47.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:47.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:47.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:47.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:47.465+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:47.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:47.466+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:47.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:47.554+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:47.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:47.554+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:47.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:47.554+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:47.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:47.555+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:48.121 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:47.988+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:47.989244+0000) 2024-04-27T04:03:48.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:47.792+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:47.792824+0000) 2024-04-27T04:03:48.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:48.299+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:48.300482+0000) 2024-04-27T04:03:48.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:48.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:48.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:48.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:48.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:48.458+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:48.688 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:48.459+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:48.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:48.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:48.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:48.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:48.689 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:48.511+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:48.689 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:48.513+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:48.689 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:48.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:48.689 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:48.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:48.689 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:48.579+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:48.689 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:48.580+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:48.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:48.797+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:48.797942+0000) 2024-04-27T04:03:49.290 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:03:48.983+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80115 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:03:49.291 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:48.978+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:48.979340+0000) 2024-04-27T04:03:49.291 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:48.990+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:48.991388+0000) 2024-04-27T04:03:49.291 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:48.978+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:48.979729+0000) 2024-04-27T04:03:49.292 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:48.977+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:48.978018+0000) 2024-04-27T04:03:49.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:49.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:49.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:49.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:49.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:49.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:49.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:49.445+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:49.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:49.301+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:49.302777+0000) 2024-04-27T04:03:49.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:49.538+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:49.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:49.538+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:49.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:49.538+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:49.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:49.539+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:49.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:49.536+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:49.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:49.536+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:49.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:49.536+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:49.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:49.537+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:50.121 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:49.993+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:49.994481+0000) 2024-04-27T04:03:50.121 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:49.799+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:49.800316+0000) 2024-04-27T04:03:50.403 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:50.306+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:50.307035+0000) 2024-04-27T04:03:50.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:50.529+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:50.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:50.529+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:50.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:50.529+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:50.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:50.530+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:50.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:50.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:50.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:50.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:50.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:50.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:50.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:50.404+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:50.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:50.534+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:50.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:50.534+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:50.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:50.534+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:50.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:50.534+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:51.121 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:50.996+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:50.996799+0000) 2024-04-27T04:03:51.121 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:50.801+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:50.802774+0000) 2024-04-27T04:03:51.404 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:51.308+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:51.309445+0000) 2024-04-27T04:03:51.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:51.484+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:51.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:51.484+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:51.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:51.484+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:51.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:51.485+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:51.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:51.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:51.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:51.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:51.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:51.454+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:51.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:51.455+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:51.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:51.572+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:51.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:51.572+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:51.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:51.572+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:51.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:51.573+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:52.061 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:52.000+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:52.001070+0000) 2024-04-27T04:03:52.061 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:51.805+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:51.805972+0000) 2024-04-27T04:03:52.569 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:52.311+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:52.312325+0000) 2024-04-27T04:03:52.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:52.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:52.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:52.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:52.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:52.460+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:52.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:52.461+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:52.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:52.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:52.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:52.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:52.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:52.494+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:52.571 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:52.495+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:52.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:52.808+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:52.808897+0000) 2024-04-27T04:03:52.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:52.569+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:52.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:52.569+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:52.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:52.569+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:52.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:52.569+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:53.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:53.003+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:53.004137+0000) 2024-04-27T04:03:53.562 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:53.315+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:53.316597+0000) 2024-04-27T04:03:53.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:53.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:53.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:53.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:53.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:53.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:53.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:53.509+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:53.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:53.503+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:53.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:53.503+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:53.713 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:53.503+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:53.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:53.504+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:53.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:53.812+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:53.813245+0000) 2024-04-27T04:03:53.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:53.562+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:53.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:53.562+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:53.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:53.562+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:53.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:53.562+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:54.294 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:03:53.995+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80156 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:03:54.295 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:53.988+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:53.988884+0000) 2024-04-27T04:03:54.295 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:53.989+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:53.990558+0000) 2024-04-27T04:03:54.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:53.987+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:53.988742+0000) 2024-04-27T04:03:54.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:54.007+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:54.008387+0000) 2024-04-27T04:03:54.596 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:54.531+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:54.596 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:54.531+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:54.596 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:54.531+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:54.596 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:54.532+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:54.597 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:54.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:54.597 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:54.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:54.597 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:54.520+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:54.597 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:54.521+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:54.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:54.319+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:54.320103+0000) 2024-04-27T04:03:54.598 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:54.595+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:54.598 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:54.595+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:54.598 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:54.595+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:54.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:54.596+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:54.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:54.814+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:54.815594+0000) 2024-04-27T04:03:55.294 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:55.009+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:55.010515+0000) 2024-04-27T04:03:55.560 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:55.321+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:55.322737+0000) 2024-04-27T04:03:55.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:55.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:55.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:55.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:55.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:55.490+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:55.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:55.490+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:55.561 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:55.514+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:55.562 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:55.514+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:55.562 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:55.514+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:55.562 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:55.514+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:55.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:55.560+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:55.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:55.560+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:55.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:55.560+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:55.819 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:55.560+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:56.121 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:56.011+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:56.012636+0000) 2024-04-27T04:03:56.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:55.819+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:55.819901+0000) 2024-04-27T04:03:56.405 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:56.325+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:56.325996+0000) 2024-04-27T04:03:56.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:56.559+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:56.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:56.559+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:56.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:56.559+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:56.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:56.559+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:56.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:56.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:56.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:56.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:56.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:56.509+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:56.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:56.510+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:56.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:56.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:56.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:56.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:56.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:56.544+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:56.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:56.544+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:57.121 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:57.013+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:57.014763+0000) 2024-04-27T04:03:57.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:56.821+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:56.822041+0000) 2024-04-27T04:03:57.405 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:57.328+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:57.328850+0000) 2024-04-27T04:03:57.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:57.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:57.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:57.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:57.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:57.468+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:57.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:57.470+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:57.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:57.585+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:57.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:57.585+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:57.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:57.585+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:57.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:57.585+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:57.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:57.505+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:57.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:57.505+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:57.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:57.505+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:57.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:57.506+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:58.121 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:58.016+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:58.016841+0000) 2024-04-27T04:03:58.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:57.823+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:57.824139+0000) 2024-04-27T04:03:58.406 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:58.332+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:58.333111+0000) 2024-04-27T04:03:58.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:58.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:58.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:58.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:58.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:58.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:58.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:58.442+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:58.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:58.478+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:58.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:58.478+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:58.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:58.478+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:58.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:58.480+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:58.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:58.588+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:58.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:58.588+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:58.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:58.588+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:58.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:58.589+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:59.122 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:58.999+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:59.000007+0000) 2024-04-27T04:03:59.122 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:03:59.003+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80195 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:03:59.123 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:59.003+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:59.003935+0000) 2024-04-27T04:03:59.123 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:59.021+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:59.021959+0000) 2024-04-27T04:03:59.123 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:58.828+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:58.829226+0000) 2024-04-27T04:03:59.124 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:59.000+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:59.000903+0000) 2024-04-27T04:03:59.405 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:59.334+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:59.335376+0000) 2024-04-27T04:03:59.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:59.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:59.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:59.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:59.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:59.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:03:59.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:03:59.427+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:59.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:59.630+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:59.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:59.630+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:59.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:59.630+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:03:59.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:03:59.630+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:03:59.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:59.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:59.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:59.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:59.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:59.444+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:03:59.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:59.446+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:00.122 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:00.023+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:00.024207+0000) 2024-04-27T04:04:00.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:03:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:03:59.831+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:03:59.832793+0000) 2024-04-27T04:04:00.405 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:00.336+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:00.337734+0000) 2024-04-27T04:04:00.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:00.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:00.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:00.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:00.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:00.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:00.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:00.445+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:00.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:00.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:00.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:00.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:00.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:00.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:00.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:00.458+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:00.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:00.652+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:00.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:00.652+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:00.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:00.652+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:00.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:00.652+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:01.122 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:01.027+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:01.028441+0000) 2024-04-27T04:04:01.122 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:00.834+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:00.834982+0000) 2024-04-27T04:04:01.405 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:01.339+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:01.340007+0000) 2024-04-27T04:04:01.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:01.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:01.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:01.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:01.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:01.466+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:01.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:01.466+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:01.681 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:01.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:01.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:01.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:01.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:01.418+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:01.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:01.419+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:01.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:01.836+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:01.837133+0000) 2024-04-27T04:04:01.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:01.680+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:01.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:01.680+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:01.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:01.680+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:01.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:01.681+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:02.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:02.029+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:02.030590+0000) 2024-04-27T04:04:02.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:02.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:02.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:02.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:02.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:02.440+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:02.710 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:02.441+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:02.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:02.410+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:02.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:02.410+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:02.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:02.410+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:02.711 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:02.411+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:02.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:02.341+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:02.342485+0000) 2024-04-27T04:04:02.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:02.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:02.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:02.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:02.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:02.709+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:02.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:02.838+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:02.839255+0000) 2024-04-27T04:04:02.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:02.710+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:03.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:03.031+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:03.032683+0000) 2024-04-27T04:04:03.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:03.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:03.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:03.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:03.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:03.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:03.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:03.400+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:03.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:03.438+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:03.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:03.438+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:03.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:03.438+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:03.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:03.438+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:03.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:03.343+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:03.344701+0000) 2024-04-27T04:04:03.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:03.719+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:03.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:03.719+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:03.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:03.719+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:03.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:03.840+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:03.841403+0000) 2024-04-27T04:04:03.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:03.720+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:04.295 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:04:04.013+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80236 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:04:04.296 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:04.009+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:04.010119+0000) 2024-04-27T04:04:04.296 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:04.034+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:04.034820+0000) 2024-04-27T04:04:04.296 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:04.009+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:04.010221+0000) 2024-04-27T04:04:04.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:04.008+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:04.009118+0000) 2024-04-27T04:04:04.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:04.346+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:04.346822+0000) 2024-04-27T04:04:04.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:04.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:04.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:04.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:04.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:04.415+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:04.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:04.416+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:04.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:04.408+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:04.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:04.408+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:04.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:04.408+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:04.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:04.409+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:04.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:04.841+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:04.843507+0000) 2024-04-27T04:04:04.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:04.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:04.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:04.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:04.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:04.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:04.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:04.713+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:05.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:05.035+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:05.036907+0000) 2024-04-27T04:04:05.683 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:05.348+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:05.350747+0000) 2024-04-27T04:04:05.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:05.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:05.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:05.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:05.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:05.419+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:05.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:05.419+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:05.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:05.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:05.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:05.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:05.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:05.454+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:05.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:05.455+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:05.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:05.843+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:05.845777+0000) 2024-04-27T04:04:05.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:05.681+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:05.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:05.681+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:05.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:05.681+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:05.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:05.682+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:06.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:06.037+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:06.039038+0000) 2024-04-27T04:04:06.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:06.351+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:06.352942+0000) 2024-04-27T04:04:06.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:06.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:06.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:06.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:06.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:06.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:06.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:06.435+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:06.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:06.417+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:06.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:06.417+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:06.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:06.417+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:06.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:06.418+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:07.040 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:06.726+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:07.040 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:06.726+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:07.040 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:06.726+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:07.041 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:06.727+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:07.041 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:06.846+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:06.848513+0000) 2024-04-27T04:04:07.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:07.039+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:07.041192+0000) 2024-04-27T04:04:07.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:07.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:07.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:07.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:07.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:07.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:07.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:07.396+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:07.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:07.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:07.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:07.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:07.698 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:07.457+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:07.699 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:07.458+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:07.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:07.353+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:07.355190+0000) 2024-04-27T04:04:07.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:07.696+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:07.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:07.696+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:07.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:07.696+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:07.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:07.850+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:07.851846+0000) 2024-04-27T04:04:07.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:07.696+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:08.295 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:08.044+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:08.045952+0000) 2024-04-27T04:04:08.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:08.355+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:08.357424+0000) 2024-04-27T04:04:08.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:08.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:08.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:08.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:08.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:08.357+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:08.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:08.357+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:08.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:08.487+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:08.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:08.487+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:08.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:08.487+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:08.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:08.488+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:09.015 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:08.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:09.015 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:08.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:09.015 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:08.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:09.015 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:08.730+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:09.016 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:08.852+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:08.854192+0000) 2024-04-27T04:04:09.296 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:04:09.022+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80275 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:04:09.296 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:09.018+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:09.020738+0000) 2024-04-27T04:04:09.296 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:09.046+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:09.048050+0000) 2024-04-27T04:04:09.297 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:09.016+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:09.018700+0000) 2024-04-27T04:04:09.297 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:09.016+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:09.018688+0000) 2024-04-27T04:04:09.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:09.359+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:09.361697+0000) 2024-04-27T04:04:09.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:09.487+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:09.694 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:09.487+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:09.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:09.487+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:09.695 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:09.488+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:09.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:09.393+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:09.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:09.393+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:09.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:09.393+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:09.695 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:09.394+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:09.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:09.854+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:09.856285+0000) 2024-04-27T04:04:09.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:09.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:09.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:09.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:09.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:09.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:09.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:09.693+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:10.353 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:10.050+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:10.052206+0000) 2024-04-27T04:04:10.663 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:10.362+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:10.364165+0000) 2024-04-27T04:04:10.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:10.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:10.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:10.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:10.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:10.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:10.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:10.432+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:10.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:10.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:10.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:10.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:10.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:10.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:10.665 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:10.456+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:10.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:10.857+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:10.859230+0000) 2024-04-27T04:04:10.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:10.662+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:10.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:10.662+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:10.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:10.662+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:10.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:10.662+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:11.356 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:11.053+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:11.055183+0000) 2024-04-27T04:04:11.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:11.391+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:11.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:11.391+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:11.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:11.391+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:11.705 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:11.392+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:11.706 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:11.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:11.706 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:11.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:11.706 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:11.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:11.706 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:11.501+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:11.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:11.365+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:11.367263+0000) 2024-04-27T04:04:11.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:11.703+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:11.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:11.703+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:11.707 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:11.703+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:11.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:11.704+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:11.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:11.859+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:11.861393+0000) 2024-04-27T04:04:12.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:12.055+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:12.057395+0000) 2024-04-27T04:04:12.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:12.542+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:12.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:12.542+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:12.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:12.542+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:12.680 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:12.543+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:12.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:12.407+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:12.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:12.407+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:12.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:12.407+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:12.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:12.409+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:12.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:12.367+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:12.369685+0000) 2024-04-27T04:04:12.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:12.678+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:12.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:12.678+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:12.681 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:12.678+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:12.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:12.861+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:12.863618+0000) 2024-04-27T04:04:12.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:12.679+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:13.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:13.058+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:13.060280+0000) 2024-04-27T04:04:13.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:13.371+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:13.373215+0000) 2024-04-27T04:04:13.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:13.715+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:13.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:13.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:13.716 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:13.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:13.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:13.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:13.717 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:13.425+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:13.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:13.514+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:13.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:13.514+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:13.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:13.514+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:13.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:13.514+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:13.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:13.865+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:13.866944+0000) 2024-04-27T04:04:13.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:13.715+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:13.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:13.715+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:13.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:13.715+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:14.296 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:04:14.031+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80316 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:04:14.297 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:14.027+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:14.029589+0000) 2024-04-27T04:04:14.297 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:14.061+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:14.063223+0000) 2024-04-27T04:04:14.297 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:14.026+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:14.028685+0000) 2024-04-27T04:04:14.297 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:14.026+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:14.028555+0000) 2024-04-27T04:04:14.668 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:14.373+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:14.375561+0000) 2024-04-27T04:04:14.669 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:14.667+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:14.669 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:14.667+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:14.669 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:14.667+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:14.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:14.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:14.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:14.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:14.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:14.385+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:14.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:14.385+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:14.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:14.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:14.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:14.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:14.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:14.501+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:14.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:14.501+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:14.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:14.867+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:14.869757+0000) 2024-04-27T04:04:14.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:14.668+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:15.356 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:15.063+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:15.065470+0000) 2024-04-27T04:04:15.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:15.355+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:15.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:15.355+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:15.357 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:15.355+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:15.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:15.356+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:15.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:15.375+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:15.377749+0000) 2024-04-27T04:04:15.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:15.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:15.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:15.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:15.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:15.550+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:15.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:15.550+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:15.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:15.872+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:15.874022+0000) 2024-04-27T04:04:15.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:15.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:15.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:15.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:15.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:15.712+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:15.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:15.713+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:16.363 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:16.066+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:16.067578+0000) 2024-04-27T04:04:16.637 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:16.379+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:16.381030+0000) 2024-04-27T04:04:16.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:16.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:16.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:16.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:16.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:16.403+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:16.638 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:16.403+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:16.638 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:16.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:16.638 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:16.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:16.639 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:16.521+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:16.639 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:16.521+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:16.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:16.875+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:16.876331+0000) 2024-04-27T04:04:16.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:16.746+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:16.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:16.746+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:16.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:16.746+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:16.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:16.746+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:17.365 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:17.069+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:17.070141+0000) 2024-04-27T04:04:17.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:17.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:17.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:17.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:17.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:17.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:17.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:17.486+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:17.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:17.382+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:17.383265+0000) 2024-04-27T04:04:17.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:17.718+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:17.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:17.718+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:17.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:17.718+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:17.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:17.371+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:17.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:17.371+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:17.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:17.371+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:17.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:17.372+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:17.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:17.878+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:17.879323+0000) 2024-04-27T04:04:17.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:17.719+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:18.366 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:18.073+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:18.074446+0000) 2024-04-27T04:04:18.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:18.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:18.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:18.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:18.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:18.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:18.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:18.443+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:18.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:18.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:18.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:18.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:18.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:18.369+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:18.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:18.369+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:18.723 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:18.385+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:18.386307+0000) 2024-04-27T04:04:19.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:18.764+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:19.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:18.764+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:19.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:18.764+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:19.022 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:18.765+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:19.023 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:18.880+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:18.881639+0000) 2024-04-27T04:04:19.297 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:04:19.043+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80355 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:04:19.297 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:19.038+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:19.039258+0000) 2024-04-27T04:04:19.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:19.038+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:19.039236+0000) 2024-04-27T04:04:19.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:19.077+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:19.078719+0000) 2024-04-27T04:04:19.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:19.038+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:19.039224+0000) 2024-04-27T04:04:19.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:19.388+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:19.389048+0000) 2024-04-27T04:04:19.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:19.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:19.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:19.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:19.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:19.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:19.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:19.389+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:19.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:19.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:19.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:19.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:19.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:19.442+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:19.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:19.444+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:20.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:19.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:20.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:19.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:20.080 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:19.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:20.081 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:19.730+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:20.081 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:19.883+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:19.884372+0000) 2024-04-27T04:04:20.363 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:20.080+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:20.080845+0000) 2024-04-27T04:04:20.363 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:20.362+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:20.363 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:20.362+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:20.363 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:20.362+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:20.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:20.364+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:20.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:20.391+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:20.391948+0000) 2024-04-27T04:04:20.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:20.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:20.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:20.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:20.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:20.455+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:20.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:20.455+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:21.083 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:20.888+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:20.889410+0000) 2024-04-27T04:04:21.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:20.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:21.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:20.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:21.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:20.772+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:21.084 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:20.773+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:21.339 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:21.083+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:21.084314+0000) 2024-04-27T04:04:21.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:21.394+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:21.395223+0000) 2024-04-27T04:04:21.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:21.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:21.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:21.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:21.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:21.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:21.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:21.340+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:21.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:21.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:21.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:21.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:21.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:21.473+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:21.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:21.474+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:22.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:21.779+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:22.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:21.779+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:22.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:21.779+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:22.088 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:21.779+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:22.089 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:21.891+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:21.892298+0000) 2024-04-27T04:04:22.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:22.088+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:22.088889+0000) 2024-04-27T04:04:22.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:22.324+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:22.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:22.324+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:22.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:22.324+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:22.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:22.325+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:22.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:22.397+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:22.397821+0000) 2024-04-27T04:04:22.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:22.493+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:22.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:22.493+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:22.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:22.493+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:22.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:22.494+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:23.092 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:22.895+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:22.896617+0000) 2024-04-27T04:04:23.093 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:22.805+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:23.093 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:22.805+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:23.093 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:22.805+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:23.093 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:22.805+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:23.376 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:23.092+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:23.093699+0000) 2024-04-27T04:04:23.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:23.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:23.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:23.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:23.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:23.339+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:23.377 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:23.340+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:23.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:23.401+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:23.402082+0000) 2024-04-27T04:04:23.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:23.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:23.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:23.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:23.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:23.499+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:23.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:23.500+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:24.097 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:24.047+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:24.048554+0000) 2024-04-27T04:04:24.098 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:04:24.052+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80396 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:04:24.098 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:23.898+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:23.898946+0000) 2024-04-27T04:04:24.099 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:24.051+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:24.052177+0000) 2024-04-27T04:04:24.099 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:23.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:24.099 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:23.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:24.099 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:23.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:24.099 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:23.842+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:24.099 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:24.049+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:24.049852+0000) 2024-04-27T04:04:24.376 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:24.097+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:24.097903+0000) 2024-04-27T04:04:24.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:24.404+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:24.404942+0000) 2024-04-27T04:04:24.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:24.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:24.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:24.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:24.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:24.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:24.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:24.384+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:24.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:24.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:24.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:24.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:24.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:24.545+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:24.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:24.547+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:25.125 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:25.099+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:25.100542+0000) 2024-04-27T04:04:25.125 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:24.901+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:24.902172+0000) 2024-04-27T04:04:25.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:24.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:25.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:24.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:25.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:24.870+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:25.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:24.870+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:25.379 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:25.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:25.379 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:25.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:25.379 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:25.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:25.379 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:25.359+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:25.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:25.408+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:25.409653+0000) 2024-04-27T04:04:25.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:25.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:25.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:25.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:25.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:25.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:25.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:25.593+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:26.125 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:26.101+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:26.102633+0000) 2024-04-27T04:04:26.125 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:25.903+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:25.904236+0000) 2024-04-27T04:04:26.125 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:25.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:26.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:25.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:26.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:25.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:26.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:25.863+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:26.384 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:26.351+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:26.384 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:26.351+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:26.384 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:26.351+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:26.385 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:26.352+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:26.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:26.413+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:26.414297+0000) 2024-04-27T04:04:26.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:26.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:26.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:26.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:26.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:26.575+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:26.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:26.576+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:27.104 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:26.906+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:26.907261+0000) 2024-04-27T04:04:27.104 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:26.828+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:27.104 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:26.828+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:27.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:26.828+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:27.105 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:26.828+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:27.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:27.104+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:27.104796+0000) 2024-04-27T04:04:27.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:27.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:27.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:27.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:27.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:27.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:27.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:27.384+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:27.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:27.415+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:27.416630+0000) 2024-04-27T04:04:27.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:27.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:27.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:27.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:27.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:27.594+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:27.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:27.595+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:28.125 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:28.106+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:28.107528+0000) 2024-04-27T04:04:28.125 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:27.908+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:27.909407+0000) 2024-04-27T04:04:28.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:27.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:28.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:27.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:28.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:27.861+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:28.126 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:27.862+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:28.386 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:28.348+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:28.386 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:28.348+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:28.386 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:28.348+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:28.386 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:28.349+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:28.640 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:28.418+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:28.418901+0000) 2024-04-27T04:04:28.910 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:28.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:28.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:28.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:28.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:28.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:28.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:28.641+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:28.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:28.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:28.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:28.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:28.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:28.826+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:28.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:28.826+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:29.182 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:04:29.061+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80435 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:04:29.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:29.057+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:29.058543+0000) 2024-04-27T04:04:29.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:29.059+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:29.059988+0000) 2024-04-27T04:04:29.183 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:29.110+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:29.111785+0000) 2024-04-27T04:04:29.184 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:28.910+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:28.911582+0000) 2024-04-27T04:04:29.184 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:29.057+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:29.058198+0000) 2024-04-27T04:04:29.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:29.420+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:29.421184+0000) 2024-04-27T04:04:29.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:29.344+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:29.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:29.344+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:29.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:29.344+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:29.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:29.345+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:29.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:29.858+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:29.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:29.858+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:29.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:29.858+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:29.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:29.859+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:29.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:29.684+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:29.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:29.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:29.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:29.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:29.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:29.686+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:29.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:29.913+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:29.914387+0000) 2024-04-27T04:04:30.366 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:30.113+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:30.114500+0000) 2024-04-27T04:04:30.366 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:30.365+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:30.366 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:30.365+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:30.654 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:30.422+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:30.423769+0000) 2024-04-27T04:04:30.655 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:30.365+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:30.655 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:30.367+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:30.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:30.654+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:30.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:30.654+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:30.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:30.654+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:30.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:30.655+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:30.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:30.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:30.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:30.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:30.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:30.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:30.917 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:30.835+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:31.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:31.118+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:31.118910+0000) 2024-04-27T04:04:31.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:30.915+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:30.916728+0000) 2024-04-27T04:04:31.635 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:31.425+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:31.425955+0000) 2024-04-27T04:04:31.635 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:31.318+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:31.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:31.318+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:31.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:31.318+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:31.636 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:31.319+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:31.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:31.801+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:31.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:31.801+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:31.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:31.801+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:31.919 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:31.802+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:31.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:31.635+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:31.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:31.635+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:31.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:31.635+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:31.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:31.636+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:31.920 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:31.918+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:31.919790+0000) 2024-04-27T04:04:32.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:32.121+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:32.122062+0000) 2024-04-27T04:04:32.597 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:32.427+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:32.428712+0000) 2024-04-27T04:04:32.597 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:32.361+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:32.597 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:32.361+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:32.598 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:32.361+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:32.598 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:32.362+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:32.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:32.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:32.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:32.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:32.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:32.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:32.923 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:32.597+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:32.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:32.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:32.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:32.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:32.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:32.787+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:32.923 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:32.788+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:33.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:32.921+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:32.922754+0000) 2024-04-27T04:04:33.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:33.125+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:33.126321+0000) 2024-04-27T04:04:33.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:33.431+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:33.432104+0000) 2024-04-27T04:04:33.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:33.387+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:33.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:33.387+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:33.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:33.387+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:33.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:33.388+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:33.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:33.813+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:33.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:33.813+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:33.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:33.813+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:33.925 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:33.813+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:33.925 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:33.611+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:33.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:33.611+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:33.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:33.611+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:33.926 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:33.612+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:34.193 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:04:34.071+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80476 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:04:34.194 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:34.067+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:34.067885+0000) 2024-04-27T04:04:34.194 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:34.066+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:34.067609+0000) 2024-04-27T04:04:34.195 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:34.128+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:34.129396+0000) 2024-04-27T04:04:34.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:33.925+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:33.926005+0000) 2024-04-27T04:04:34.195 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:34.066+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:34.067724+0000) 2024-04-27T04:04:34.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:34.434+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:34.434844+0000) 2024-04-27T04:04:34.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:34.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:34.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:34.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:34.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:34.358+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:34.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:34.360+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:34.929 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:34.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:34.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:34.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:34.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:34.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:34.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:34.635+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:34.930 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:34.811+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:34.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:34.811+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:34.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:34.811+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:34.931 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:34.811+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:35.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:34.929+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:34.930402+0000) 2024-04-27T04:04:35.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:35.131+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:35.132162+0000) 2024-04-27T04:04:35.682 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:35.436+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:35.437046+0000) 2024-04-27T04:04:35.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:35.342+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:35.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:35.342+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:35.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:35.342+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:35.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:35.344+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:35.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:35.681+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:35.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:35.681+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:35.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:35.681+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:35.932 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:35.682+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:35.932 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:35.796+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:35.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:35.796+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:35.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:35.796+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:35.933 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:35.797+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:36.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:35.931+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:35.932778+0000) 2024-04-27T04:04:36.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:36.135+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:36.136404+0000) 2024-04-27T04:04:36.649 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:36.438+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:36.439385+0000) 2024-04-27T04:04:36.650 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:36.343+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:36.650 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:36.343+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:36.650 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:36.343+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:36.650 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:36.344+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:36.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:36.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:36.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:36.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:36.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:36.649+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:36.935 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:36.650+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:36.935 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:36.769+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:36.936 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:36.769+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:36.936 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:36.769+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:36.936 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:36.769+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:37.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:37.140+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:37.141061+0000) 2024-04-27T04:04:37.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:36.934+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:36.935595+0000) 2024-04-27T04:04:37.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:37.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:37.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:37.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:37.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:37.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:37.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:37.383+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:37.648 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:37.441+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:37.442191+0000) 2024-04-27T04:04:37.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:37.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:37.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:37.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:37.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:37.647+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:37.937 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:37.648+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:37.937 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:37.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:37.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:37.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:37.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:37.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:37.938 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:37.775+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:38.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:38.143+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:38.144260+0000) 2024-04-27T04:04:38.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:37.936+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:37.937643+0000) 2024-04-27T04:04:38.606 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:38.444+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:38.445199+0000) 2024-04-27T04:04:38.607 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:38.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:38.607 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:38.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:38.607 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:38.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:38.607 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:38.423+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:38.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:38.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:38.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:38.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:38.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:38.754+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:38.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:38.754+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:38.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:38.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:38.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:38.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:38.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:38.606+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:38.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:38.607+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:39.202 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:39.077+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:39.078269+0000) 2024-04-27T04:04:39.203 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:04:39.081+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80515 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:04:39.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:39.077+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:39.078209+0000) 2024-04-27T04:04:39.203 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:39.145+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:39.146402+0000) 2024-04-27T04:04:39.203 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:38.939+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:38.939898+0000) 2024-04-27T04:04:39.204 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:39.076+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:39.077556+0000) 2024-04-27T04:04:39.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:39.447+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:39.447922+0000) 2024-04-27T04:04:39.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:39.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:39.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:39.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:39.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:39.395+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:39.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:39.396+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:39.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:39.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:39.942 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:39.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:39.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:39.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:39.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:39.558+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:39.943 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:39.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:39.943 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:39.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:39.943 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:39.742+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:39.943 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:39.743+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:40.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:40.149+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:40.150572+0000) 2024-04-27T04:04:40.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:39.942+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:39.943226+0000) 2024-04-27T04:04:40.605 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:40.450+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:40.451086+0000) 2024-04-27T04:04:40.605 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:40.396+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:40.605 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:40.396+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:40.606 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:40.396+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:40.606 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:40.397+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:40.946 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:40.700+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:40.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:40.700+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:40.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:40.700+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:40.947 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:40.700+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:40.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:40.605+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:40.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:40.605+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:40.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:40.605+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:40.947 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:40.606+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:41.213 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:41.152+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:41.153347+0000) 2024-04-27T04:04:41.213 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:40.946+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:40.947535+0000) 2024-04-27T04:04:41.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:41.453+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:41.454043+0000) 2024-04-27T04:04:41.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:41.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:41.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:41.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:41.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:41.417+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:41.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:41.418+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:41.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:41.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:41.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:41.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:41.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:41.724+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:41.950 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:41.724+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:41.950 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:41.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:41.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:41.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:41.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:41.642+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:41.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:41.643+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:41.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:41.949+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:41.950009+0000) 2024-04-27T04:04:42.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:42.157+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:42.158082+0000) 2024-04-27T04:04:42.609 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:42.455+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:42.456190+0000) 2024-04-27T04:04:42.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:42.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:42.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:42.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:42.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:42.461+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:42.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:42.462+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:42.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:42.609+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:42.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:42.609+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:42.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:42.609+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:42.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:42.610+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:42.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:42.694+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:42.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:42.694+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:42.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:42.694+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:42.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:42.694+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:43.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:42.955+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:42.955841+0000) 2024-04-27T04:04:43.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:43.160+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:43.161140+0000) 2024-04-27T04:04:43.502 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:43.458+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:43.458881+0000) 2024-04-27T04:04:43.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:43.502+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:43.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:43.502+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:43.958 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:43.502+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:43.959 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:43.503+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:43.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:43.732+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:43.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:43.732+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:43.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:43.732+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:43.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:43.733+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:43.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:43.565+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:43.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:43.565+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:43.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:43.565+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:43.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:43.566+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:44.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:04:44.090+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80556 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:04:44.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:44.085+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:44.086394+0000) 2024-04-27T04:04:44.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:44.165+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:44.165937+0000) 2024-04-27T04:04:44.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:43.958+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:43.959027+0000) 2024-04-27T04:04:44.210 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:44.085+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:44.086339+0000) 2024-04-27T04:04:44.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:44.085+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:44.085960+0000) 2024-04-27T04:04:44.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:44.460+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:44.461050+0000) 2024-04-27T04:04:44.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:44.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:44.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:44.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:44.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:44.501+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:44.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:44.502+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:44.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:44.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:44.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:44.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:44.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:44.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:44.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:44.600+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:45.127 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:44.962+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:44.963569+0000) 2024-04-27T04:04:45.127 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:44.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:45.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:44.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:45.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:44.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:45.128 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:44.733+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:45.410 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:45.169+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:45.170575+0000) 2024-04-27T04:04:45.686 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:45.462+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:45.463653+0000) 2024-04-27T04:04:45.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:45.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:45.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:45.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:45.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:45.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:45.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:45.507+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:45.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:45.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:45.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:45.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:45.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:45.634+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:45.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:45.635+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:45.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:45.686+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:45.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:45.686+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:45.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:45.686+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:45.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:45.687+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:46.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:46.172+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:46.173713+0000) 2024-04-27T04:04:46.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:45.967+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:45.967848+0000) 2024-04-27T04:04:46.531 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:46.466+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:46.466918+0000) 2024-04-27T04:04:46.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:46.531+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:46.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:46.531+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:46.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:46.531+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:46.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:46.532+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:46.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:46.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:46.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:46.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:46.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:46.614+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:46.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:46.615+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:46.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:46.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:46.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:46.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:46.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:46.733+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:46.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:46.733+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:47.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:46.969+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:46.970632+0000) 2024-04-27T04:04:47.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:47.175+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:47.176488+0000) 2024-04-27T04:04:47.508 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:47.468+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:47.469118+0000) 2024-04-27T04:04:47.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:47.508+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:47.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:47.508+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:47.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:47.508+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:47.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:47.509+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:47.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:47.728+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:47.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:47.728+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:47.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:47.728+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:47.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:47.728+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:47.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:47.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:47.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:47.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:47.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:47.620+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:47.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:47.622+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:48.225 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:48.177+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:48.178764+0000) 2024-04-27T04:04:48.225 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:47.972+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:47.973639+0000) 2024-04-27T04:04:48.485 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:48.472+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:48.473385+0000) 2024-04-27T04:04:48.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:48.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:48.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:48.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:48.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:48.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:48.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:48.486+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:48.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:48.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:48.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:48.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:48.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:48.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:48.973 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:48.667+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:48.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:48.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:48.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:48.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:48.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:48.729+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:48.974 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:48.730+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:49.225 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:49.095+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:49.096290+0000) 2024-04-27T04:04:49.225 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:04:49.100+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80595 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:04:49.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:49.095+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:49.096792+0000) 2024-04-27T04:04:49.226 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:49.180+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:49.180914+0000) 2024-04-27T04:04:49.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:48.974+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:48.975772+0000) 2024-04-27T04:04:49.226 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:49.095+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:49.096539+0000) 2024-04-27T04:04:49.680 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:49.474+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:49.475650+0000) 2024-04-27T04:04:49.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:49.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:49.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:49.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:49.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:49.462+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:49.681 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:49.463+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:49.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:49.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:49.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:49.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:49.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:49.679+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:49.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:49.681+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:49.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:49.702+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:49.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:49.702+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:49.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:49.702+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:49.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:49.702+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:50.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:50.182+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:50.183138+0000) 2024-04-27T04:04:50.301 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:49.977+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:49.977976+0000) 2024-04-27T04:04:50.664 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:50.477+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:50.477909+0000) 2024-04-27T04:04:50.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:50.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:50.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:50.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:50.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:50.443+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:50.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:50.444+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:50.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:50.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:50.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:50.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:50.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:50.722+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:50.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:50.723+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:50.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:50.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:50.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:50.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:50.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:50.664+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:50.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:50.664+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:51.301 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:50.979+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:50.980582+0000) 2024-04-27T04:04:51.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:51.185+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:51.186189+0000) 2024-04-27T04:04:51.665 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:51.479+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:51.480055+0000) 2024-04-27T04:04:51.666 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:51.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:51.666 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:51.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:51.666 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:51.428+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:51.666 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:51.429+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:51.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:51.665+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:51.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:51.665+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:51.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:51.665+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:51.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:51.666+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:51.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:51.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:51.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:51.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:51.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:51.761+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:51.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:51.762+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:52.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:52.187+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:52.188551+0000) 2024-04-27T04:04:52.301 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:51.983+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:51.983900+0000) 2024-04-27T04:04:52.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:52.482+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:52.483144+0000) 2024-04-27T04:04:52.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:52.698+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:52.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:52.698+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:52.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:52.698+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:52.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:52.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:52.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:52.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:52.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:52.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:52.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:52.426+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:52.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:52.699+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:52.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:52.808+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:52.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:52.808+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:52.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:52.808+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:52.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:52.809+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:53.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:53.190+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:53.191372+0000) 2024-04-27T04:04:53.301 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:52.985+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:52.986098+0000) 2024-04-27T04:04:53.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:53.484+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:53.485517+0000) 2024-04-27T04:04:53.698 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:53.698+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:53.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:53.698+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:53.699 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:53.698+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:53.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:53.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:53.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:53.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:53.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:53.388+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:53.699 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:53.390+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:53.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:53.698+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:53.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:53.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:53.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:53.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:53.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:53.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:53.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:53.765+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:54.230 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:04:54.109+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80636 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:04:54.230 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:54.105+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:54.106481+0000) 2024-04-27T04:04:54.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:53.988+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:53.989188+0000) 2024-04-27T04:04:54.231 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:54.106+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:54.107160+0000) 2024-04-27T04:04:54.231 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:54.106+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:54.106845+0000) 2024-04-27T04:04:54.231 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:54.193+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:54.194531+0000) 2024-04-27T04:04:54.489 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:54.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:54.489 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:54.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:54.489 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:54.431+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:54.489 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:54.431+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:54.797 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:54.489+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:54.489871+0000) 2024-04-27T04:04:54.797 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:54.663+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:54.797 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:54.663+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:54.798 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:54.663+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:54.798 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:54.663+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:55.128 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:54.797+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:55.128 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:54.797+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:55.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:54.797+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:55.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:54.798+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:55.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:54.991+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:54.992181+0000) 2024-04-27T04:04:55.383 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:55.195+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:55.196604+0000) 2024-04-27T04:04:55.676 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:55.492+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:55.493142+0000) 2024-04-27T04:04:55.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:55.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:55.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:55.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:55.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:55.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:55.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:55.384+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:55.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:55.677+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:55.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:55.677+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:55.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:55.677+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:55.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:55.677+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:55.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:55.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:55.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:55.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:55.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:55.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:55.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:55.807+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:56.301 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:55.994+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:55.994822+0000) 2024-04-27T04:04:56.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:56.200+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:56.200825+0000) 2024-04-27T04:04:56.573 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:56.495+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:56.495813+0000) 2024-04-27T04:04:56.573 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:56.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:56.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:56.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:56.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:56.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:56.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:56.426+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:56.851 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:56.674+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:56.851 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:56.674+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:56.851 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:56.674+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:56.851 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:56.675+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:57.128 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:56.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:57.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:56.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:57.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:56.851+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:57.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:56.852+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:57.129 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:56.998+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:56.999122+0000) 2024-04-27T04:04:57.401 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:57.202+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:57.203474+0000) 2024-04-27T04:04:57.401 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:57.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:57.401 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:57.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:57.674 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:57.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:57.674 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:57.402+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:57.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:57.499+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:57.500624+0000) 2024-04-27T04:04:57.674 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:57.673+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:57.675 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:57.673+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:57.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:57.674+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:57.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:57.674+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:57.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:57.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:57.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:57.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:57.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:57.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:57.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:57.845+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:58.301 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:58.205+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:58.206429+0000) 2024-04-27T04:04:58.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:58.002+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:58.003437+0000) 2024-04-27T04:04:58.706 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:58.504+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:58.505423+0000) 2024-04-27T04:04:58.706 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:58.706+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:58.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:58.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:58.706 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:58.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:58.707 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:58.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:58.707 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:58.423+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:58.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:58.706+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:58.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:58.706+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:04:58.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:58.706+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:58.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:58.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:58.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:58.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:58.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:58.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:04:58.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:58.836+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:04:59.301 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:04:59.118+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80675 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:04:59.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:59.114+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:59.115680+0000) 2024-04-27T04:04:59.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:59.209+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:59.210663+0000) 2024-04-27T04:04:59.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:59.005+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:59.006354+0000) 2024-04-27T04:04:59.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:59.114+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:59.115357+0000) 2024-04-27T04:04:59.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:59.113+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:59.114785+0000) 2024-04-27T04:04:59.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:59.506+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:04:59.507675+0000) 2024-04-27T04:04:59.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:59.421+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:59.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:59.421+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:59.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:59.421+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:04:59.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:04:59.422+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:00.008 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:59.738+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:00.009 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:59.738+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:00.009 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:59.738+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:00.009 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:04:59.738+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:00.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:59.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:00.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:59.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:00.009 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:59.869+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:00.010 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:04:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:04:59.870+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:00.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:00.212+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:00.212940+0000) 2024-04-27T04:05:00.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:00.008+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:00.009626+0000) 2024-04-27T04:05:00.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:00.511+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:00.511909+0000) 2024-04-27T04:05:00.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:00.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:00.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:00.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:00.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:00.399+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:00.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:00.400+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:01.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:00.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:01.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:00.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:01.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:00.784+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:01.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:00.785+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:01.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:00.880+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:01.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:00.880+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:01.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:00.880+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:01.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:00.882+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:01.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:01.011+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:01.011956+0000) 2024-04-27T04:05:01.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:01.214+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:01.215536+0000) 2024-04-27T04:05:01.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:01.514+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:01.515095+0000) 2024-04-27T04:05:01.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:01.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:01.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:01.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:01.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:01.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:01.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:01.426+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:02.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:01.809+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:02.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:01.809+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:02.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:01.809+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:02.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:01.810+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:02.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:01.868+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:02.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:01.868+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:02.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:01.868+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:02.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:01.868+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:02.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:02.015+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:02.016283+0000) 2024-04-27T04:05:02.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:02.219+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:02.220326+0000) 2024-04-27T04:05:02.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:02.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:02.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:02.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:02.413 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:02.377+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:02.413 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:02.378+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:02.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:02.517+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:02.517814+0000) 2024-04-27T04:05:02.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:02.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:02.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:02.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:02.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:02.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:02.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:02.855+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:02.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:02.770+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:02.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:02.770+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:02.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:02.770+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:02.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:02.770+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:03.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:03.221+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:03.222608+0000) 2024-04-27T04:05:03.302 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:03.019+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:03.020619+0000) 2024-04-27T04:05:03.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:03.520+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:03.521218+0000) 2024-04-27T04:05:03.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:03.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:03.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:03.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:03.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:03.405+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:03.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:03.406+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:04.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:03.837+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:04.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:03.837+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:04.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:03.837+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:04.024 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:03.838+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:04.025 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:03.771+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:04.025 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:03.771+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:04.025 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:03.771+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:04.025 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:03.772+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:04.302 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:04.123+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:04.124039+0000) 2024-04-27T04:05:04.303 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:05:04.127+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80716 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:05:04.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:04.123+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:04.123842+0000) 2024-04-27T04:05:04.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:04.224+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:04.224928+0000) 2024-04-27T04:05:04.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:04.024+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:04.025000+0000) 2024-04-27T04:05:04.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:04.124+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:04.124865+0000) 2024-04-27T04:05:04.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:04.523+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:04.523928+0000) 2024-04-27T04:05:04.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:04.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:04.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:04.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:04.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:04.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:04.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:04.384+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:05.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:04.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:05.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:04.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:05.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:04.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:05.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:04.786+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:05.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:04.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:05.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:04.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:05.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:04.806+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:05.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:04.807+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:05.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:05.027+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:05.027839+0000) 2024-04-27T04:05:05.412 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:05.226+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:05.227520+0000) 2024-04-27T04:05:05.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:05.527+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:05.528177+0000) 2024-04-27T04:05:05.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:05.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:05.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:05.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:05.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:05.414+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:05.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:05.415+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:06.129 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:05.786+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:06.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:05.786+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:06.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:05.786+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:06.130 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:05.786+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:06.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:05.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:06.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:05.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:06.130 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:05.830+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:06.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:05.831+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:06.131 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:06.030+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:06.031025+0000) 2024-04-27T04:05:06.384 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:06.229+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:06.229942+0000) 2024-04-27T04:05:06.384 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:06.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:06.384 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:06.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:06.384 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:06.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:06.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:06.384+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:06.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:06.531+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:06.532923+0000) 2024-04-27T04:05:07.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:06.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:07.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:06.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:07.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:06.822+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:07.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:06.824+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:07.034 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:06.738+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:07.034 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:06.738+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:07.034 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:06.738+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:07.034 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:06.738+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:07.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:07.230+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:07.232143+0000) 2024-04-27T04:05:07.303 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:07.031+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:07.033789+0000) 2024-04-27T04:05:07.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:07.536+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:07.537816+0000) 2024-04-27T04:05:07.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:07.408+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:07.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:07.408+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:07.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:07.408+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:07.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:07.409+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:08.035 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:07.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:08.035 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:07.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:08.035 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:07.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:08.036 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:07.775+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:08.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:07.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:08.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:07.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:08.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:07.735+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:08.036 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:07.735+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:08.302 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:08.232+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:08.234332+0000) 2024-04-27T04:05:08.303 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:08.034+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:08.036343+0000) 2024-04-27T04:05:08.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:08.538+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:08.540499+0000) 2024-04-27T04:05:08.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:08.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:08.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:08.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:08.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:08.455+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:08.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:08.457+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:08.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:08.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:08.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:08.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:08.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:08.710+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:08.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:08.711+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:08.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:08.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:08.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:08.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:08.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:08.729+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:08.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:08.731+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:09.302 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:05:09.135+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80755 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:05:09.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:09.130+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:09.132791+0000) 2024-04-27T04:05:09.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:09.236+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:09.238582+0000) 2024-04-27T04:05:09.303 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:09.131+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:09.133604+0000) 2024-04-27T04:05:09.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:09.036+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:09.038787+0000) 2024-04-27T04:05:09.304 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:09.133+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:09.134944+0000) 2024-04-27T04:05:09.677 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:09.542+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:09.544744+0000) 2024-04-27T04:05:09.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:09.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:09.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:09.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:09.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:09.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:09.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:09.471+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:09.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:09.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:09.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:09.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:09.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:09.745+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:09.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:09.746+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:09.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:09.676+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:09.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:09.676+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:09.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:09.676+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:09.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:09.676+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:10.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:10.239+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:10.240815+0000) 2024-04-27T04:05:10.303 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:10.039+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:10.040845+0000) 2024-04-27T04:05:10.664 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:10.545+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:10.546978+0000) 2024-04-27T04:05:10.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:10.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:10.664 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:10.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:10.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:10.478+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:10.665 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:10.479+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:10.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:10.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:10.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:10.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:10.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:10.699+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:10.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:10.700+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:10.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:10.663+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:10.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:10.663+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:10.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:10.663+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:10.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:10.663+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:11.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:11.243+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:11.245792+0000) 2024-04-27T04:05:11.303 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:11.041+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:11.042998+0000) 2024-04-27T04:05:11.629 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:11.549+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:11.551245+0000) 2024-04-27T04:05:11.629 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:11.499+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:11.629 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:11.499+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:11.630 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:11.499+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:11.630 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:11.500+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:11.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:11.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:11.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:11.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:11.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:11.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:11.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:11.662+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:11.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:11.628+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:11.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:11.628+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:11.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:11.628+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:11.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:11.629+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:12.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:12.043+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:12.045178+0000) 2024-04-27T04:05:12.508 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:12.248+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:12.250343+0000) 2024-04-27T04:05:12.508 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:12.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:12.509 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:12.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:12.509 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:12.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:12.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:12.508+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:12.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:12.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:12.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:12.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:12.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:12.618+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:12.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:12.618+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:12.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:12.551+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:12.553557+0000) 2024-04-27T04:05:12.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:12.653+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:12.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:12.653+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:12.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:12.653+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:12.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:12.654+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:13.303 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:13.045+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:13.047422+0000) 2024-04-27T04:05:13.303 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:13.252+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:13.254633+0000) 2024-04-27T04:05:13.555 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:13.522+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:13.555 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:13.522+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:13.555 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:13.522+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:13.556 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:13.523+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:13.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:13.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:13.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:13.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:13.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:13.589+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:13.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:13.589+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:13.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:13.554+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:13.555915+0000) 2024-04-27T04:05:13.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:13.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:13.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:13.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:13.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:13.693+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:13.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:13.693+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:14.255 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:14.141+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:14.143650+0000) 2024-04-27T04:05:14.256 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:05:14.146+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80796 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:05:14.256 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:14.140+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:14.142113+0000) 2024-04-27T04:05:14.257 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:14.047+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:14.049771+0000) 2024-04-27T04:05:14.257 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:14.141+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:14.142836+0000) 2024-04-27T04:05:14.508 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:14.255+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:14.256799+0000) 2024-04-27T04:05:14.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:14.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:14.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:14.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:14.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:14.631+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:14.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:14.631+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:14.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:14.558+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:14.559943+0000) 2024-04-27T04:05:14.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:14.672+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:14.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:14.672+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:14.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:14.672+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:14.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:14.672+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:14.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:14.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:14.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:14.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:14.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:14.507+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:14.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:14.509+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:15.258 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:15.050+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:15.052070+0000) 2024-04-27T04:05:15.515 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:15.257+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:15.259174+0000) 2024-04-27T04:05:15.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:15.641+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:15.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:15.641+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:15.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:15.641+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:15.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:15.642+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:15.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:15.560+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:15.562313+0000) 2024-04-27T04:05:15.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:15.674+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:15.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:15.674+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:15.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:15.674+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:15.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:15.675+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:15.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:15.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:15.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:15.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:15.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:15.514+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:15.972 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:15.516+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:16.368 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:16.052+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:16.054761+0000) 2024-04-27T04:05:16.368 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:16.260+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:16.261983+0000) 2024-04-27T04:05:16.655 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:16.562+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:16.564563+0000) 2024-04-27T04:05:16.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:16.654+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:16.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:16.654+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:16.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:16.654+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:16.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:16.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:16.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:16.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:16.657 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:16.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:16.657 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:16.479+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:16.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:16.655+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:16.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:16.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:16.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:16.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:16.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:16.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:16.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:16.676+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:17.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:17.055+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:17.057667+0000) 2024-04-27T04:05:17.414 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:17.262+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:17.264305+0000) 2024-04-27T04:05:17.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:17.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:17.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:17.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:17.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:17.475+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:17.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:17.476+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:17.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:17.565+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:17.566859+0000) 2024-04-27T04:05:17.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:17.617+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:17.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:17.617+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:17.704 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:17.617+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:17.705 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:17.617+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:17.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:17.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:17.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:17.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:17.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:17.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:17.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:17.704+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:18.414 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:18.264+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:18.266405+0000) 2024-04-27T04:05:18.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:18.058+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:18.059977+0000) 2024-04-27T04:05:18.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:18.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:18.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:18.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:18.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:18.516+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:18.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:18.517+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:18.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:18.571+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:18.572012+0000) 2024-04-27T04:05:18.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:18.614+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:18.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:18.614+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:18.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:18.614+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:18.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:18.614+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:19.061 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:18.749+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:19.061 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:18.749+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:19.061 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:18.749+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:19.061 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:18.749+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:19.414 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:05:19.157+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80835 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:05:19.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:19.061+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:19.062249+0000) 2024-04-27T04:05:19.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:19.153+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:19.154630+0000) 2024-04-27T04:05:19.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:19.152+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:19.153357+0000) 2024-04-27T04:05:19.415 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:19.152+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:19.152922+0000) 2024-04-27T04:05:19.415 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:19.267+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:19.268562+0000) 2024-04-27T04:05:19.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:19.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:19.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:19.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:19.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:19.520+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:19.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:19.521+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:19.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:19.573+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:19.574779+0000) 2024-04-27T04:05:19.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:19.591+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:19.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:19.591+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:19.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:19.591+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:19.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:19.592+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:20.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:19.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:20.063 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:19.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:20.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:19.766+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:20.064 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:19.767+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:20.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:20.063+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:20.064474+0000) 2024-04-27T04:05:20.414 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:20.272+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:20.273538+0000) 2024-04-27T04:05:20.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:20.577+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:20.577994+0000) 2024-04-27T04:05:20.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:20.606+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:20.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:20.606+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:20.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:20.606+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:20.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:20.606+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:20.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:20.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:20.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:20.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:20.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:20.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:20.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:20.512+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:21.065 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:20.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:21.066 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:20.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:21.066 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:20.724+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:21.066 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:20.725+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:21.414 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:21.277+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:21.278405+0000) 2024-04-27T04:05:21.414 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:21.065+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:21.066702+0000) 2024-04-27T04:05:21.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:21.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:21.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:21.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:21.693 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:21.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:21.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:21.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:21.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:21.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:21.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:21.561+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:21.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:21.562+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:21.694 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:21.579+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:21.580674+0000) 2024-04-27T04:05:21.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:21.638+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:21.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:21.638+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:21.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:21.638+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:21.695 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:21.639+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:21.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:21.694+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:22.414 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:22.282+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:22.283442+0000) 2024-04-27T04:05:22.415 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:22.068+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:22.069127+0000) 2024-04-27T04:05:22.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:22.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:22.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:22.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:22.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:22.546+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:22.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:22.547+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:22.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:22.582+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:22.582981+0000) 2024-04-27T04:05:22.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:22.593+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:22.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:22.593+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:22.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:22.593+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:22.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:22.593+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:22.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:22.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:22.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:22.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:22.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:22.648+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:22.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:22.649+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:23.414 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:23.287+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:23.288490+0000) 2024-04-27T04:05:23.415 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:23.070+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:23.071467+0000) 2024-04-27T04:05:23.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:23.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:23.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:23.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:23.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:23.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:23.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:23.583+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:23.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:23.584+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:23.585326+0000) 2024-04-27T04:05:23.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:23.630+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:23.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:23.630+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:23.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:23.631+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:23.721 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:23.631+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:23.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:23.600+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:23.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:23.600+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:23.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:23.600+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:23.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:23.601+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:24.289 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:24.162+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:24.163488+0000) 2024-04-27T04:05:24.289 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:05:24.168+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80876 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:05:24.289 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:24.073+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:24.074417+0000) 2024-04-27T04:05:24.290 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:24.163+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:24.163844+0000) 2024-04-27T04:05:24.290 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:24.161+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:24.162582+0000) 2024-04-27T04:05:24.586 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:24.292+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:24.293544+0000) 2024-04-27T04:05:24.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:24.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:24.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:24.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:24.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:24.602+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:24.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:24.602+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:24.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:24.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:24.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:24.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:24.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:24.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:24.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:24.621+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:24.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:24.586+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:24.587769+0000) 2024-04-27T04:05:24.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:24.680+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:24.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:24.681+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:24.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:24.681+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:24.972 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:24.681+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:25.415 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:25.077+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:25.078774+0000) 2024-04-27T04:05:25.415 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:25.294+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:25.295709+0000) 2024-04-27T04:05:25.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:25.589+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:25.590170+0000) 2024-04-27T04:05:25.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:25.609+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:25.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:25.609+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:25.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:25.609+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:25.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:25.610+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:25.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:25.599+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:25.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:25.599+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:25.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:25.599+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:25.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:25.600+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:26.082 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:25.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:26.082 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:25.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:26.082 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:25.730+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:26.082 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:25.731+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:26.366 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:26.297+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:26.297932+0000) 2024-04-27T04:05:26.367 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:26.082+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:26.083050+0000) 2024-04-27T04:05:26.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:26.592+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:26.593085+0000) 2024-04-27T04:05:26.627 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:26.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:26.627 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:26.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:26.627 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:26.615+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:26.627 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:26.616+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:26.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:26.626+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:26.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:26.626+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:26.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:26.626+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:26.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:26.627+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:26.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:26.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:26.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:26.777+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:26.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:26.778+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:26.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:26.778+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:27.415 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:27.301+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:27.302185+0000) 2024-04-27T04:05:27.415 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:27.086+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:27.087122+0000) 2024-04-27T04:05:27.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:27.594+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:27.595310+0000) 2024-04-27T04:05:27.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:27.644+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:27.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:27.644+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:27.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:27.644+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:27.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:27.646+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:27.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:27.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:27.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:27.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:27.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:27.597+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:27.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:27.598+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:28.089 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:27.801+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:28.089 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:27.801+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:28.089 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:27.801+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:28.089 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:27.801+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:28.415 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:28.303+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:28.304493+0000) 2024-04-27T04:05:28.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:28.089+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:28.090211+0000) 2024-04-27T04:05:28.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:28.596+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:28.597529+0000) 2024-04-27T04:05:28.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:28.651+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:28.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:28.651+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:28.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:28.651+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:28.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:28.653+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:28.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:28.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:28.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:28.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:28.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:28.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:28.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:28.633+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:29.132 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:29.091+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:29.092562+0000) 2024-04-27T04:05:29.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:28.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:29.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:28.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:29.132 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:28.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:29.133 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:28.851+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:29.415 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:05:29.177+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80915 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:05:29.415 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:29.172+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:29.173551+0000) 2024-04-27T04:05:29.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:29.174+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:29.175266+0000) 2024-04-27T04:05:29.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:29.174+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:29.175203+0000) 2024-04-27T04:05:29.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:29.305+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:29.306656+0000) 2024-04-27T04:05:29.713 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:29.598+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:29.599792+0000) 2024-04-27T04:05:29.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:29.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:29.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:29.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:29.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:29.643+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:29.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:29.644+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:29.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:29.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:29.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:29.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:29.714 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:29.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:29.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:29.593+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:29.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:29.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:29.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:29.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:29.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:29.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:29.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:29.846+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:30.365 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:30.309+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:30.309824+0000) 2024-04-27T04:05:30.365 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:30.094+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:30.095714+0000) 2024-04-27T04:05:30.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:30.601+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:30.601961+0000) 2024-04-27T04:05:30.619 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:30.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:30.619 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:30.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:30.619 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:30.598+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:30.619 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:30.599+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:30.889 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:30.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:30.891 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:30.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:30.891 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:30.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:30.891 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:30.619+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:31.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:31.097+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:31.097831+0000) 2024-04-27T04:05:31.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:30.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:31.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:30.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:31.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:30.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:31.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:30.890+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:31.495 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:31.311+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:31.312032+0000) 2024-04-27T04:05:31.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:31.603+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:31.604070+0000) 2024-04-27T04:05:31.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:31.631+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:31.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:31.631+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:31.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:31.631+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:31.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:31.631+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:31.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:31.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:31.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:31.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:31.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:31.559+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:31.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:31.559+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:32.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:32.099+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:32.099911+0000) 2024-04-27T04:05:32.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:31.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:32.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:31.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:32.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:31.897+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:32.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:31.898+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:32.497 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:32.313+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:32.314665+0000) 2024-04-27T04:05:32.901 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:32.605+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:32.606318+0000) 2024-04-27T04:05:32.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:32.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:32.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:32.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:32.902 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:32.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:32.902 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:32.659+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:32.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:32.608+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:32.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:32.608+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:32.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:32.608+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:32.902 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:32.609+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:33.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:33.101+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:33.102595+0000) 2024-04-27T04:05:33.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:32.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:33.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:32.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:33.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:32.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:33.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:32.902+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:33.497 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:33.317+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:33.317933+0000) 2024-04-27T04:05:33.891 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:33.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:33.892 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:33.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:33.892 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:33.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:33.892 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:33.627+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:33.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:33.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:33.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:33.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:33.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:33.632+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:33.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:33.634+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:33.893 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:33.608+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:33.609323+0000) 2024-04-27T04:05:33.893 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:33.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:33.893 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:33.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:33.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:33.891+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:34.178 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:34.106+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:34.107737+0000) 2024-04-27T04:05:34.179 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:33.892+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:34.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:34.182+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:34.183230+0000) 2024-04-27T04:05:34.470 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:05:34.187+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80956 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:05:34.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:34.182+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:34.183265+0000) 2024-04-27T04:05:34.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:34.322+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:34.323075+0000) 2024-04-27T04:05:34.471 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:34.182+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:34.183015+0000) 2024-04-27T04:05:34.770 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:34.610+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:34.611650+0000) 2024-04-27T04:05:34.770 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:34.652+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:34.770 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:34.652+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:34.771 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:34.652+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:34.771 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:34.653+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:34.771 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:34.605+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:34.771 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:34.605+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:34.771 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:34.605+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:34.771 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:34.606+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:35.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:34.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:35.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:34.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:35.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:34.845+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:35.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:34.846+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:35.415 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:35.112+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:35.112848+0000) 2024-04-27T04:05:35.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:35.325+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:35.326095+0000) 2024-04-27T04:05:35.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:35.615+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:35.616407+0000) 2024-04-27T04:05:35.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:35.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:35.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:35.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:35.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:35.625+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:35.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:35.626+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:35.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:35.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:35.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:35.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:35.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:35.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:35.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:35.570+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:36.115 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:35.813+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:36.115 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:35.813+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:36.115 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:35.813+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:36.115 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:35.813+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:36.366 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:36.327+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:36.328245+0000) 2024-04-27T04:05:36.367 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:36.115+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:36.116158+0000) 2024-04-27T04:05:36.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:36.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:36.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:36.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:36.618 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:36.590+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:36.619 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:36.591+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:36.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:36.656+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:36.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:36.656+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:36.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:36.656+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:36.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:36.657+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:36.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:36.618+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:36.619318+0000) 2024-04-27T04:05:36.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:36.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:36.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:36.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:36.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:36.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:36.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:36.801+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:37.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:37.329+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:37.330382+0000) 2024-04-27T04:05:37.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:37.118+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:37.118999+0000) 2024-04-27T04:05:37.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:37.622+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:37.623591+0000) 2024-04-27T04:05:37.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:37.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:37.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:37.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:37.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:37.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:37.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:37.647+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:37.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:37.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:37.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:37.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:37.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:37.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:37.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:37.592+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:38.120 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:37.838+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:38.120 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:37.838+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:38.120 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:37.838+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:38.121 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:37.839+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:38.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:38.332+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:38.333183+0000) 2024-04-27T04:05:38.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:38.120+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:38.121292+0000) 2024-04-27T04:05:38.667 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:38.625+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:38.626385+0000) 2024-04-27T04:05:38.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:38.621+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:38.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:38.621+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:38.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:38.621+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:38.668 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:38.621+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:38.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:38.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:38.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:38.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:38.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:38.667+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:38.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:38.668+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:38.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:38.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:38.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:38.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:38.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:38.844+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:38.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:38.844+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:39.416 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:05:39.197+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 80995 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:05:39.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:39.192+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:39.193745+0000) 2024-04-27T04:05:39.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:39.193+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:39.193831+0000) 2024-04-27T04:05:39.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:39.335+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:39.336292+0000) 2024-04-27T04:05:39.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:39.123+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:39.124250+0000) 2024-04-27T04:05:39.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:39.192+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:39.193361+0000) 2024-04-27T04:05:39.670 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:39.628+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:39.629299+0000) 2024-04-27T04:05:39.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:39.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:39.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:39.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:39.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:39.575+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:39.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:39.576+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:39.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:39.670+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:39.672 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:39.670+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:39.672 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:39.670+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:39.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:39.672+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:39.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:39.801+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:39.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:39.801+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:39.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:39.801+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:39.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:39.802+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:40.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:40.337+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:40.338369+0000) 2024-04-27T04:05:40.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:40.126+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:40.126920+0000) 2024-04-27T04:05:40.708 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:40.630+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:40.631502+0000) 2024-04-27T04:05:40.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:40.623+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:40.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:40.623+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:40.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:40.623+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:40.709 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:40.624+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:40.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:40.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:40.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:40.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:40.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:40.798+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:40.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:40.798+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:40.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:40.708+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:40.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:40.708+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:40.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:40.708+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:40.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:40.710+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:41.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:41.342+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:41.342956+0000) 2024-04-27T04:05:41.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:41.130+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:41.131193+0000) 2024-04-27T04:05:41.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:41.635+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:41.635812+0000) 2024-04-27T04:05:41.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:41.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:41.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:41.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:41.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:41.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:41.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:41.609+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:42.133 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:41.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:42.134 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:41.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:42.134 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:41.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:42.134 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:41.725+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:42.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:41.792+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:42.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:41.792+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:42.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:41.792+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:42.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:41.793+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:42.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:42.344+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:42.345048+0000) 2024-04-27T04:05:42.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:42.134+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:42.135186+0000) 2024-04-27T04:05:42.717 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:42.637+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:42.638706+0000) 2024-04-27T04:05:42.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:42.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:42.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:42.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:42.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:42.591+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:42.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:42.592+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:42.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:42.718+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:42.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:42.718+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:42.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:42.718+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:42.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:42.719+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:42.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:42.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:42.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:42.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:42.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:42.841+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:42.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:42.842+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:43.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:43.346+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:43.347285+0000) 2024-04-27T04:05:43.419 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:43.138+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:43.138819+0000) 2024-04-27T04:05:43.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:43.640+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:43.640880+0000) 2024-04-27T04:05:43.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:43.602+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:43.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:43.602+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:43.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:43.602+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:43.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:43.603+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:44.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:43.801+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:44.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:43.801+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:44.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:43.801+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:44.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:43.802+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:44.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:43.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:44.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:43.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:44.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:43.759+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:44.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:43.761+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:44.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:44.202+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:44.203214+0000) 2024-04-27T04:05:44.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:44.348+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:44.349435+0000) 2024-04-27T04:05:44.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:44.201+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:44.202752+0000) 2024-04-27T04:05:44.417 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:05:44.210+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81036 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:05:44.418 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:44.140+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:44.141009+0000) 2024-04-27T04:05:44.418 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:44.203+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:44.204350+0000) 2024-04-27T04:05:44.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:44.642+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:44.643757+0000) 2024-04-27T04:05:44.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:44.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:44.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:44.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:44.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:44.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:44.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:44.564+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:45.134 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:44.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:45.134 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:44.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:45.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:44.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:45.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:44.756+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:45.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:44.833+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:45.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:44.833+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:45.136 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:44.833+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:45.136 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:44.833+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:45.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:45.350+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:45.351650+0000) 2024-04-27T04:05:45.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:45.142+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:45.143769+0000) 2024-04-27T04:05:45.712 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:45.645+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:45.645927+0000) 2024-04-27T04:05:45.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:45.528+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:45.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:45.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:45.713 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:45.529+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:45.714 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:45.530+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:45.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:45.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:45.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:45.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:45.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:45.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:45.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:45.713+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:45.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:45.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:45.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:45.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:45.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:45.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:45.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:45.785+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:46.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:46.353+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:46.354467+0000) 2024-04-27T04:05:46.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:46.145+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:46.146444+0000) 2024-04-27T04:05:46.684 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:46.647+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:46.648181+0000) 2024-04-27T04:05:46.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:46.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:46.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:46.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:46.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:46.511+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:46.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:46.513+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:46.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:46.781+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:46.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:46.781+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:46.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:46.781+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:46.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:46.781+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:46.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:46.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:46.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:46.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:46.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:46.685+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:46.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:46.685+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:47.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:47.356+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:47.357332+0000) 2024-04-27T04:05:47.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:47.148+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:47.149336+0000) 2024-04-27T04:05:47.692 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:47.649+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:47.650404+0000) 2024-04-27T04:05:47.692 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:47.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:47.692 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:47.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:47.692 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:47.534+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:47.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:47.536+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:47.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:47.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:47.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:47.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:47.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:47.692+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:47.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:47.693+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:47.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:47.783+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:47.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:47.783+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:47.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:47.783+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:47.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:47.783+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:48.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:48.151+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:48.152267+0000) 2024-04-27T04:05:48.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:48.359+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:48.360241+0000) 2024-04-27T04:05:48.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:48.652+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:48.653309+0000) 2024-04-27T04:05:48.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:48.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:48.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:48.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:48.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:48.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:48.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:48.660+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:48.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:48.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:48.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:48.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:48.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:48.524+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:48.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:48.526+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:49.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:48.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:49.134 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:48.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:49.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:48.774+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:49.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:48.774+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:49.417 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:05:49.222+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81075 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:05:49.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:49.215+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:49.215833+0000) 2024-04-27T04:05:49.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:49.215+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:49.215966+0000) 2024-04-27T04:05:49.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:49.363+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:49.364536+0000) 2024-04-27T04:05:49.418 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:49.154+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:49.154904+0000) 2024-04-27T04:05:49.418 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:49.215+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:49.216767+0000) 2024-04-27T04:05:49.703 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:49.655+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:49.656278+0000) 2024-04-27T04:05:49.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:49.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:49.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:49.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:49.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:49.527+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:49.704 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:49.529+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:49.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:49.704+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:49.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:49.704+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:49.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:49.704+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:49.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:49.705+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:49.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:49.788+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:49.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:49.788+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:49.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:49.788+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:49.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:49.789+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:50.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:50.157+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:50.158068+0000) 2024-04-27T04:05:50.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:50.368+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:50.368986+0000) 2024-04-27T04:05:50.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:50.659+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:50.660488+0000) 2024-04-27T04:05:50.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:50.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:50.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:50.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:50.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:50.485+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:50.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:50.486+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:51.134 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:50.732+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:51.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:50.732+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:51.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:50.732+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:51.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:50.733+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:51.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:50.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:51.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:50.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:51.136 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:50.820+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:51.136 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:50.821+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:51.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:51.161+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:51.162422+0000) 2024-04-27T04:05:51.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:51.372+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:51.373792+0000) 2024-04-27T04:05:51.693 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:51.661+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:51.662764+0000) 2024-04-27T04:05:51.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:51.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:51.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:51.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:51.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:51.452+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:51.694 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:51.453+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:51.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:51.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:51.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:51.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:51.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:51.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:51.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:51.694+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:51.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:51.795+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:51.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:51.795+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:51.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:51.795+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:51.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:51.796+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:52.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:52.165+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:52.166551+0000) 2024-04-27T04:05:52.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:52.378+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:52.378893+0000) 2024-04-27T04:05:52.711 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:52.664+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:52.665202+0000) 2024-04-27T04:05:52.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:52.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:52.711 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:52.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:52.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:52.489+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:52.712 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:52.490+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:52.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:52.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:52.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:52.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:52.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:52.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:52.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:52.712+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:52.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:52.811+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:52.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:52.811+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:52.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:52.811+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:52.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:52.811+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:53.417 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:53.169+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:53.169919+0000) 2024-04-27T04:05:53.418 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:53.383+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:53.383986+0000) 2024-04-27T04:05:53.696 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:53.666+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:53.667575+0000) 2024-04-27T04:05:53.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:53.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:53.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:53.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:53.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:53.459+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:53.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:53.460+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:53.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:53.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:53.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:53.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:53.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:53.696+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:53.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:53.697+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:53.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:53.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:53.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:53.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:53.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:53.821+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:53.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:53.821+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:54.232 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:05:54.231+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81116 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:05:54.232 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:54.226+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:54.227749+0000) 2024-04-27T04:05:54.232 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:54.226+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:54.227581+0000) 2024-04-27T04:05:54.233 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:54.173+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:54.174306+0000) 2024-04-27T04:05:54.233 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:54.227+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:54.228323+0000) 2024-04-27T04:05:54.531 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:54.388+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:54.388892+0000) 2024-04-27T04:05:54.532 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:54.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:54.532 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:54.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:54.532 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:54.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:54.532 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:54.427+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:54.800 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:54.669+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:54.669975+0000) 2024-04-27T04:05:54.800 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:54.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:54.800 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:54.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:54.800 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:54.741+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:54.800 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:54.742+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:55.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:54.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:55.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:54.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:55.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:54.800+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:55.135 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:54.800+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:55.390 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:55.175+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:55.176408+0000) 2024-04-27T04:05:55.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:55.391+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:55.392067+0000) 2024-04-27T04:05:55.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:55.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:55.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:55.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:55.673 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:55.470+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:55.674 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:55.471+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:55.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:55.673+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:55.674228+0000) 2024-04-27T04:05:55.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:55.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:55.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:55.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:55.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:55.850+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:55.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:55.850+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:55.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:55.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:55.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:55.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:55.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:55.757+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:55.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:55.758+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:56.393 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:56.177+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:56.178504+0000) 2024-04-27T04:05:56.676 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:56.394+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:56.394941+0000) 2024-04-27T04:05:56.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:56.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:56.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:56.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:56.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:56.426+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:56.677 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:56.426+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:56.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:56.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:56.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:56.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:56.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:56.717+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:56.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:56.718+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:56.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:56.676+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:56.677607+0000) 2024-04-27T04:05:56.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:56.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:56.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:56.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:56.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:56.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:56.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:56.827+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:57.396 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:57.182+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:57.182840+0000) 2024-04-27T04:05:57.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:57.396+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:57.397480+0000) 2024-04-27T04:05:57.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:57.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:57.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:57.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:57.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:57.448+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:57.671 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:57.450+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:57.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:57.670+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:57.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:57.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:57.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:57.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:57.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:57.672+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:57.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:57.679+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:57.679969+0000) 2024-04-27T04:05:57.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:57.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:57.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:57.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:57.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:57.794+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:57.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:57.795+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:58.398 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:58.185+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:58.186118+0000) 2024-04-27T04:05:58.674 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:58.399+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:58.399797+0000) 2024-04-27T04:05:58.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:58.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:58.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:58.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:58.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:58.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:58.675 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:58.443+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:58.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:58.682+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:58.683183+0000) 2024-04-27T04:05:58.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:58.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:58.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:58.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:58.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:58.824+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:58.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:58.825+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:58.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:58.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:58.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:58.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:58.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:58.674+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:58.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:58.676+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:59.400 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:05:59.241+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81155 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:05:59.401 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:59.236+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:59.236918+0000) 2024-04-27T04:05:59.401 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:59.189+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:59.190481+0000) 2024-04-27T04:05:59.401 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:59.236+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:59.237616+0000) 2024-04-27T04:05:59.402 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:59.236+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:59.237567+0000) 2024-04-27T04:05:59.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:59.401+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:59.401992+0000) 2024-04-27T04:05:59.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:59.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:59.685 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:59.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:59.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:59.487+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:05:59.686 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:05:59.488+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:59.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:59.687+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:59.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:59.687+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:59.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:59.687+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:05:59.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:05:59.688+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:05:59.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:59.685+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:05:59.686319+0000) 2024-04-27T04:05:59.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:59.872+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:59.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:59.872+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:59.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:59.872+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:05:59.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:05:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:05:59.872+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:00.308 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:00.191+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:00.192636+0000) 2024-04-27T04:06:00.682 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:00.404+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:00.405194+0000) 2024-04-27T04:06:00.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:00.498+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:00.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:00.498+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:00.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:00.498+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:00.683 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:00.499+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:00.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:00.687+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:00.688594+0000) 2024-04-27T04:06:00.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:00.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:00.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:00.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:00.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:00.914+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:00.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:00.914+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:00.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:00.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:00.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:00.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:00.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:00.682+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:00.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:00.683+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:01.406 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:01.194+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:01.194826+0000) 2024-04-27T04:06:01.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:01.406+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:01.407435+0000) 2024-04-27T04:06:01.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:01.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:01.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:01.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:01.691 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:01.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:01.692 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:01.479+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:01.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:01.691+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:01.691801+0000) 2024-04-27T04:06:01.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:01.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:01.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:01.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:01.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:01.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:01.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:01.902+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:01.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:01.697+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:01.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:01.697+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:01.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:01.697+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:01.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:01.698+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:02.408 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:02.196+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:02.197453+0000) 2024-04-27T04:06:02.662 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:02.408+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:02.409687+0000) 2024-04-27T04:06:02.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:02.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:02.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:02.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:02.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:02.441+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:02.663 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:02.443+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:02.663 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:02.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:02.663 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:02.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:02.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:02.662+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:02.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:02.663+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:02.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:02.693+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:02.694328+0000) 2024-04-27T04:06:02.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:02.909+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:02.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:02.909+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:02.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:02.909+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:02.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:02.910+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:03.410 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:03.200+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:03.200894+0000) 2024-04-27T04:06:03.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:03.410+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:03.411774+0000) 2024-04-27T04:06:03.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:03.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:03.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:03.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:03.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:03.442+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:03.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:03.443+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:03.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:03.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:03.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:03.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:03.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:03.711+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:03.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:03.712+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:03.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:03.698+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:03.699050+0000) 2024-04-27T04:06:03.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:03.920+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:03.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:03.920+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:03.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:03.920+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:03.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:03.921+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:04.243 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:04.203+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:04.204109+0000) 2024-04-27T04:06:04.550 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:06:04.251+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81196 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:06:04.550 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:04.247+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:04.248050+0000) 2024-04-27T04:06:04.551 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:04.246+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:04.246926+0000) 2024-04-27T04:06:04.551 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:04.413+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:04.414296+0000) 2024-04-27T04:06:04.551 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:04.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:04.551 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:04.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:04.551 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:04.464+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:04.551 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:04.465+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:04.551 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:04.246+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:04.247793+0000) 2024-04-27T04:06:04.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:04.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:04.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:04.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:04.943 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:04.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:04.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:04.694+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:04.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:04.701+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:04.702095+0000) 2024-04-27T04:06:04.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:04.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:04.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:04.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:04.944 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:04.943+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:05.205 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:04.944+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:05.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:05.205+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:05.206240+0000) 2024-04-27T04:06:05.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:05.418+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:05.419324+0000) 2024-04-27T04:06:05.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:05.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:05.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:05.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:05.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:05.424+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:05.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:05.426+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:05.911 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:05.703+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:05.704256+0000) 2024-04-27T04:06:05.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:05.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:05.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:05.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:05.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:05.659+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:05.911 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:05.660+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:06.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:05.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:06.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:05.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:06.208 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:05.911+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:06.209 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:05.911+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:06.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:06.208+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:06.209253+0000) 2024-04-27T04:06:06.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:06.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:06.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:06.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:06.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:06.422+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:06.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:06.423+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:06.424413+0000) 2024-04-27T04:06:06.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:06.424+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:06.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:06.705+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:06.706483+0000) 2024-04-27T04:06:06.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:06.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:06.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:06.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:06.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:06.707+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:06.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:06.708+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:07.210 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:06.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:07.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:06.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:07.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:06.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:07.211 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:06.956+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:07.469 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:07.210+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:07.211702+0000) 2024-04-27T04:06:07.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:07.428+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:07.429587+0000) 2024-04-27T04:06:07.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:07.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:07.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:07.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:07.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:07.437+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:07.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:07.438+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:07.912 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:07.707+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:07.708617+0000) 2024-04-27T04:06:07.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:07.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:07.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:07.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:07.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:07.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:07.913 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:07.703+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:08.213 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:07.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:08.214 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:07.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:08.214 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:07.912+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:08.214 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:07.912+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:08.466 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:08.432+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:08.434563+0000) 2024-04-27T04:06:08.467 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:08.212+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:08.214617+0000) 2024-04-27T04:06:08.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:08.709+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:08.710802+0000) 2024-04-27T04:06:08.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:08.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:08.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:08.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:08.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:08.465+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:08.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:08.467+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:08.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:08.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:08.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:08.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:08.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:08.712+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:08.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:08.713+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:09.217 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:08.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:09.217 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:08.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:09.218 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:08.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:09.218 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:08.952+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:09.469 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:06:09.260+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81235 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:06:09.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:09.254+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:09.256440+0000) 2024-04-27T04:06:09.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:09.215+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:09.217106+0000) 2024-04-27T04:06:09.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:09.255+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:09.257036+0000) 2024-04-27T04:06:09.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:09.254+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:09.255982+0000) 2024-04-27T04:06:09.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:09.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:09.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:09.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:09.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:09.429+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:09.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:09.429+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:09.471 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:09.435+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:09.436838+0000) 2024-04-27T04:06:09.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:09.711+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:09.713503+0000) 2024-04-27T04:06:09.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:09.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:09.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:09.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:09.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:09.719+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:09.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:09.720+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:10.300 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:10.217+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:10.219298+0000) 2024-04-27T04:06:10.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:09.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:10.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:09.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:10.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:09.999+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:10.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:09.999+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:10.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:10.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:10.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:10.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:10.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:10.383+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:10.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:10.384+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:10.561 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:10.437+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:10.438914+0000) 2024-04-27T04:06:10.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:10.714+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:10.716366+0000) 2024-04-27T04:06:10.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:10.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:10.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:10.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:10.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:10.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:10.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:10.673+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:11.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:11.219+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:11.221449+0000) 2024-04-27T04:06:11.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:11.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:11.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:11.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:11.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:11.014+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:11.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:11.014+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:11.563 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:11.371+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:11.564 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:11.371+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:11.564 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:11.371+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:11.564 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:11.372+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:11.564 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:11.439+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:11.441430+0000) 2024-04-27T04:06:11.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:11.716+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:11.718708+0000) 2024-04-27T04:06:11.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:11.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:11.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:11.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:11.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:11.713+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:11.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:11.714+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:12.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:12.222+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:12.224705+0000) 2024-04-27T04:06:12.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:11.984+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:12.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:11.984+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:12.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:11.984+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:12.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:11.985+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:12.565 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:12.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:12.565 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:12.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:12.565 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:12.400+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:12.565 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:12.401+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:12.566 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:12.441+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:12.443664+0000) 2024-04-27T04:06:12.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:12.719+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:12.720854+0000) 2024-04-27T04:06:12.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:12.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:12.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:12.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:12.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:12.725+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:12.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:12.726+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:13.309 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:13.225+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:13.227073+0000) 2024-04-27T04:06:13.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:13.007+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:13.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:13.007+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:13.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:13.007+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:13.310 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:13.008+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:13.565 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:13.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:13.565 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:13.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:13.565 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:13.382+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:13.566 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:13.383+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:13.566 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:13.444+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:13.445959+0000) 2024-04-27T04:06:13.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:13.722+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:13.724123+0000) 2024-04-27T04:06:13.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:13.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:13.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:13.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:13.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:13.676+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:13.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:13.677+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:14.262 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:13.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:14.263 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:13.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:14.263 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:13.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:14.263 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:13.988+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:14.263 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:14.227+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:14.229353+0000) 2024-04-27T04:06:14.565 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:06:14.270+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81276 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:06:14.566 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:14.265+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:14.266975+0000) 2024-04-27T04:06:14.566 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:14.265+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:14.267387+0000) 2024-04-27T04:06:14.566 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:14.265+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:14.266966+0000) 2024-04-27T04:06:14.567 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:14.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:14.567 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:14.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:14.567 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:14.368+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:14.567 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:14.368+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:14.567 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:14.446+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:14.448115+0000) 2024-04-27T04:06:14.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:14.725+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:14.727278+0000) 2024-04-27T04:06:14.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:14.964+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:14.965 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:14.684+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:14.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:14.684+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:14.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:14.684+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:14.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:14.685+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:15.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:14.964+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:15.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:14.964+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:15.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:14.964+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:15.569 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:15.231+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:15.233707+0000) 2024-04-27T04:06:15.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:15.327+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:15.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:15.327+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:15.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:15.328+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:15.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:15.328+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:15.570 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:15.448+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:15.450202+0000) 2024-04-27T04:06:15.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:15.729+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:15.731536+0000) 2024-04-27T04:06:15.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:15.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:15.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:15.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:15.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:15.678+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:15.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:15.679+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:16.236 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:15.997+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:16.236 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:15.997+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:16.236 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:15.997+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:16.237 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:15.997+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:16.574 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:16.235+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:16.236840+0000) 2024-04-27T04:06:16.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:16.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:16.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:16.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:16.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:16.282+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:16.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:16.283+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:16.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:16.451+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:16.453211+0000) 2024-04-27T04:06:16.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:16.733+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:16.735785+0000) 2024-04-27T04:06:16.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:16.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:16.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:16.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:16.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:16.705+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:16.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:16.706+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:17.237 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:16.997+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:17.238 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:16.997+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:17.238 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:16.997+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:17.238 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:16.998+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:17.575 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:17.237+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:17.239003+0000) 2024-04-27T04:06:17.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:17.294+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:17.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:17.294+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:17.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:17.294+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:17.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:17.296+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:17.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:17.454+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:17.456139+0000) 2024-04-27T04:06:17.854 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-04-27T04:06:17.859 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T04:06:17.859 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-04-27T04:06:17.860 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-04-27T04:06:17.873 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-27T04:06:17.884 INFO:tasks.cephadm:Teardown begin 2024-04-27T04:06:17.884 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/cephadm.py", line 2240, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-04-27T04:06:17.889 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T04:06:17.918 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:17.737+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:17.739064+0000) 2024-04-27T04:06:17.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:17.736+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:17.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:17.736+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:17.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:17.736+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:17.919 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:17.737+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:17.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:06:17.925 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:06:18.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:17.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:18.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:17.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:18.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:17.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:18.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:17.969+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:18.576 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:18.240+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:18.241851+0000) 2024-04-27T04:06:18.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:18.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:18.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:18.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:18.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:18.323+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:18.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:18.325+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:18.576 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:18.456+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:18.458286+0000) 2024-04-27T04:06:18.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:18.739+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:18.741344+0000) 2024-04-27T04:06:18.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:18.697+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:18.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:18.697+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:18.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:18.697+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:18.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:18.698+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:19.243 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:18.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:19.243 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:18.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:19.243 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:18.993+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:19.243 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:18.993+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:19.576 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:06:19.280+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81315 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:06:19.576 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:19.242+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:19.244169+0000) 2024-04-27T04:06:19.576 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:19.274+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:19.276717+0000) 2024-04-27T04:06:19.577 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:19.275+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:19.277716+0000) 2024-04-27T04:06:19.577 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:19.275+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:19.276967+0000) 2024-04-27T04:06:19.577 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:19.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:19.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:19.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:19.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:19.345+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:19.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:19.346+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:19.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:19.458+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:19.460414+0000) 2024-04-27T04:06:19.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:19.742+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:19.743617+0000) 2024-04-27T04:06:19.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:19.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:19.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:19.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:19.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:19.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:19.955 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:19.716+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:19.956 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:19.717+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:20.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:19.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:20.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:19.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:20.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:19.955+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:20.581 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:20.247+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:20.248525+0000) 2024-04-27T04:06:20.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:20.303+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:20.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:20.303+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:20.581 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:20.303+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:20.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:20.303+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:20.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:20.461+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:20.462585+0000) 2024-04-27T04:06:20.956 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:20.745+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:20.745955+0000) 2024-04-27T04:06:20.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:20.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:20.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:20.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:20.957 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:20.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:20.958 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:20.670+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:20.990 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:06:21.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:20.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:21.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:20.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:21.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:20.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:21.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:20.957+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:21.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:21.283+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:21.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:21.283+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:21.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:21.283+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:21.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:21.283+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:21.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:21.464+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:21.464900+0000) 2024-04-27T04:06:21.585 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:21.250+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:21.251274+0000) 2024-04-27T04:06:21.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:21.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:21.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:21.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:21.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:21.646+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:21.951 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:21.647+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:21.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:21.749+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:21.750286+0000) 2024-04-27T04:06:21.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:21.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:21.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:21.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:22.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:21.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:22.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:21.951+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:22.585 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:22.253+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:22.254299+0000) 2024-04-27T04:06:22.585 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:22.243+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:22.586 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:22.243+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:22.586 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:22.243+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:22.586 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:22.243+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:22.586 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:22.468+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:22.469092+0000) 2024-04-27T04:06:22.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:22.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:22.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:22.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:22.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:22.671+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:22.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:22.672+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:22.940 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:22.752+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:22.753737+0000) 2024-04-27T04:06:22.940 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:22.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:22.941 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:22.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:22.941 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:22.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:23.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:22.940+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:23.585 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:23.258+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:23.258942+0000) 2024-04-27T04:06:23.586 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:23.292+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:23.586 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:23.292+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:23.586 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:23.292+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:23.586 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:23.293+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:23.586 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:23.471+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:23.472180+0000) 2024-04-27T04:06:23.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:23.755+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:23.756496+0000) 2024-04-27T04:06:23.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:23.959+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:23.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:23.703+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:23.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:23.703+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:23.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:23.703+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:23.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:23.704+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:24.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:23.959+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:24.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:23.959+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:24.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:23.959+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:24.285 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:24.260+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:24.261363+0000) 2024-04-27T04:06:24.586 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:06:24.292+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81356 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:06:24.586 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:24.285+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:24.286367+0000) 2024-04-27T04:06:24.587 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:24.286+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:24.287397+0000) 2024-04-27T04:06:24.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:24.285+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:24.286483+0000) 2024-04-27T04:06:24.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:24.311+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:24.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:24.311+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:24.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:24.311+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:24.587 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:24.312+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:24.588 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:24.475+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:24.476517+0000) 2024-04-27T04:06:24.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:24.760+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:24.760864+0000) 2024-04-27T04:06:24.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:24.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:24.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:24.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:24.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:24.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:24.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:24.715+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:24.949 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:24.716+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:25.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:24.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:25.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:24.948+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:25.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:24.949+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:25.588 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:25.362+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:25.588 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:25.362+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:25.589 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:25.362+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:25.589 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:25.363+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:25.589 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:25.479+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:25.479944+0000) 2024-04-27T04:06:25.589 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:25.263+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:25.264388+0000) 2024-04-27T04:06:25.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:25.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:25.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:25.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:25.938 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:25.755+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:25.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:25.756+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:25.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:25.762+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:25.763672+0000) 2024-04-27T04:06:25.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:25.938+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:25.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:25.938+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:25.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:25.938+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:26.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:25.939+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:26.481 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:26.265+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:26.266721+0000) 2024-04-27T04:06:26.481 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:26.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:26.482 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:26.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:26.482 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:26.389+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:26.482 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:26.390+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:26.766 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:26.481+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:26.482541+0000) 2024-04-27T04:06:27.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:26.787+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:27.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:26.787+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:27.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:26.787+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:27.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:26.787+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:27.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:26.766+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:26.766878+0000) 2024-04-27T04:06:27.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:26.962+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:27.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:26.962+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:27.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:26.962+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:27.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:26.963+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:27.421 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:27.270+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:27.271027+0000) 2024-04-27T04:06:27.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:27.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:27.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:27.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:27.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:27.425+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:27.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:27.425+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:27.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:27.485+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:27.486800+0000) 2024-04-27T04:06:28.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:27.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:28.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:27.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:28.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:27.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:28.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:27.765+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:28.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:27.770+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:27.770821+0000) 2024-04-27T04:06:28.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:27.938+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:28.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:27.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:28.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:27.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:28.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:27.939+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:28.421 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:28.273+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:28.274149+0000) 2024-04-27T04:06:28.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:28.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:28.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:28.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:28.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:28.434+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:28.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:28.435+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:28.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:28.490+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:28.491100+0000) 2024-04-27T04:06:29.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:28.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:29.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:28.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:29.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:28.775+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:29.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:28.775+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:29.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:28.772+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:28.773273+0000) 2024-04-27T04:06:29.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:28.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:29.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:28.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:29.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:28.968+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:29.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:28.968+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:29.402 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:06:29.302+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81395 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:06:29.403 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:29.297+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:29.298352+0000) 2024-04-27T04:06:29.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:29.296+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:29.297178+0000) 2024-04-27T04:06:29.403 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:29.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:29.403 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:29.276+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:29.276904+0000) 2024-04-27T04:06:29.404 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:29.297+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:29.298787+0000) 2024-04-27T04:06:29.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:29.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:29.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:29.402+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:29.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:29.404+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:29.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:29.493+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:29.494164+0000) 2024-04-27T04:06:29.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:06:29.992 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:06:30.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:29.775+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:29.776486+0000) 2024-04-27T04:06:30.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:30.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:30.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:30.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:30.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:30.008+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:30.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:30.008+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:30.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:29.767+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:30.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:29.767+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:30.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:29.767+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:30.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:29.769+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:30.421 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:30.279+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:30.280176+0000) 2024-04-27T04:06:30.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:30.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:30.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:30.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:30.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:30.427+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:30.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:30.428+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:30.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:30.498+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:30.499176+0000) 2024-04-27T04:06:31.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:30.797+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:31.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:30.797+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:31.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:30.797+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:31.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:30.798+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:31.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:30.777+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:30.778666+0000) 2024-04-27T04:06:31.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:30.970+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:31.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:30.970+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:31.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:30.970+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:31.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:30.970+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:31.421 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:31.282+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:31.282998+0000) 2024-04-27T04:06:31.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:31.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:31.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:31.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:31.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:31.444+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:31.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:31.445+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:31.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:31.501+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:31.502152+0000) 2024-04-27T04:06:32.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:31.781+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:31.781954+0000) 2024-04-27T04:06:32.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:31.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:32.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:31.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:32.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:31.951+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:32.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:31.951+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:32.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:31.808+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:32.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:31.808+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:32.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:31.808+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:32.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:31.809+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:32.412 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:32.285+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:32.286095+0000) 2024-04-27T04:06:32.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:32.412+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:32.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:32.412+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:32.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:32.412+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:32.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:32.413+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:32.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:32.504+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:32.504802+0000) 2024-04-27T04:06:33.054 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:06:33.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:32.783+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:32.784642+0000) 2024-04-27T04:06:33.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:32.976+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:33.139 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:32.976+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:33.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:32.976+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:33.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:32.977+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:33.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:32.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:33.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:32.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:33.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:32.802+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:33.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:32.803+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:33.421 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:33.287+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:33.288760+0000) 2024-04-27T04:06:33.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:33.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:33.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:33.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:33.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:33.457+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:33.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:33.458+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:33.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:33.508+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:33.509083+0000) 2024-04-27T04:06:34.139 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:33.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:34.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:33.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:34.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:33.769+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:34.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:33.770+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:34.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:33.787+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:33.787884+0000) 2024-04-27T04:06:34.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:34.018+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:34.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:34.018+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:34.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:34.018+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:34.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:34.019+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:34.422 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:06:34.311+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81436 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:06:34.422 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:34.306+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:34.307595+0000) 2024-04-27T04:06:34.423 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:34.306+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:34.307689+0000) 2024-04-27T04:06:34.423 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:34.290+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:34.291030+0000) 2024-04-27T04:06:34.423 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:34.307+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:34.308336+0000) 2024-04-27T04:06:34.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:34.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:34.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:34.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:34.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:34.503+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:34.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:34.503+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:34.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:34.511+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:34.512389+0000) 2024-04-27T04:06:35.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:34.789+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:34.790105+0000) 2024-04-27T04:06:35.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:34.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:35.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:34.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:35.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:34.798+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:35.049 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:34.798+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:35.312 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:35.292+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:35.293157+0000) 2024-04-27T04:06:35.312 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:35.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:35.313 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:35.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:35.313 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:35.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:35.313 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:35.049+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:35.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:35.514+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:35.515310+0000) 2024-04-27T04:06:35.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:35.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:35.610 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:35.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:35.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:35.525+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:35.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:35.526+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:35.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:35.794+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:35.794882+0000) 2024-04-27T04:06:35.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:35.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:35.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:35.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:35.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:35.785+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:35.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:35.786+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:36.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:36.016+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:36.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:36.016+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:36.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:36.016+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:36.297 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:36.016+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:36.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:36.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:36.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:36.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:36.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:36.486+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:36.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:36.488+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:36.611 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:36.516+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:36.517396+0000) 2024-04-27T04:06:36.611 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:36.296+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:36.297528+0000) 2024-04-27T04:06:36.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:36.797+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:36.798101+0000) 2024-04-27T04:06:36.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:36.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:36.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:36.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:36.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:36.817+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:36.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:36.818+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:37.300 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:36.976+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:37.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:36.976+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:37.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:36.976+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:37.301 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:36.977+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:37.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:37.301+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:37.301896+0000) 2024-04-27T04:06:37.613 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:37.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:37.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:37.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:37.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:37.477+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:37.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:37.478+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:37.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:37.518+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:37.519514+0000) 2024-04-27T04:06:37.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:37.799+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:37.800341+0000) 2024-04-27T04:06:37.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:37.837+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:37.939 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:37.837+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:37.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:37.837+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:37.940 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:37.837+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:38.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:37.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:38.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:37.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:38.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:37.939+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:38.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:37.940+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:38.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:38.304+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:38.305017+0000) 2024-04-27T04:06:38.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:38.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:38.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:38.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:38.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:38.519+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:38.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:38.520+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:38.614 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:38.520+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:38.521658+0000) 2024-04-27T04:06:38.922 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:38.804+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:38.805186+0000) 2024-04-27T04:06:38.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:38.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:38.922 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:38.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:38.923 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:38.794+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:38.923 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:38.795+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:39.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:38.922+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:39.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:38.922+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:39.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:38.922+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:39.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:38.922+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:39.613 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:06:39.320+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81476 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:06:39.614 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:39.316+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:39.317544+0000) 2024-04-27T04:06:39.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:39.308+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:39.309392+0000) 2024-04-27T04:06:39.615 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:39.316+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:39.317087+0000) 2024-04-27T04:06:39.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:39.317+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:39.317823+0000) 2024-04-27T04:06:39.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:39.523+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:39.523823+0000) 2024-04-27T04:06:39.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:39.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:39.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:39.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:39.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:39.540+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:39.615 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:39.540+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:39.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:39.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:39.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:39.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:39.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:39.790+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:39.927 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:39.791+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:39.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:39.808+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:39.809551+0000) 2024-04-27T04:06:39.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:39.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:39.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:39.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:39.928 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:39.927+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:40.183 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:39.927+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:40.435 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:40.310+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:40.311639+0000) 2024-04-27T04:06:40.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:40.526+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:40.527598+0000) 2024-04-27T04:06:40.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:40.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:40.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:40.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:40.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:40.538+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:40.723 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:40.539+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:41.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:40.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:41.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:40.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:41.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:40.782+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:41.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:40.783+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:41.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:40.811+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:40.811818+0000) 2024-04-27T04:06:41.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:40.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:41.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:40.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:41.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:40.929+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:41.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:40.930+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:41.422 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:41.315+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:41.316337+0000) 2024-04-27T04:06:41.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:41.529+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:41.529841+0000) 2024-04-27T04:06:41.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:41.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:41.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:41.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:41.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:41.563+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:41.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:41.564+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:42.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:41.828+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:42.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:41.828+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:42.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:41.828+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:42.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:41.829+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:42.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:41.813+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:41.814080+0000) 2024-04-27T04:06:42.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:41.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:42.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:41.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:42.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:41.954+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:42.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:41.955+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:42.423 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:42.320+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:42.321083+0000) 2024-04-27T04:06:42.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:42.532+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:42.533062+0000) 2024-04-27T04:06:42.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:42.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:42.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:42.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:42.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:42.565+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:42.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:42.566+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:43.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:42.828+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:43.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:42.828+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:43.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:42.828+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:43.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:42.829+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:43.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:42.815+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:42.816193+0000) 2024-04-27T04:06:43.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:42.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:43.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:42.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:43.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:42.956+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:43.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:42.956+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:43.423 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:43.323+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:43.324180+0000) 2024-04-27T04:06:43.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:43.534+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:43.535745+0000) 2024-04-27T04:06:43.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:43.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:43.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:43.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:43.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:43.581+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:43.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:43.582+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:44.093 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:43.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:44.093 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:43.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:44.093 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:43.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:44.093 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:43.844+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:44.094 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:43.817+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:43.818362+0000) 2024-04-27T04:06:44.094 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:43.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:44.094 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:43.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:44.094 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:43.966+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:44.094 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:43.967+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:44.349 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:06:44.331+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81516 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:06:44.350 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:44.325+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:44.325993+0000) 2024-04-27T04:06:44.350 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:44.326+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:44.327773+0000) 2024-04-27T04:06:44.350 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:44.325+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:44.326059+0000) 2024-04-27T04:06:44.350 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:44.325+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:44.326322+0000) 2024-04-27T04:06:44.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:44.539+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:44.540546+0000) 2024-04-27T04:06:44.622 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:44.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:44.623 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:44.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:44.623 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:44.582+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:44.623 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:44.583+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:44.888 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:44.820+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:44.821299+0000) 2024-04-27T04:06:45.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:06:45.056 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:06:45.140 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:44.959+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:45.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:44.959+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:45.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:44.959+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:45.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:44.960+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:45.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:44.888+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:45.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:44.888+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:45.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:44.888+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:45.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:44.889+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:45.423 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:45.329+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:45.330399+0000) 2024-04-27T04:06:45.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:45.541+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:45.542624+0000) 2024-04-27T04:06:45.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:45.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:45.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:45.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:45.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:45.609+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:45.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:45.611+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:46.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:45.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:46.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:45.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:46.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:45.856+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:46.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:45.857+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:46.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:45.822+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:45.823385+0000) 2024-04-27T04:06:46.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:45.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:46.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:45.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:46.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:45.985+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:46.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:45.986+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:46.423 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:46.334+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:46.335113+0000) 2024-04-27T04:06:46.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:46.543+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:46.544742+0000) 2024-04-27T04:06:46.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:46.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:46.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:46.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:46.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:46.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:46.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:46.626+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:47.140 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:46.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:47.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:46.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:47.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:46.882+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:47.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:46.883+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:47.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:46.826+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:46.827674+0000) 2024-04-27T04:06:47.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:47.034+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:47.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:47.034+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:47.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:47.034+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:47.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:47.034+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:47.423 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:47.337+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:47.338161+0000) 2024-04-27T04:06:47.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:47.546+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:47.546855+0000) 2024-04-27T04:06:47.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:47.665+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:47.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:47.665+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:47.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:47.665+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:47.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:47.666+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:48.130 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:06:48.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:47.829+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:47.829972+0000) 2024-04-27T04:06:48.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:48.035+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:48.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:48.035+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:48.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:48.035+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:48.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:48.036+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:48.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:47.920+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:48.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:47.920+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:48.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:47.920+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:48.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:47.920+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:48.423 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:48.340+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:48.340850+0000) 2024-04-27T04:06:48.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:48.548+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:48.548961+0000) 2024-04-27T04:06:48.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:48.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:48.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:48.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:48.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:48.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:48.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:48.628+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:49.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:48.831+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:48.832274+0000) 2024-04-27T04:06:49.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:49.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:49.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:49.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:49.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:49.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:49.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:49.042+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:49.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:48.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:49.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:48.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:49.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:48.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:49.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:48.885+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:49.423 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:06:49.339+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81556 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:06:49.424 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:49.336+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:49.337462+0000) 2024-04-27T04:06:49.424 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:49.335+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:49.336576+0000) 2024-04-27T04:06:49.424 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:49.334+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:49.335786+0000) 2024-04-27T04:06:49.425 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:49.342+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:49.343216+0000) 2024-04-27T04:06:49.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:49.550+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:49.551236+0000) 2024-04-27T04:06:49.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:49.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:49.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:49.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:49.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:49.666+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:49.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:49.667+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:50.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:49.934+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:50.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:49.934+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:50.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:49.934+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:50.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:49.935+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:50.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:49.833+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:49.834529+0000) 2024-04-27T04:06:50.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:50.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:50.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:50.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:50.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:50.065+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:50.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:50.065+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:50.423 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:50.345+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:50.346641+0000) 2024-04-27T04:06:50.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:50.552+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:50.553451+0000) 2024-04-27T04:06:50.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:50.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:50.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:50.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:50.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:50.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:50.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:50.665+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:51.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:06:51.132 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:06:51.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:50.835+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:50.836748+0000) 2024-04-27T04:06:51.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:51.064+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:51.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:51.064+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:51.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:51.064+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:51.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:51.064+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:51.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:50.913+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:51.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:50.913+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:51.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:50.913+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:51.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:50.914+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:51.198 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:06:51.198 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:06:51.199 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-27T04:06:51.199 DEBUG:teuthology.orchestra.run.smithi029:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-27T04:06:51.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:06:51.215 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:06:51.423 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:51.348+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:51.348909+0000) 2024-04-27T04:06:51.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:51.556+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:51.557697+0000) 2024-04-27T04:06:51.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:51.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:51.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:51.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:51.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:51.629+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:51.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:51.630+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:52.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:51.944+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:52.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:51.944+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:52.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:51.944+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:52.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:51.945+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:52.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:51.840+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:51.841006+0000) 2024-04-27T04:06:52.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:52.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:52.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:52.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:52.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:52.037+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:52.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:52.038+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:52.423 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:52.350+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:52.351134+0000) 2024-04-27T04:06:52.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:52.559+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:52.559999+0000) 2024-04-27T04:06:52.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:52.616+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:52.784 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:52.616+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:52.785 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:52.616+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:52.785 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:52.617+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:53.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:52.955+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:53.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:52.955+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:53.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:52.955+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:53.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:52.957+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:53.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:52.843+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:52.844299+0000) 2024-04-27T04:06:53.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:53.047+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:53.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:53.047+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:53.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:53.047+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:53.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:53.048+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:53.424 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:53.352+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:53.353589+0000) 2024-04-27T04:06:53.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:53.561+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:53.562256+0000) 2024-04-27T04:06:53.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:53.649+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:53.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:53.649+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:53.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:53.649+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:53.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:53.650+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:54.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:54.006+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:54.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:54.006+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:54.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:54.006+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:54.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:54.007+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:54.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:53.845+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:53.846509+0000) 2024-04-27T04:06:54.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:54.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:54.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:54.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:54.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:54.038+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:54.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:54.039+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:54.270 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:06:54.424 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:06:54.348+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81596 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:06:54.424 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:54.344+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:54.345316+0000) 2024-04-27T04:06:54.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:54.344+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:54.345402+0000) 2024-04-27T04:06:54.425 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:54.345+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:54.346329+0000) 2024-04-27T04:06:54.425 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:54.357+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:54.357936+0000) 2024-04-27T04:06:54.678 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:54.563+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:54.564621+0000) 2024-04-27T04:06:54.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:54.849+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:54.849940+0000) 2024-04-27T04:06:54.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:54.678+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:54.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:54.678+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:54.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:54.678+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:54.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:54.680+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:55.315 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:55.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:55.315 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:55.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:55.315 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:55.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:55.315 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:55.061+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:55.316 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:55.017+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:55.316 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:55.017+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:55.316 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:55.017+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:55.316 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:55.018+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:55.567 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:55.359+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:55.360244+0000) 2024-04-27T04:06:55.851 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:55.568+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:55.568827+0000) 2024-04-27T04:06:55.851 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:55.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:55.851 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:55.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:55.851 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:55.667+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:55.851 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:55.668+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:56.141 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:56.056+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:56.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:56.056+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:56.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:56.056+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:56.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:56.057+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:56.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:55.851+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:55.852007+0000) 2024-04-27T04:06:56.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:56.098+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:56.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:56.098+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:56.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:56.098+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:56.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:56.099+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:56.424 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:56.361+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:56.362368+0000) 2024-04-27T04:06:56.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:56.570+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:56.571236+0000) 2024-04-27T04:06:56.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:56.656+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:56.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:56.656+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:56.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:56.656+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:56.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:56.656+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:57.141 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:56.853+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:56.854410+0000) 2024-04-27T04:06:57.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:57.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:57.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:57.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:57.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:57.096+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:57.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:57.096+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:57.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:57.021+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:57.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:57.021+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:57.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:57.021+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:57.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:57.022+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:57.424 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:57.363+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:57.364558+0000) 2024-04-27T04:06:57.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:57.572+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:57.573323+0000) 2024-04-27T04:06:57.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:57.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:57.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:57.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:57.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:57.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:57.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:57.639+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:58.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:58.027+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:58.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:58.027+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:58.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:58.027+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:58.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:58.028+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:58.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:57.855+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:57.856551+0000) 2024-04-27T04:06:58.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:58.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:58.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:58.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:58.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:58.061+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:58.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:58.062+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:58.424 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:58.365+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:58.366763+0000) 2024-04-27T04:06:58.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:58.574+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:58.575428+0000) 2024-04-27T04:06:58.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:58.665+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:58.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:58.665+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:58.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:58.665+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:58.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:58.665+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:59.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:59.064+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:59.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:59.064+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:59.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:59.064+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:06:59.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:59.065+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:59.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:58.858+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:58.859452+0000) 2024-04-27T04:06:59.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:59.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:59.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:59.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:59.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:59.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:06:59.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:59.054+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:06:59.424 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:06:59.357+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81636 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:06:59.425 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:59.355+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:59.356250+0000) 2024-04-27T04:06:59.425 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:59.353+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:59.354014+0000) 2024-04-27T04:06:59.426 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:59.353+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:59.354210+0000) 2024-04-27T04:06:59.426 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:06:59.368+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:59.368896+0000) 2024-04-27T04:06:59.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:59.576+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:59.577548+0000) 2024-04-27T04:06:59.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:59.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:59.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:59.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:59.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:59.617+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:06:59.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:06:59.618+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:00.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:00.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:00.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:00.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:00.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:00.091+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:00.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:00.093+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:00.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:06:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:06:59.861+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:06:59.862361+0000) 2024-04-27T04:07:00.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:00.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:00.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:00.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:00.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:00.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:00.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:00.054+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:00.424 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:00.370+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:00.371637+0000) 2024-04-27T04:07:00.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:00.578+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:00.579685+0000) 2024-04-27T04:07:00.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:00.614+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:00.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:00.615+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:00.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:00.615+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:00.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:00.616+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:01.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:01.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:01.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:01.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:01.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:01.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:01.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:01.066+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:01.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:00.863+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:00.864699+0000) 2024-04-27T04:07:01.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:01.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:01.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:01.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:01.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:01.024+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:01.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:01.025+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:01.424 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:01.374+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:01.374915+0000) 2024-04-27T04:07:01.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:01.581+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:01.581877+0000) 2024-04-27T04:07:01.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:01.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:01.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:01.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:01.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:01.611+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:01.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:01.611+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:02.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:02.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:02.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:02.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:02.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:02.097+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:02.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:02.097+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:02.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:01.868+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:01.868983+0000) 2024-04-27T04:07:02.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:02.046+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:02.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:02.046+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:02.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:02.046+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:02.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:02.047+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:02.425 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:02.376+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:02.377021+0000) 2024-04-27T04:07:02.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:02.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:02.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:02.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:02.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:02.572+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:02.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:02.573+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:02.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:02.584+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:02.585108+0000) 2024-04-27T04:07:03.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:03.107+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:03.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:03.107+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:03.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:03.107+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:03.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:03.108+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:03.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:02.871+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:02.871912+0000) 2024-04-27T04:07:03.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:03.062+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:03.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:03.062+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:03.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:03.062+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:03.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:03.062+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:03.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:07:03.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:07:03.425 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:03.378+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:03.379472+0000) 2024-04-27T04:07:03.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:03.586+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:03.587284+0000) 2024-04-27T04:07:03.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:03.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:03.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:03.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:03.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:03.619+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:03.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:03.620+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:04.142 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:03.874+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:03.875144+0000) 2024-04-27T04:07:04.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:04.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:04.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:04.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:04.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:04.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:04.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:04.042+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:04.425 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:07:04.367+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81676 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:07:04.426 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:04.362+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:04.363437+0000) 2024-04-27T04:07:04.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:04.363+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:04.363976+0000) 2024-04-27T04:07:04.426 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:04.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:04.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:04.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:04.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:04.143+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:04.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:04.144+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:04.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:04.362+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:04.362945+0000) 2024-04-27T04:07:04.427 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:04.383+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:04.384042+0000) 2024-04-27T04:07:04.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:04.589+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:04.590022+0000) 2024-04-27T04:07:04.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:04.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:04.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:04.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:04.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:04.618+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:04.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:04.619+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:05.142 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:05.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:05.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:05.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:05.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:05.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:05.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:05.100+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:05.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:04.876+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:04.877246+0000) 2024-04-27T04:07:05.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:05.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:05.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:05.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:05.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:05.080+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:05.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:05.081+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:05.425 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:05.387+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:05.388382+0000) 2024-04-27T04:07:05.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:05.592+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:05.593159+0000) 2024-04-27T04:07:05.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:05.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:05.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:05.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:05.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:05.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:05.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:05.596+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:06.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:06.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:06.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:06.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:06.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:06.066+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:06.143 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:06.067+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:06.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:05.878+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:05.879321+0000) 2024-04-27T04:07:06.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:06.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:06.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:06.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:06.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:06.036+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:06.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:06.037+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:06.338 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:07:06.425 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:06.389+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:06.390474+0000) 2024-04-27T04:07:06.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:06.595+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:06.595813+0000) 2024-04-27T04:07:06.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:06.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:06.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:06.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:06.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:06.628+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:06.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:06.629+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:07.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:06.881+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:06.882701+0000) 2024-04-27T04:07:07.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:07.070+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:07.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:07.070+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:07.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:07.070+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:07.144 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:07.071+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:07.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:07.041+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:07.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:07.041+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:07.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:07.041+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:07.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:07.042+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:07.425 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:07.391+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:07.392584+0000) 2024-04-27T04:07:07.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:07.597+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:07.597938+0000) 2024-04-27T04:07:07.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:07.637+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:07.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:07.637+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:07.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:07.637+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:07.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:07.638+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:08.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:07.886+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:07.887024+0000) 2024-04-27T04:07:08.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:08.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:08.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:08.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:08.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:08.058+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:08.143 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:08.058+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:08.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:08.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:08.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:08.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:08.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:08.081+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:08.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:08.082+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:08.647 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:08.393+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:08.394715+0000) 2024-04-27T04:07:08.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:08.599+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:08.600101+0000) 2024-04-27T04:07:08.647 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:08.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:08.648 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:08.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:08.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:08.891+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:08.891920+0000) 2024-04-27T04:07:08.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:08.647+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:08.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:08.648+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:09.316 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:09.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:09.317 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:09.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:09.317 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:09.128+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:09.317 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:09.129+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:09.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:09.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:09.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:09.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:09.318 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:09.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:09.318 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:09.054+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:09.601 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:07:09.375+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81716 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:07:09.601 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:09.372+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:09.374499+0000) 2024-04-27T04:07:09.601 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:09.372+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:09.374077+0000) 2024-04-27T04:07:09.602 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:09.372+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:09.373857+0000) 2024-04-27T04:07:09.602 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:09.397+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:09.399098+0000) 2024-04-27T04:07:09.895 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:09.600+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:09.602276+0000) 2024-04-27T04:07:09.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:09.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:09.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:09.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:09.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:09.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:09.896 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:09.665+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:10.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:10.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:10.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:10.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:10.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:10.096+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:10.185 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:10.097+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:10.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:09.895+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:09.896841+0000) 2024-04-27T04:07:10.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:10.068+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:10.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:10.068+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:10.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:10.068+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:10.186 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:10.068+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:10.438 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:10.399+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:10.401118+0000) 2024-04-27T04:07:10.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:10.602+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:10.604484+0000) 2024-04-27T04:07:10.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:10.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:10.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:10.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:10.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:10.620+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:10.698 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:10.621+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:10.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:10.897+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:10.899794+0000) 2024-04-27T04:07:11.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:11.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:11.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:11.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:11.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:11.033+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:11.317 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:11.034+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:11.317 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:11.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:11.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:11.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:11.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:11.098+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:11.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:11.099+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:11.605 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:11.401+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:11.403200+0000) 2024-04-27T04:07:11.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:11.604+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:11.606568+0000) 2024-04-27T04:07:11.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:11.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:11.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:11.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:11.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:11.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:11.901 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:11.635+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:12.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:11.900+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:11.902107+0000) 2024-04-27T04:07:12.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:12.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:12.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:12.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:12.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:12.012+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:12.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:12.013+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:12.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:12.104+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:12.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:12.104+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:12.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:12.104+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:12.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:12.105+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:12.607 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:12.403+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:12.405293+0000) 2024-04-27T04:07:12.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:12.607+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:12.608812+0000) 2024-04-27T04:07:12.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:12.644+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:12.903 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:12.644+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:12.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:12.644+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:12.904 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:12.645+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:13.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:13.115+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:13.234 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:13.115+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:13.234 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:13.115+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:13.235 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:13.116+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:13.235 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:12.902+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:12.904502+0000) 2024-04-27T04:07:13.235 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:13.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:13.235 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:13.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:13.235 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:13.029+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:13.235 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:13.029+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:13.609 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:13.406+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:13.407970+0000) 2024-04-27T04:07:13.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:13.609+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:13.611119+0000) 2024-04-27T04:07:13.905 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:13.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:13.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:13.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:13.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:13.633+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:13.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:13.634+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:14.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:13.905+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:13.906800+0000) 2024-04-27T04:07:14.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:13.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:14.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:13.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:14.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:13.991+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:14.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:13.991+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:14.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:14.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:14.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:14.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:14.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:14.075+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:14.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:14.076+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:14.612 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:07:14.385+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81757 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:07:14.612 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:14.380+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:14.382225+0000) 2024-04-27T04:07:14.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:14.380+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:14.381852+0000) 2024-04-27T04:07:14.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:14.380+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:14.382342+0000) 2024-04-27T04:07:14.613 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:14.408+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:14.410084+0000) 2024-04-27T04:07:14.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:14.611+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:14.613411+0000) 2024-04-27T04:07:14.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:14.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:14.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:14.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:14.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:14.638+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:14.908 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:14.639+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:15.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:15.047+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:15.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:15.047+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:15.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:15.047+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:15.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:15.049+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:15.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:14.907+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:14.909120+0000) 2024-04-27T04:07:15.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:14.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:15.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:14.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:15.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:14.955+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:15.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:14.955+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:15.614 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:15.410+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:15.412311+0000) 2024-04-27T04:07:15.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:15.614+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:15.616125+0000) 2024-04-27T04:07:15.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:15.651+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:15.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:15.651+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:15.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:15.651+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:15.911 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:15.651+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:16.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:16.025+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:16.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:16.025+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:16.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:16.025+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:16.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:16.026+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:16.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:15.910+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:15.912066+0000) 2024-04-27T04:07:16.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:15.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:16.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:15.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:16.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:15.952+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:16.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:15.952+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:16.617 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:16.413+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:16.415300+0000) 2024-04-27T04:07:16.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:16.617+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:16.619036+0000) 2024-04-27T04:07:16.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:16.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:16.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:16.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:16.913 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:16.634+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:16.914 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:16.635+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:17.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:17.028+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:17.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:17.028+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:17.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:17.028+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:17.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:17.029+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:17.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:16.912+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:16.914462+0000) 2024-04-27T04:07:17.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:16.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:17.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:16.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:17.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:16.965+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:17.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:16.966+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:17.620 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:17.415+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:17.417743+0000) 2024-04-27T04:07:17.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:17.620+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:17.621886+0000) 2024-04-27T04:07:17.915 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:17.670+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:17.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:17.670+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:17.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:17.670+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:17.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:17.671+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:18.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:17.915+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:17.916817+0000) 2024-04-27T04:07:18.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:17.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:18.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:17.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:18.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:17.947+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:18.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:17.947+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:18.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:18.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:18.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:18.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:18.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:18.001+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:18.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:18.001+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:18.339 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:07:18.340 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:07:18.623 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:18.419+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:18.421019+0000) 2024-04-27T04:07:18.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:18.622+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:18.624603+0000) 2024-04-27T04:07:18.918 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:18.645+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:18.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:18.645+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:18.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:18.645+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:18.919 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:18.647+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:19.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:19.018+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:19.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:19.018+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:19.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:19.018+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:19.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:19.018+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:19.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:18.917+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:18.919397+0000) 2024-04-27T04:07:19.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:18.919+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:19.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:18.919+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:19.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:18.919+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:19.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:18.920+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:19.598 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:07:19.395+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81797 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:07:19.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:19.390+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:19.392236+0000) 2024-04-27T04:07:19.599 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:19.389+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:19.391378+0000) 2024-04-27T04:07:19.599 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:19.390+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:19.392435+0000) 2024-04-27T04:07:19.599 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:19.422+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:19.423944+0000) 2024-04-27T04:07:19.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:19.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:19.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:19.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:19.891 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:19.597+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:19.891 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:19.599+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:19.891 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:19.627+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:19.628869+0000) 2024-04-27T04:07:20.144 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:19.988+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:20.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:19.988+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:20.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:19.988+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:20.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:19.989+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:20.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:19.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:20.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:19.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:20.145 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:19.889+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:20.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:19.890+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:20.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:19.920+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:19.922514+0000) 2024-04-27T04:07:20.692 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:20.426+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:20.428244+0000) 2024-04-27T04:07:20.692 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:20.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:20.692 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:20.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:20.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:20.584+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:20.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:20.586+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:20.693 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:20.629+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:20.630961+0000) 2024-04-27T04:07:20.957 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:20.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:20.957 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:20.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:20.957 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:20.893+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:20.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:20.894+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:20.958 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:20.923+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:20.924795+0000) 2024-04-27T04:07:21.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:21.025+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:21.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:21.025+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:21.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:21.025+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:21.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:21.026+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:21.406 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:07:21.616 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:21.429+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:21.430621+0000) 2024-04-27T04:07:21.888 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:21.616+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:21.888 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:21.616+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:21.888 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:21.616+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:21.888 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:21.617+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:21.888 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:21.632+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:21.633179+0000) 2024-04-27T04:07:22.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:22.052+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:22.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:22.052+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:22.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:22.052+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:22.146 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:22.053+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:22.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:21.888+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:22.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:21.888+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:22.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:21.888+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:22.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:21.888+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:22.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:21.928+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:21.929088+0000) 2024-04-27T04:07:22.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:22.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:22.696 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:22.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:22.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:22.625+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:22.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:22.626+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:22.697 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:22.634+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:22.635366+0000) 2024-04-27T04:07:22.697 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:22.432+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:22.433065+0000) 2024-04-27T04:07:22.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:22.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:22.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:22.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:22.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:22.851+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:22.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:22.852+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:22.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:22.931+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:22.932760+0000) 2024-04-27T04:07:23.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:23.058+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:23.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:23.058+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:23.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:23.058+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:23.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:23.059+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:23.606 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:23.435+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:23.436203+0000) 2024-04-27T04:07:23.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:23.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:23.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:23.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:23.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:23.842+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:23.934 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:23.843+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:23.934 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:23.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:23.935 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:23.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:23.935 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:23.607+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:23.935 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:23.607+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:23.938 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:23.636+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:23.637497+0000) 2024-04-27T04:07:24.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:23.934+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:23.935040+0000) 2024-04-27T04:07:24.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:24.057+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:24.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:24.057+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:24.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:24.057+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:24.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:24.058+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:24.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:07:24.408 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:07:24.482 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:07:24.483 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:07:24.483 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-27T04:07:24.485 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675525/remote/smithi029/crash 2024-04-27T04:07:24.486 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/crash -- . 2024-04-27T04:07:24.513 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/4c587076-0439-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675525/remote/smithi052/crash 2024-04-27T04:07:24.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:07:24.515 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:07:24.638 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:07:24.405+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81837 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:07:24.639 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:24.401+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:24.402081+0000) 2024-04-27T04:07:24.639 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:24.400+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:24.401771+0000) 2024-04-27T04:07:24.639 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:24.400+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:24.401261+0000) 2024-04-27T04:07:24.640 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:24.438+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:24.438901+0000) 2024-04-27T04:07:24.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:24.638+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:24.639740+0000) 2024-04-27T04:07:24.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:24.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:24.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:24.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:24.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:24.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:24.890 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:24.641+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:25.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:25.076+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:25.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:25.076+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:25.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:25.076+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:25.145 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:25.077+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:25.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:24.890+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:25.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:24.890+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:25.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:24.890+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:25.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:24.890+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:25.146 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:24.937+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:24.937823+0000) 2024-04-27T04:07:25.678 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:25.441+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:25.442106+0000) 2024-04-27T04:07:25.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:25.642+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:25.642841+0000) 2024-04-27T04:07:25.679 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:25.678+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:25.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:25.678+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:25.680 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:25.678+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:25.938 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:25.679+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:25.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:25.881+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:25.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:25.881+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:25.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:25.881+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:25.939 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:25.882+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:26.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:25.939+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:25.939960+0000) 2024-04-27T04:07:26.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:26.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:26.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:26.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:26.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:26.065+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:26.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:26.066+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:26.646 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:26.443+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:26.444258+0000) 2024-04-27T04:07:26.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:26.646+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:26.647591+0000) 2024-04-27T04:07:26.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:26.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:26.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:26.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:26.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:26.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:26.931 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:26.662+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:27.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:27.107+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:27.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:27.107+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:27.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:27.107+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:27.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:27.109+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:27.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:26.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:27.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:26.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:27.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:26.931+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:27.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:26.931+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:27.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:26.941+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:26.942561+0000) 2024-04-27T04:07:27.586 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:07:27.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:27.447+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:27.448596+0000) 2024-04-27T04:07:27.943 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:27.649+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:27.649815+0000) 2024-04-27T04:07:27.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:27.657+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:27.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:27.657+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:27.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:27.657+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:27.944 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:27.658+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:28.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:28.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:28.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:28.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:28.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:28.118+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:28.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:28.118+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:28.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:27.943+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:27.944741+0000) 2024-04-27T04:07:28.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:27.949+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:28.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:27.949+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:28.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:27.949+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:28.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:27.950+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:28.651 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:28.451+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:28.451893+0000) 2024-04-27T04:07:28.916 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:28.652+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:28.653079+0000) 2024-04-27T04:07:28.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:28.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:28.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:28.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:28.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:28.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:28.917 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:28.687+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:29.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:28.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:29.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:28.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:29.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:28.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:29.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:28.917+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:29.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:28.946+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:28.947276+0000) 2024-04-27T04:07:29.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:29.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:29.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:29.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:29.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:29.093+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:29.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:29.093+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:29.654 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:07:29.414+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81877 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:07:29.655 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:29.410+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:29.410970+0000) 2024-04-27T04:07:29.655 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:29.409+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:29.409910+0000) 2024-04-27T04:07:29.655 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:29.409+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:29.410586+0000) 2024-04-27T04:07:29.655 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:29.453+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:29.454626+0000) 2024-04-27T04:07:29.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:29.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:29.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:29.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:29.948 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:29.875+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:29.949 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:29.875+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:29.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:29.654+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:29.655374+0000) 2024-04-27T04:07:29.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:29.729+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:29.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:29.729+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:29.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:29.729+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:29.949 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:29.730+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:30.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:29.948+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:29.949469+0000) 2024-04-27T04:07:30.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:30.137+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:30.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:30.137+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:30.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:30.138+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:30.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:30.139+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:30.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:30.458+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:30.458845+0000) 2024-04-27T04:07:30.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:30.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:30.952 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:30.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:30.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:30.857+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:30.953 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:30.857+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:30.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:30.658+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:30.659495+0000) 2024-04-27T04:07:30.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:30.687+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:30.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:30.687+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:30.953 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:30.687+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:30.954 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:30.688+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:31.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:30.952+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:30.953541+0000) 2024-04-27T04:07:31.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:31.184+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:31.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:31.184+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:31.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:31.184+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:31.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:31.185+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:31.661 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:31.462+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:31.463717+0000) 2024-04-27T04:07:31.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:31.860+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:31.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:31.860+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:31.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:31.860+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:31.955 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:31.861+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:31.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:31.661+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:31.662376+0000) 2024-04-27T04:07:31.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:31.709+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:31.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:31.709+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:31.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:31.709+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:31.956 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:31.709+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:32.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:31.955+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:31.956204+0000) 2024-04-27T04:07:32.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:32.136+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:32.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:32.136+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:32.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:32.136+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:32.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:32.137+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:32.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:32.467+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:32.468734+0000) 2024-04-27T04:07:32.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:32.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:32.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:32.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:32.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:32.849+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:32.959 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:32.849+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:32.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:32.664+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:32.665515+0000) 2024-04-27T04:07:32.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:32.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:32.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:32.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:32.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:32.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:32.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:32.732+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:33.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:32.959+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:32.960436+0000) 2024-04-27T04:07:33.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:33.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:33.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:33.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:33.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:33.131+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:33.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:33.132+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:33.667 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:33.471+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:33.472031+0000) 2024-04-27T04:07:33.962 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:33.809+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:33.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:33.809+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:33.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:33.809+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:33.963 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:33.810+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:33.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:33.667+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:33.668385+0000) 2024-04-27T04:07:33.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:33.764+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:33.963 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:33.764+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:33.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:33.764+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:33.964 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:33.765+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:34.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:33.962+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:33.963764+0000) 2024-04-27T04:07:34.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:34.083+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:34.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:34.083+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:34.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:34.083+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:34.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:34.084+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:34.669 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:34.419+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:34.420366+0000) 2024-04-27T04:07:34.670 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:07:34.423+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81917 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:07:34.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:34.418+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:34.419520+0000) 2024-04-27T04:07:34.670 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:34.419+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:34.420456+0000) 2024-04-27T04:07:34.671 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:34.473+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:34.474195+0000) 2024-04-27T04:07:34.965 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:34.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:34.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:34.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:34.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:34.834+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:34.966 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:34.835+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:34.966 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:34.670+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:34.670963+0000) 2024-04-27T04:07:34.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:34.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:34.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:34.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:34.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:34.769+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:34.967 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:34.770+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:35.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:34.965+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:34.966657+0000) 2024-04-27T04:07:35.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:35.052+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:35.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:35.052+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:35.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:35.052+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:35.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:35.054+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:35.672 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:35.476+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:35.476820+0000) 2024-04-27T04:07:35.967 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:35.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:35.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:35.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:35.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:35.785+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:35.968 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:35.786+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:35.968 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:35.673+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:35.673949+0000) 2024-04-27T04:07:35.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:35.801+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:35.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:35.801+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:35.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:35.801+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:35.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:35.802+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:36.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:35.968+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:35.968966+0000) 2024-04-27T04:07:36.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:36.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:36.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:36.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:36.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:36.022+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:36.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:36.024+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:36.588 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:07:36.589 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:07:36.677 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:36.478+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:36.479462+0000) 2024-04-27T04:07:36.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:36.802+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:36.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:36.802+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:36.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:36.802+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:36.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:36.803+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:36.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:36.677+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:36.678767+0000) 2024-04-27T04:07:36.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:36.809+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:36.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:36.809+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:36.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:36.809+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:36.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:36.810+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:37.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:36.971+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:36.972203+0000) 2024-04-27T04:07:37.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:37.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:37.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:37.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:37.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:37.024+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:37.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:37.025+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:37.682 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:37.480+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:37.481558+0000) 2024-04-27T04:07:37.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:37.682+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:37.683568+0000) 2024-04-27T04:07:37.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:37.805+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:37.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:37.805+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:37.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:37.805+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:37.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:37.806+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:37.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:37.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:37.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:37.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:37.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:37.827+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:37.971 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:37.828+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:38.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:37.973+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:37.974322+0000) 2024-04-27T04:07:38.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:38.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:38.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:38.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:38.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:38.042+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:38.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:38.042+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:38.684 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:38.482+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:38.483703+0000) 2024-04-27T04:07:38.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:38.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:38.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:38.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:38.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:38.862+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:38.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:38.863+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:38.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:38.685+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:38.685850+0000) 2024-04-27T04:07:38.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:38.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:38.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:38.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:38.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:38.780+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:38.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:38.781+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:39.319 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:38.975+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:38.976591+0000) 2024-04-27T04:07:39.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:39.023+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:39.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:39.023+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:39.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:39.023+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:39.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:39.024+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:39.646 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:07:39.687 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:07:39.432+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81957 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:07:39.687 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:39.428+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:39.429272+0000) 2024-04-27T04:07:39.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:39.429+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:39.430159+0000) 2024-04-27T04:07:39.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:39.485+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:39.486416+0000) 2024-04-27T04:07:39.688 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:39.427+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:39.428007+0000) 2024-04-27T04:07:39.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:39.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:39.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:39.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:39.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:39.901+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:39.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:39.901+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:39.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:39.687+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:39.688581+0000) 2024-04-27T04:07:39.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:39.740+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:39.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:39.740+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:39.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:39.740+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:39.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:39.741+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:40.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:39.978+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:39.978902+0000) 2024-04-27T04:07:40.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:40.068+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:40.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:40.068+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:40.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:40.068+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:40.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:40.069+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:40.690 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:40.487+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:40.488477+0000) 2024-04-27T04:07:40.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:40.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:40.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:40.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:40.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:40.917+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:40.951 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:40.918+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:40.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:40.690+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:40.691610+0000) 2024-04-27T04:07:40.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:40.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:40.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:40.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:40.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:40.707+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:40.952 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:40.707+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:41.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:40.982+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:40.982911+0000) 2024-04-27T04:07:41.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:41.040+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:41.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:41.041+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:41.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:41.041+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:41.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:41.041+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:41.662 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:41.490+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:41.491341+0000) 2024-04-27T04:07:41.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:41.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:41.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:41.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:41.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:41.662+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:41.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:41.663+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:41.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:41.693+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:41.693845+0000) 2024-04-27T04:07:42.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:42.041+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:42.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:42.041+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:42.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:42.041+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:42.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:42.041+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:42.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:41.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:42.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:41.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:42.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:41.961+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:42.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:41.961+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:42.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:41.984+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:41.985348+0000) 2024-04-27T04:07:42.648 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:42.493+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:42.494345+0000) 2024-04-27T04:07:42.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:42.648+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:42.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:42.648+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:42.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:42.648+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:42.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:42.650+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:42.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:42.695+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:42.695969+0000) 2024-04-27T04:07:43.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:42.992+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:43.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:42.992+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:43.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:42.992+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:43.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:42.992+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:43.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:42.986+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:42.987528+0000) 2024-04-27T04:07:43.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:42.990+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:43.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:42.990+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:43.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:42.990+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:43.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:42.991+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:43.611 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:43.495+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:43.496487+0000) 2024-04-27T04:07:43.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:43.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:43.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:43.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:43.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:43.612+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:43.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:43.613+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:43.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:43.697+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:43.698049+0000) 2024-04-27T04:07:44.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:43.989+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:43.989883+0000) 2024-04-27T04:07:44.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:44.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:44.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:44.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:44.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:44.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:44.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:44.016+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:44.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:44.021+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:44.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:44.021+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:44.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:44.021+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:44.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:44.022+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:44.589 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:44.438+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:44.438890+0000) 2024-04-27T04:07:44.589 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:44.437+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:44.437880+0000) 2024-04-27T04:07:44.590 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:07:44.443+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 81997 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:07:44.590 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:44.437+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:44.438048+0000) 2024-04-27T04:07:44.590 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:44.497+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:44.498673+0000) 2024-04-27T04:07:44.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:44.589+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:44.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:44.589+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:44.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:44.589+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:44.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:44.590+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:44.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:44.700+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:44.700825+0000) 2024-04-27T04:07:45.320 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:45.004+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:45.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:45.004+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:45.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:45.004+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:45.321 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:45.005+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:45.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:44.993+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:44.994206+0000) 2024-04-27T04:07:45.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:45.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:45.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:45.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:45.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:45.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:45.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:45.007+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:45.702 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:45.500+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:45.501343+0000) 2024-04-27T04:07:45.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:45.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:45.702 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:45.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:45.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:45.568+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:45.703 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:45.569+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:45.961 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:45.702+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:45.703450+0000) 2024-04-27T04:07:46.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:45.961+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:46.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:45.961+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:46.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:45.961+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:46.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:45.962+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:46.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:45.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:46.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:45.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:46.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:45.980+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:46.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:45.981+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:46.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:45.995+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:45.996328+0000) 2024-04-27T04:07:46.594 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:46.502+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:46.503456+0000) 2024-04-27T04:07:46.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:46.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:46.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:46.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:46.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:46.595+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:46.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:46.596+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:46.960 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:46.705+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:46.706586+0000) 2024-04-27T04:07:47.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:46.986+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:47.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:46.986+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:47.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:46.986+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:47.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:46.986+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:47.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:46.997+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:46.998462+0000) 2024-04-27T04:07:47.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:46.960+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:47.220 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:46.960+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:47.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:46.960+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:47.221 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:46.961+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:47.626 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:47.504+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:47.505566+0000) 2024-04-27T04:07:47.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:47.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:47.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:47.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:47.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:47.627+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:47.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:47.628+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:47.940 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:47.707+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:47.708781+0000) 2024-04-27T04:07:47.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:47.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:47.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:47.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:47.941 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:47.940+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:48.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:47.941+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:48.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:48.000+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:48.001749+0000) 2024-04-27T04:07:48.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:48.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:48.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:48.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:48.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:48.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:48.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:48.029+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:48.654 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:48.509+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:48.509842+0000) 2024-04-27T04:07:48.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:48.655+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:48.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:48.655+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:48.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:48.655+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:48.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:48.656+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:48.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:48.712+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:48.712998+0000) 2024-04-27T04:07:49.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:49.003+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:49.003949+0000) 2024-04-27T04:07:49.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:49.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:49.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:49.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:49.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:49.048+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:49.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:49.049+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:49.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:48.971+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:49.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:48.971+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:49.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:48.971+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:49.322 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:48.971+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:49.573 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:07:49.452+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 82037 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:07:49.573 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:49.448+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:49.448844+0000) 2024-04-27T04:07:49.574 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:49.448+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:49.448903+0000) 2024-04-27T04:07:49.574 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:49.513+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:49.514225+0000) 2024-04-27T04:07:49.574 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:49.449+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:49.449822+0000) 2024-04-27T04:07:49.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:49.671+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:49.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:49.671+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:49.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:49.671+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:49.929 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:49.672+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:49.930 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:49.716+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:49.717273+0000) 2024-04-27T04:07:50.189 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:49.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:50.189 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:49.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:50.189 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:49.929+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:50.189 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:49.930+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:50.189 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:50.008+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:50.008890+0000) 2024-04-27T04:07:50.190 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:50.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:50.190 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:50.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:50.190 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:50.015+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:50.190 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:50.016+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:50.717 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:50.517+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:50.518520+0000) 2024-04-27T04:07:50.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:50.661+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:50.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:50.661+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:50.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:50.661+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:50.718 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:50.662+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:50.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:50.718+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:50.719314+0000) 2024-04-27T04:07:50.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:50.923+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:50.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:50.923+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:50.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:50.923+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:50.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:50.924+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:51.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:50.972+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:51.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:50.972+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:51.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:50.972+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:51.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:50.973+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:51.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:51.010+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:51.011251+0000) 2024-04-27T04:07:51.640 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:51.520+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:51.520834+0000) 2024-04-27T04:07:51.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:07:51.648 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:07:51.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:51.641+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:51.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:51.641+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:51.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:51.641+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:51.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:51.642+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:51.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:51.720+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:51.721690+0000) 2024-04-27T04:07:51.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:51.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:51.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:51.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:51.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:51.884+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:51.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:51.886+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:52.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:52.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:52.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:52.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:52.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:52.010+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:52.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:52.011+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:52.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:52.012+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:52.013561+0000) 2024-04-27T04:07:52.673 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:52.525+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:52.525937+0000) 2024-04-27T04:07:52.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:52.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:52.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:52.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:52.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:52.673+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:52.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:52.674+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:52.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:52.725+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:52.726679+0000) 2024-04-27T04:07:52.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:52.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:52.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:52.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:52.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:52.844+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:52.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:52.844+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:53.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:53.015+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:53.016758+0000) 2024-04-27T04:07:53.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:53.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:53.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:53.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:53.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:53.045+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:53.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:53.046+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:53.652 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:53.527+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:53.528121+0000) 2024-04-27T04:07:53.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:53.653+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:53.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:53.653+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:53.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:53.653+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:53.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:53.654+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:53.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:53.730+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:53.731776+0000) 2024-04-27T04:07:53.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:53.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:53.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:53.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:53.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:53.854+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:53.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:53.854+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:54.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:54.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:54.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:54.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:54.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:54.006+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:54.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:54.007+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:54.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:54.018+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:54.018844+0000) 2024-04-27T04:07:54.582 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:54.457+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:54.458795+0000) 2024-04-27T04:07:54.583 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:07:54.464+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 82077 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:07:54.583 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:54.458+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:54.459149+0000) 2024-04-27T04:07:54.583 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:54.532+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:54.533301+0000) 2024-04-27T04:07:54.584 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:54.457+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:54.458581+0000) 2024-04-27T04:07:54.718 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:07:54.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:54.819+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:54.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:54.819+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:54.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:54.819+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:54.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:54.820+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:54.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:54.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:54.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:54.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:54.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:54.624+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:54.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:54.625+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:54.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:54.734+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:54.735421+0000) 2024-04-27T04:07:55.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:54.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:55.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:54.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:55.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:54.987+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:55.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:54.988+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:55.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:55.020+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:55.021043+0000) 2024-04-27T04:07:55.663 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:55.535+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:55.535904+0000) 2024-04-27T04:07:55.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:55.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:55.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:55.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:55.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:55.834+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:55.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:55.835+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:55.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:55.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:55.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:55.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:55.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:55.664+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:55.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:55.665+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:55.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:55.736+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:55.737543+0000) 2024-04-27T04:07:56.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:55.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:56.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:55.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:56.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:55.992+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:56.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:55.993+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:56.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:56.022+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:56.023228+0000) 2024-04-27T04:07:56.658 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:56.537+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:56.538214+0000) 2024-04-27T04:07:56.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:56.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:56.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:56.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:56.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:56.659+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:56.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:56.660+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:56.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:56.738+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:56.739720+0000) 2024-04-27T04:07:56.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:56.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:56.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:56.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:56.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:56.827+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:56.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:56.828+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:57.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:57.026+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:57.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:57.026+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:57.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:57.026+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:57.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:57.026+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:57.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:57.026+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:57.027445+0000) 2024-04-27T04:07:57.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:57.539+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:57.540322+0000) 2024-04-27T04:07:57.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:07:57.720 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:07:57.790 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:07:57.790 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:07:57.791 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-27T04:07:57.791 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-04-27T04:07:57.828 INFO:tasks.cephadm:Compressing logs... 2024-04-27T04:07:57.828 DEBUG:teuthology.orchestra.run.smithi029:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-27T04:07:57.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:07:57.871 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:07:57.895 INFO:teuthology.orchestra.run.smithi029.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi029.njjezc.log 2024-04-27T04:07:57.895 INFO:teuthology.orchestra.run.smithi029.stderr:: No such file or directory 2024-04-27T04:07:57.896 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph.log 2024-04-27T04:07:57.896 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-mon.smithi029.log 2024-04-27T04:07:57.896 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi029.njjezc.log: /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-27T04:07:57.896 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-mon.smithi029.log: gzip -5 --verbose -- /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-27T04:07:57.896 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-osd.2.log: 87.2% -- replaced with /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-27T04:07:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-27T04:07:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-27T04:07:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi029.dyrjrd.log 2024-04-27T04:07:57.899 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi029.log 2024-04-27T04:07:57.899 INFO:teuthology.orchestra.run.smithi029.stderr: 90.5% -- replaced with /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-27T04:07:57.906 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi029.dyrjrd.log: gzip -5 --verbose -- /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-27T04:07:57.907 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi029.log: gzip -5 --verbose -- /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-27T04:07:57.908 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.6% -- replaced with /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-27T04:07:57.909 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-mgr.smithi029.ewvdzs.log 2024-04-27T04:07:57.921 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-volume.log: 94.7% 91.4% -- replaced with /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-27T04:07:57.922 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-27T04:07:57.930 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-mgr.smithi029.ewvdzs.log: -- replaced with /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi029.log.gz 2024-04-27T04:07:57.944 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cephadm.log: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-27T04:07:57.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:57.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:57.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:57.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:57.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:57.815+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:57.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:57.816+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:57.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:57.688+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:57.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:57.688+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:57.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:57.688+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:57.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:57.690+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:57.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:57.741+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:57.741871+0000) 2024-04-27T04:07:58.029 INFO:teuthology.orchestra.run.smithi029.stderr: 95.3% -- replaced with /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi029.dyrjrd.log.gz 2024-04-27T04:07:58.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:58.029+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:58.029835+0000) 2024-04-27T04:07:58.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:58.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:58.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:58.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:58.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:58.072+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:58.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:58.072+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:58.675 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:58.542+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:58.542933+0000) 2024-04-27T04:07:58.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:58.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:58.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:58.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:58.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:58.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:58.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:58.677+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:58.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:58.743+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:58.743955+0000) 2024-04-27T04:07:58.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:58.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:58.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:58.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:58.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:58.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:58.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:58.840+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:59.322 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:59.034+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:59.034832+0000) 2024-04-27T04:07:59.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:59.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:59.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:59.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:59.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:59.106+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:07:59.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:59.106+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:59.626 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:59.484+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:59.485176+0000) 2024-04-27T04:07:59.627 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:07:59.486+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:59.487755+0000) 2024-04-27T04:07:59.627 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:07:59.489+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 82117 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:07:59.627 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:59.478+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:59.478999+0000) 2024-04-27T04:07:59.627 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:59.544+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:59.545604+0000) 2024-04-27T04:07:59.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:59.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:59.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:59.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:59.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:59.839+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:07:59.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:07:59.840+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:59.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:59.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:59.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:59.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:59.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:59.718+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:07:59.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:59.719+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:07:59.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:07:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:07:59.746+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:07:59.746825+0000) 2024-04-27T04:08:00.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:00.037+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:00.038538+0000) 2024-04-27T04:08:00.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:00.109+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:00.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:00.109+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:00.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:00.109+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:00.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:00.109+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:00.633 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-mgr.smithi029.ewvdzs.log: file size changed while zipping 2024-04-27T04:08:00.633 INFO:teuthology.orchestra.run.smithi029.stderr: 90.0% -- replaced with /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-mgr.smithi029.ewvdzs.log.gz 2024-04-27T04:08:00.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:00.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:00.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:00.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:00.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:00.719+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:00.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:00.548+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:00.548829+0000) 2024-04-27T04:08:00.930 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:08:00.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:00.722+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:00.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:00.749+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:00.749821+0000) 2024-04-27T04:08:00.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:00.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:00.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:00.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:00.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:00.796+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:00.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:00.797+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:01.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:01.041+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:01.042151+0000) 2024-04-27T04:08:01.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:01.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:01.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:01.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:01.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:01.150+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:01.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:01.150+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:01.710 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:01.551+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:01.551831+0000) 2024-04-27T04:08:01.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:01.708+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:01.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:01.708+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:01.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:01.708+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:01.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:01.710+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:01.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:01.752+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:01.753047+0000) 2024-04-27T04:08:01.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:01.832+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:01.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:01.832+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:01.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:01.832+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:01.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:01.833+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:02.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:02.053+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:02.054081+0000) 2024-04-27T04:08:02.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:02.141+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:02.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:02.141+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:02.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:02.141+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:02.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:02.141+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:02.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:02.554+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:02.554833+0000) 2024-04-27T04:08:02.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:02.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:02.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:02.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:02.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:02.860+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:02.997 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:02.861+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:02.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:02.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:02.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:02.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:02.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:02.731+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:02.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:02.732+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:02.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:02.754+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:02.755665+0000) 2024-04-27T04:08:03.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:03.070+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:03.070982+0000) 2024-04-27T04:08:03.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:03.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:03.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:03.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:03.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:03.143+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:03.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:03.143+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:03.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:03.557+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:03.557834+0000) 2024-04-27T04:08:04.087 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:03.833+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:04.087 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:03.833+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:04.087 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:03.833+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:04.087 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:03.834+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:04.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:03.759+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:03.759913+0000) 2024-04-27T04:08:04.094 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:03.764+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:04.095 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:03.764+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:04.095 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:03.764+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:04.095 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:03.764+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:04.110 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-osd.2.log: file size changed while zipping 2024-04-27T04:08:04.110 INFO:teuthology.orchestra.run.smithi029.stderr: 90.4% -- replaced with /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-27T04:08:04.435 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:04.087+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:04.088008+0000) 2024-04-27T04:08:04.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:04.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:04.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:04.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:04.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:04.180+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:04.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:04.180+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:04.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:08:04.502+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 82153 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:08:04.722 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:04.502+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:04.503225+0000) 2024-04-27T04:08:04.722 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:04.501+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:04.502147+0000) 2024-04-27T04:08:04.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:04.496+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:04.497701+0000) 2024-04-27T04:08:04.723 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:04.560+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:04.561630+0000) 2024-04-27T04:08:04.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:04.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:04.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:04.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:04.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:04.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:04.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:04.744+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:04.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:04.762+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:04.762958+0000) 2024-04-27T04:08:05.000 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:04.833+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:05.000 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:04.833+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:05.000 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:04.833+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:05.000 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:04.834+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:05.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:05.096+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:05.096844+0000) 2024-04-27T04:08:05.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:05.191+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:05.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:05.191+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:05.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:05.191+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:05.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:05.192+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:05.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:05.565+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:05.566124+0000) 2024-04-27T04:08:06.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:05.756+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:06.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:05.756+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:06.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:05.756+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:06.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:05.757+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:06.104 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:05.764+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:05.765634+0000) 2024-04-27T04:08:06.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:05.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:06.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:05.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:06.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:05.807+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:06.105 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:05.808+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:06.366 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:06.102+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:06.103675+0000) 2024-04-27T04:08:06.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:06.186+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:06.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:06.186+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:06.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:06.186+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:06.367 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:06.187+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:06.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:06.569+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:06.569828+0000) 2024-04-27T04:08:07.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:06.768+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:06.768799+0000) 2024-04-27T04:08:07.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:06.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:07.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:06.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:07.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:06.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:07.037 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:06.801+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:07.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:06.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:07.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:06.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:07.038 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:06.764+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:07.039 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:06.765+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:07.164 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-osd.1.log: file size changed while zipping 2024-04-27T04:08:07.165 INFO:teuthology.orchestra.run.smithi029.stderr: 90.0% -- replaced with /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-27T04:08:07.323 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:07.115+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:07.115909+0000) 2024-04-27T04:08:07.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:07.227+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:07.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:07.227+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:07.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:07.227+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:07.324 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:07.228+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:07.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:07.571+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:07.572682+0000) 2024-04-27T04:08:07.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:07.772+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:07.773027+0000) 2024-04-27T04:08:07.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:07.799+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:07.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:07.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:07.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:07.800+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:07.998 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:07.800+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:07.999 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:07.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:07.999 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:07.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:07.999 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:07.738+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:07.999 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:07 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:07.739+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:08.273 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:08.121+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:08.122278+0000) 2024-04-27T04:08:08.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:08.273+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:08.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:08.273+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:08.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:08.273+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:08.575 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:08.274+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:08.576 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:08.575+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:08.576263+0000) 2024-04-27T04:08:08.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:08.777+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:08.777959+0000) 2024-04-27T04:08:08.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:08.799+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:08.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:08.799+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:08.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:08.799+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:08.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:08.800+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:08.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:08.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:08.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:08.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:08.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:08.702+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:08.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:08 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:08.703+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:09.284 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:09.126+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:09.127538+0000) 2024-04-27T04:08:09.577 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:08:09.513+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 82193 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:08:09.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:09.513+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:09.514066+0000) 2024-04-27T04:08:09.578 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:09.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:09.578 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:09.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:09.578 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:09.284+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:09.578 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:09.285+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:09.579 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:09.510+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:09.510918+0000) 2024-04-27T04:08:09.579 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:09.516+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:09.517703+0000) 2024-04-27T04:08:09.931 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:08:09.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:08:09.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:09.780+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:09.781682+0000) 2024-04-27T04:08:09.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:09.812+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:09.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:09.812+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:09.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:09.812+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:09.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:09.814+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:09.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:09.578+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:09.578905+0000) 2024-04-27T04:08:09.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:09.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:09.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:09.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:09.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:09.753+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:09.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:09 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:09.754+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:10.315 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:10.132+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:10.132800+0000) 2024-04-27T04:08:10.579 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:10.316+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:10.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:10.316+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:10.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:10.316+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:10.580 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:10.316+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:10.586 INFO:teuthology.orchestra.run.smithi029.stderr: 92.5% -- replaced with /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi029.njjezc.log.gz 2024-04-27T04:08:10.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:10.783+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:10.785159+0000) 2024-04-27T04:08:10.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:10.822+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:10.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:10.822+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:10.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:10.822+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:10.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:10.824+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:10.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:10.580+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:10.581069+0000) 2024-04-27T04:08:10.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:10.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:10.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:10.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:10.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:10.706+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:10.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:10 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:10.708+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:11.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:11.136+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:11.137799+0000) 2024-04-27T04:08:11.296 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:11.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:11.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:11.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:11.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:11.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:11.583 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:11.296+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:11.851 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:11.786+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:11.788522+0000) 2024-04-27T04:08:11.851 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:11.582+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:11.584546+0000) 2024-04-27T04:08:11.851 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:11.690+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:11.851 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:11.690+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:11.851 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:11.690+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:11.852 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:11.691+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:12.140 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:11.850+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:12.140 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:11.850+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:12.140 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:11.850+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:12.140 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:11 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:11.851+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:12.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:12.139+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:12.141479+0000) 2024-04-27T04:08:12.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:12.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:12.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:12.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:12.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:12.338+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:12.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:12.338+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:12.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:12.586+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:12.587902+0000) 2024-04-27T04:08:12.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:12.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:12.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:12.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:12.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:12.668+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:12.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:12.669+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:12.994 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:08:13.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:12.791+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:12.793232+0000) 2024-04-27T04:08:13.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:12.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:13.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:12.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:13.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:12.824+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:13.145 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:12 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:12.824+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:13.436 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:13.144+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:13.146190+0000) 2024-04-27T04:08:13.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:13.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:13.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:13.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:13.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:13.302+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:13.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:13.303+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:13.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:13.589+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:13.591374+0000) 2024-04-27T04:08:13.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:13.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:13.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:13.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:13.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:13.672+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:13.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:13.673+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:14.148 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:13.793+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:13.795694+0000) 2024-04-27T04:08:14.148 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:13.808+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:14.148 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:13.808+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:14.148 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:13.808+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:14.149 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:13 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:13.810+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:14.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:14.147+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:14.149395+0000) 2024-04-27T04:08:14.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:14.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:14.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:14.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:14.437 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:14.257+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:14.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:14.257+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:14.714 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:08:14.524+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 82233 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:08:14.715 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:14.518+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:14.520236+0000) 2024-04-27T04:08:14.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:14.519+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:14.521110+0000) 2024-04-27T04:08:14.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:14.592+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:14.594458+0000) 2024-04-27T04:08:14.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:14.714+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:14.715 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:14.714+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:14.716 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:14.714+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:14.716 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:14.517+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:14.519197+0000) 2024-04-27T04:08:14.907 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-osd.0.log: file size changed while zipping 2024-04-27T04:08:14.908 INFO:teuthology.orchestra.run.smithi029.stderr: 95.1% -- replaced with /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-27T04:08:14.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:14.716+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:14.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:14.797+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:14.798827+0000) 2024-04-27T04:08:14.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:14.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:14.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:14.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:14.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:14.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:14.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:14 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:14.838+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:15.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:15.150+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:15.152559+0000) 2024-04-27T04:08:15.278 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:15.277+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:15.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:15.277+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:15.279 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:15.277+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:15.595 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:15.278+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:15.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:15.595+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:15.596965+0000) 2024-04-27T04:08:15.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:15.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:15.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:15.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:15.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:15.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:15.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:15.667+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:15.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:15.790+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:15.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:15.790+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:15.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:15.790+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:15.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:15.791+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:15.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:15 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:15.801+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:15.803721+0000) 2024-04-27T04:08:16.253 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:16.154+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:16.156465+0000) 2024-04-27T04:08:16.598 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:16.306+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:16.598 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:16.306+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:16.598 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:16.306+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:16.598 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:16.306+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:16.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:16.762+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:16.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:16.762+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:16.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:16.762+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:16.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:16.764+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:16.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:16.806+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:16.808786+0000) 2024-04-27T04:08:16.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:16.597+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:16.599338+0000) 2024-04-27T04:08:16.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:16.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:16.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:16.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:16.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:16.624+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:16.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:16 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:16.625+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:17.295 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:17.158+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:17.160537+0000) 2024-04-27T04:08:17.602 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:17.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:17.602 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:17.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:17.602 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:17.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:17.602 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:17.295+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:17.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:17.734+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:17.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:17.734+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:17.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:17.734+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:17.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:17.735+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:17.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:17.812+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:17.813886+0000) 2024-04-27T04:08:17.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:17.600+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:17.602656+0000) 2024-04-27T04:08:17.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:17.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:17.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:17.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:17.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:17.621+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:17.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:17 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:17.622+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:18.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:18.162+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:18.164188+0000) 2024-04-27T04:08:18.599 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:18.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:18.599 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:18.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:18.600 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:18.341+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:18.600 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:18.342+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:18.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:18.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:18.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:18.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:18.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:18.599+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:18.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:18.600+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:18.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:18.605+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:18.607686+0000) 2024-04-27T04:08:18.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:18.696+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:18.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:18.696+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:18.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:18.696+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:18.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:18.696+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:18.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:18 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:18.816+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:18.818133+0000) 2024-04-27T04:08:19.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:19.165+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:19.167310+0000) 2024-04-27T04:08:19.609 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:08:19.534+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 82273 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:08:19.609 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:19.530+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:19.531859+0000) 2024-04-27T04:08:19.610 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:19.529+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:19.531335+0000) 2024-04-27T04:08:19.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:19.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:19.610 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:19.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:19.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:19.383+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:19.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:19.384+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:19.611 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:19.529+0000 7f30ec0a3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:19.531612+0000) 2024-04-27T04:08:19.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:19.608+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:19.610709+0000) 2024-04-27T04:08:19.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:19.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:19.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:19.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:19.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:19.640+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:19.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:19.641+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:19.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:19.736+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:19.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:19.736+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:19.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:19.736+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:19.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:19.737+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:19.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:19 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:19.819+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:19.821348+0000) 2024-04-27T04:08:20.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:20.168+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:20.169824+0000) 2024-04-27T04:08:20.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:20.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:20.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:20.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:20.612 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:20.359+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:20.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:20.359+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:20.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:20.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:20.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:20.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:20.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:20.739+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:20.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:20.739+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:20.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:20.822+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:20.824068+0000) 2024-04-27T04:08:20.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:20.611+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:20.613624+0000) 2024-04-27T04:08:20.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:20.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:20.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:20.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:20.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:20.660+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:20.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:20 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:20.661+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:21.315 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:21.170+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:21.172431+0000) 2024-04-27T04:08:21.615 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:21.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:21.615 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:21.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:21.615 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:21.315+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:21.615 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:21.315+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:21.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:21.693+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:21.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:21.693+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:21.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:21.693+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:21.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:21.693+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:21.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:21.824+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:21.826384+0000) 2024-04-27T04:08:21.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:21.614+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:21.616251+0000) 2024-04-27T04:08:21.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:21.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:21.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:21.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:21.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:21.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:21.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:21 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:21.694+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:22.325 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:22.173+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:22.175536+0000) 2024-04-27T04:08:22.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:22.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:22.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:22.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:22.618 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:22.354+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:22.619 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:22.355+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:22.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:22.617+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:22.619183+0000) 2024-04-27T04:08:22.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:22.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:22.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:22.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:22.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:22.693+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:22.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:22.695+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:22.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:22.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:22.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:22.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:22.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:22.675+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:22.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:22.675+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:22.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:22 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:22.827+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:22.829287+0000) 2024-04-27T04:08:23.308 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:23.176+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:23.177771+0000) 2024-04-27T04:08:23.308 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:23.309+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:23.621 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:23.309+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:23.621 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:23.309+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:23.621 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:23.309+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:23.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:23.725+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:23.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:23.725+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:23.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:23.725+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:23.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:23.726+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:23.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:23.830+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:23.831475+0000) 2024-04-27T04:08:23.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:23.621+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:23.622149+0000) 2024-04-27T04:08:23.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:23.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:23.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:23.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:23.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:23.655+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:23.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:23 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:23.656+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:24.294 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:24.179+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:24.180693+0000) 2024-04-27T04:08:24.553 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:08:24.544+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 82313 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:08:24.553 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:24.540+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:24.540834+0000) 2024-04-27T04:08:24.553 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:24.541+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:24.542217+0000) 2024-04-27T04:08:24.554 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:24.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:24.554 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:24.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:24.554 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:24.295+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:24.554 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:24.295+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:24.554 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:24.541+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:24.541944+0000) 2024-04-27T04:08:24.833 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:24.680+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:24.834 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:24.680+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:24.834 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:24.680+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:24.834 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:24.681+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:24.834 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:24.624+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:24.625154+0000) 2024-04-27T04:08:24.834 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:24.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:24.834 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:24.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:24.835 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:24.632+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:24.835 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:24.632+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:24.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:08:24.996 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:08:25.153 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:24 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:24.833+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:24.834537+0000) 2024-04-27T04:08:25.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:25.182+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:25.183073+0000) 2024-04-27T04:08:25.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:25.314+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:25.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:25.314+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:25.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:25.314+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:25.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:25.315+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:25.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:25.626+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:25.627282+0000) 2024-04-27T04:08:25.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:25.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:25.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:25.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:25.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:25.628+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:25.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:25.629+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:25.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:25.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:25.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:25.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:25.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:25.658+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:25.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:25.659+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:26.154 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:25 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:25.837+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:25.837963+0000) 2024-04-27T04:08:26.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:26.185+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:26.186533+0000) 2024-04-27T04:08:26.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:26.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:26.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:26.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:26.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:26.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:26.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:26.295+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:26.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:26.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:26.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:26.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:26.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:26.640+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:26.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:26.640+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:26.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:26.628+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:26.629553+0000) 2024-04-27T04:08:26.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:26.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:26.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:26.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:26.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:26.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:26.722 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:26.644+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-mon.smithi029.log: file size changed while zipping 2024-04-27T04:08:26.828 INFO:teuthology.orchestra.run.smithi029.stderr: 98.2% -- replaced with /var/log/ceph/4c587076-0439-11ef-bc93-c7b262605968/ceph-mon.smithi029.log.gz 2024-04-27T04:08:26.829 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-04-27T04:08:26.830 INFO:teuthology.orchestra.run.smithi029.stderr:real 0m28.946s 2024-04-27T04:08:26.830 INFO:teuthology.orchestra.run.smithi029.stderr:user 1m14.629s 2024-04-27T04:08:26.830 INFO:teuthology.orchestra.run.smithi029.stderr:sys 0m1.645s 2024-04-27T04:08:27.154 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:26 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:26.839+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:26.840292+0000) 2024-04-27T04:08:27.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:27.187+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:27.188650+0000) 2024-04-27T04:08:27.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:27.268+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:27.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:27.268+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:27.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:27.268+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:27.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:27.268+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:27.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:27.678+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:27.719 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:27.678+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:27.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:27.678+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:27.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:27.680+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:27.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:27.631+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:27.632731+0000) 2024-04-27T04:08:27.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:27.654+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:27.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:27.654+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:27.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:27.654+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:27.721 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:27.655+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:28.062 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:08:28.154 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:27 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:27.843+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:27.844577+0000) 2024-04-27T04:08:28.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:28.190+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:28.190822+0000) 2024-04-27T04:08:28.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:28.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:28.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:28.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:28.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:28.278+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:28.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:28.278+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:28.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:28.636+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:28.637495+0000) 2024-04-27T04:08:28.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:28.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:28.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:28.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:28.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:28.666+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:28.720 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:28.667+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:28.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:28.685+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:28.720 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:28.685+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:28.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:28.685+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:28.721 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:28.687+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:29.154 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:28 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:28.846+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:28.846823+0000) 2024-04-27T04:08:29.438 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:29.192+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:29.193645+0000) 2024-04-27T04:08:29.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:29.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:29.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:29.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:29.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:29.307+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:29.439 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:29.308+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:29.700 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:29.549+0000 7f30ed0a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:29.549894+0000) 2024-04-27T04:08:29.700 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:08:29.553+0000 7f35f8d65700 -1 mon.smithi029@0(probing) e2 get_health_metrics reporting 82353 slow ops, oldest is log(1 entries from seq 819 at 2024-04-27T02:06:22.830345+0000) 2024-04-27T04:08:29.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:29.550+0000 7f3a2115b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:29.551194+0000) 2024-04-27T04:08:29.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:29.700+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:29.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:29.700+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:29.701 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:29.700+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:29.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:29.549+0000 7f49288c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:29.550332+0000) 2024-04-27T04:08:29.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:29.639+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:29.640603+0000) 2024-04-27T04:08:29.701 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:29.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:29.702 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:29.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:29.702 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:29.669+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:29.702 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:29.670+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:29.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:29.701+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:29.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:29 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:29.848+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:29.849521+0000) 2024-04-27T04:08:30.326 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:30.196+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:30.196920+0000) 2024-04-27T04:08:30.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:30.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:30.643 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:30.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:30.644 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:30.343+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:30.644 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:30.344+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:30.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:30.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:30.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:30.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:30.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:30.644+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:30.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:30.644+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:30.645647+0000) 2024-04-27T04:08:30.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:30.645+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:30.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:30.728+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:30.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:30.728+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:30.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:30.729+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:30.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:30.730+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:30.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:30 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:30.853+0000 7f3a12cfd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:30.854561+0000) 2024-04-27T04:08:31.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:08:31.064 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:08:31.134 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:08:31.134 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:08:31.135 INFO:tasks.cephadm:Removing cluster... 2024-04-27T04:08:31.135 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4c587076-0439-11ef-bc93-c7b262605968 --force 2024-04-27T04:08:31.291 INFO:teuthology.orchestra.run.smithi029.stdout:Deleting cluster with fsid: 4c587076-0439-11ef-bc93-c7b262605968 2024-04-27T04:08:31.326 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:31.198+0000 7f30dd444700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:31.199076+0000) 2024-04-27T04:08:31.612 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:08:31 smithi029 systemd[1]: Stopping Ceph mon.smithi029 for 4c587076-0439-11ef-bc93-c7b262605968... 2024-04-27T04:08:31.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:31.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:31.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:31.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:31.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:31.369+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:31.613 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:31.370+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:31.969 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:08:31.741+0000 7f35fbd6b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi029 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T04:08:31.970 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029[26765]: 2024-04-27T04:08:31.741+0000 7f35fbd6b700 -1 mon.smithi029@0(probing) e2 *** Got Signal Terminated *** 2024-04-27T04:08:31.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:31.749+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:31.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:31.749+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:31.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:31.749+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:31.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:31.751+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:31.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:31.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:31.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:31.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:31.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:31.638+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:31.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:31.639+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:31.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:31 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:31.649+0000 7f4919c65700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-27T03:08:31.650679+0000) 2024-04-27T04:08:32.590 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:32.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:32.590 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:32.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:32.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:32.328+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:32.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:32.328+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:32.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:32.714+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:32.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:32.714+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:32.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:32.714+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:32.971 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:32.714+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:32.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:32.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:32.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:32.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:32.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:32.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:32.972 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:32 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:32.592+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:33.415 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:08:33 smithi029 podman[84606]: 2024-04-27 04:08:33.083144593 +0000 UTC m=+1.530705030 container died 33130493e80ded41989df919b6ae566012e0ee93d10a59b55f00bb543ccc9c59 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0, maintainer=Guillaume Abrioux , RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.build-date=20231212, GIT_CLEAN=True) 2024-04-27T04:08:33.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:33.293+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:33.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:33.293+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:33.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:33.293+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:33.415 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:33.294+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:33.686 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:08:33 smithi029 podman[84606]: 2024-04-27 04:08:33.617644913 +0000 UTC m=+2.065205393 container cleanup 33130493e80ded41989df919b6ae566012e0ee93d10a59b55f00bb543ccc9c59 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=HEAD, org.label-schema.build-date=20231212, ceph=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-27T04:08:33.686 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:08:33 smithi029 bash[84606]: ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029 2024-04-27T04:08:33.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:33.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:33.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:33.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:33.687 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:33.558+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:33.688 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:33.559+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:33.969 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:08:33 smithi029 podman[84619]: 2024-04-27 04:08:33.887558912 +0000 UTC m=+0.800417397 container remove 33130493e80ded41989df919b6ae566012e0ee93d10a59b55f00bb543ccc9c59 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-mon-smithi029, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux ) 2024-04-27T04:08:33.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:33.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:33.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:33.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:33.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:33.686+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:33.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:33 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:33.687+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:34.309 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:08:34 smithi029 systemd[1]: ceph-4c587076-0439-11ef-bc93-c7b262605968@mon.smithi029.service: Deactivated successfully. 2024-04-27T04:08:34.309 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:08:34 smithi029 systemd[1]: Stopped Ceph mon.smithi029 for 4c587076-0439-11ef-bc93-c7b262605968. 2024-04-27T04:08:34.309 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Apr 27 04:08:34 smithi029 systemd[1]: ceph-4c587076-0439-11ef-bc93-c7b262605968@mon.smithi029.service: Consumed 1h 54min 4.231s CPU time. 2024-04-27T04:08:34.586 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:34.309+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:34.586 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:34.309+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:34.586 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:34.309+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:34.586 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:34.310+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:34.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:34.679+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:34.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:34.679+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:34.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:34.679+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:34.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:34.680+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:34.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:34.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:34.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:34.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:34.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:34.586+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:34.971 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:34 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:34.587+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:35.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:35.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:35.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:35.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:35.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:35.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:35.626 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:35.282+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:35.898 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:35.693+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:35.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:35.693+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:35.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:35.693+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:35.899 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:35.693+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:35.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:35.626+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:35.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:35.626+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:35.899 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:35.626+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:35.900 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:35 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:35.627+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:36.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:36.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:36.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:36.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:36.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:36.281+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:36.591 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:36.281+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:36.841 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:36.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:36.841 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:36.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:36.842 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:36.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:36.842 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:36.592+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:36.842 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:36.688+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:36.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:36.688+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:36.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:36.688+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:36.843 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:36 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:36.689+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:37.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:37.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:37.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:37.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:37.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:37.240+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:37.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:37.240+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:37.732 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:37.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:37.732 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:37.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:37.732 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:37.590+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:37.733 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:37.591+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:38.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:37.732+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:38.079 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:37.732+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:38.079 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:37.732+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:38.079 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:37 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:37.734+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:38.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:38.211+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:38.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:38.212+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:38.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:38.212+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:38.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:38.212+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:38.944 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:38.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:38.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:38.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:38.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:38.592+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:38.945 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:38.593+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:38.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:38.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:38.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:38.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:38.945 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:38.743+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:38.946 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:38 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:38.744+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:39.539 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:39.256+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:39.539 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:39.256+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:39.539 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:39.256+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:39.540 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:39.256+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:39.892 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:39.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:39.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:39.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:39.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:39.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:39.893 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:39.633+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:39.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:39.782+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:39.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:39.782+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:39.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:39.782+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:39.893 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:39 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:39.784+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:40.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:40.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:40.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:40.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:40.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:40.225+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:40.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:40.226+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:40.812 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:40.613+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:40.812 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:40.613+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:40.812 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:40.613+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:40.812 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:40.614+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:41.176 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:40.812+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:41.176 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:40.812+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:41.176 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:40.812+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:41.176 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:40 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:40.813+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:41.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:41.176+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:41.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:41.176+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:41.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:41.176+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:41.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:41.177+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:41.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:41.821+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:41.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:41.821+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:41.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:41.821+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:41.970 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:41.822+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:41.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:41.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:41.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:41.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:41.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:41.633+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:41.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:41 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:41.634+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:42.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:42.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:42.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:42.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:42.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:42.169+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:42.417 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:42.169+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:42.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:42.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:42.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:42.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:42.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:42.591+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:42.719 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:42.592+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:43.129 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:42.849+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:43.129 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:42.849+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:43.129 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:42.849+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:43.130 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:42 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:42.849+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:43.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:43.129+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:43.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:43.129+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:43.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:43.129+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:43.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:43.130+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:43.840 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:43.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:43.841 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:43.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:43.841 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:43.579+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:43.841 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:43.580+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:44.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:43.877+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:44.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:43.877+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:44.109 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:43.877+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:44.110 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:43 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:43.877+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:44.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:44.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:44.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:44.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:44.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:44.110+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:44.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:44.110+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:44.800 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:44.626+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:44.800 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:44.626+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:44.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:44.626+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:44.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:44.627+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:45.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:44.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:45.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:44.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:45.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:44.837+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:45.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:44 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:44.838+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:45.087 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:45 smithi029 systemd[1]: Stopping Ceph osd.0 for 4c587076-0439-11ef-bc93-c7b262605968... 2024-04-27T04:08:45.469 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:45.304+0000 7f3a1e956700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T04:08:45.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:45.304+0000 7f3a1e956700 -1 osd.0 46 *** Got signal Terminated *** 2024-04-27T04:08:45.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:45.304+0000 7f3a1e956700 -1 osd.0 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T04:08:45.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:45.087+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:45.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:45.087+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:45.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:45.087+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:45.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:45.087+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:45.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:45.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:45.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:45.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:45.883 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:45.580+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:45.884 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:45.580+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:46.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:45.884+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:46.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:45.884+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:46.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:45.884+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:46.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:45 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:45.885+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:46.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:46.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:46.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:46.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:46.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:46.043+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:46.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:46.044+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:46.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:46.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:46.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:46.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:46.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:46.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:46.930 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:46.532+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:47.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:46.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:47.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:46.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:47.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:46.930+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:47.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:46 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:46.931+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:47.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:47.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:47.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:47.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:47.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:47.028+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:47.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:47.029+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:47.959 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:47.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:47.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:47.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:47.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:47.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:47.960 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:47.495+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:48.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:47.960+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:48.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:47.960+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:48.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:47.960+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:48.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:47 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:47.961+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:48.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:48.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:48.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:48.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:48.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:48.041+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:48.221 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:48.042+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:48.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:48.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:48.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:48.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:48.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:48.512+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:48.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:48.512+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:49.469 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:48.988+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:49.469 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:48.988+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:49.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:48.988+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:49.470 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:48 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:48.988+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:49.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:49.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:49.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:49.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:49.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:49.032+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:49.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:49.033+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:49.962 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:49.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:49.965 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:49.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:49.965 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:49.495+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:49.966 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:49.495+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:50.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:49.963+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:50.219 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:49.963+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:50.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:49.963+0000 7f3a1af4c700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.630593+0000 (oldest deadline 2024-04-27T04:03:07.630593+0000) 2024-04-27T04:08:50.220 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:49 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0[41683]: 2024-04-27T04:08:49.964+0000 7f3a1af4c700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24363.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:50.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:50.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:50.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:50.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:50.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:50.053+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:50.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:50.053+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:50.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:50.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:50.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:50.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:50.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:50.498+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:50.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:50 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:50.498+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:51.275 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:51 smithi029 podman[85578]: 2024-04-27 04:08:51.016761163 +0000 UTC m=+5.838265030 container died b4688a5b18ec86b73e7710e973d62f99e22a32ed8e6c36b1dff5f961c1842a8e (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.build-date=20231212, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, CEPH_POINT_RELEASE=-18.2.0) 2024-04-27T04:08:51.275 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:51.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:51.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:51.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:51.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:51.042+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:51.276 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:51.042+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:51.526 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:51 smithi029 podman[85578]: 2024-04-27 04:08:51.392774995 +0000 UTC m=+6.214278875 container cleanup b4688a5b18ec86b73e7710e973d62f99e22a32ed8e6c36b1dff5f961c1842a8e (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-27T04:08:51.527 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:51 smithi029 bash[85578]: ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0 2024-04-27T04:08:51.969 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:51 smithi029 podman[85595]: 2024-04-27 04:08:51.712084394 +0000 UTC m=+0.691505323 container remove b4688a5b18ec86b73e7710e973d62f99e22a32ed8e6c36b1dff5f961c1842a8e (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_CLEAN=True, RELEASE=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T04:08:51.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:51.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:51.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:51.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:51.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:51.527+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:51.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:51 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:51.527+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:52.314 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:52.091+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:52.314 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:52.091+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:52.314 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:52.091+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:52.314 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:52.091+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:52.315 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:52 smithi029 podman[85692]: 2024-04-27 04:08:52.064037094 +0000 UTC m=+0.113219114 container create 49b9e25cf2d72f65f9ce1ff6c970b2f76090881f8e2629f03499fb4830c9b9d9 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, CEPH_POINT_RELEASE=-18.2.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0) 2024-04-27T04:08:52.315 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:52 smithi029 podman[85692]: 2024-04-27 04:08:51.96992427 +0000 UTC m=+0.019106298 image pull dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007 2024-04-27T04:08:52.315 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:52 smithi029 podman[85692]: 2024-04-27 04:08:52.311683643 +0000 UTC m=+0.360865669 container init 49b9e25cf2d72f65f9ce1ff6c970b2f76090881f8e2629f03499fb4830c9b9d9 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-27T04:08:52.315 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:52 smithi029 podman[85692]: 2024-04-27 04:08:52.315166782 +0000 UTC m=+0.364348804 container start 49b9e25cf2d72f65f9ce1ff6c970b2f76090881f8e2629f03499fb4830c9b9d9 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, ceph=True) 2024-04-27T04:08:52.584 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:52 smithi029 podman[85692]: 2024-04-27 04:08:52.420605373 +0000 UTC m=+0.469787409 container attach 49b9e25cf2d72f65f9ce1ff6c970b2f76090881f8e2629f03499fb4830c9b9d9 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, ceph=True, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308) 2024-04-27T04:08:52.584 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:52.518+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:52.584 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:52.518+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:52.585 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:52.518+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:52.585 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:52 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:52.519+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:52.965 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:52 smithi029 podman[85692]: 2024-04-27 04:08:52.618419112 +0000 UTC m=+0.667601121 container died 49b9e25cf2d72f65f9ce1ff6c970b2f76090881f8e2629f03499fb4830c9b9d9 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20231212, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD) 2024-04-27T04:08:53.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:53.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:53.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:53.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:53.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:53.054+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:53.220 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:53.054+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:53.574 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:53 smithi029 podman[85719]: 2024-04-27 04:08:53.304274438 +0000 UTC m=+0.676583106 container remove 49b9e25cf2d72f65f9ce1ff6c970b2f76090881f8e2629f03499fb4830c9b9d9 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux ) 2024-04-27T04:08:53.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:53 smithi029 systemd[1]: ceph-4c587076-0439-11ef-bc93-c7b262605968@osd.0.service: Deactivated successfully. 2024-04-27T04:08:53.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:53 smithi029 systemd[1]: Stopped Ceph osd.0 for 4c587076-0439-11ef-bc93-c7b262605968. 2024-04-27T04:08:53.575 INFO:journalctl@ceph.osd.0.smithi029.stdout:Apr 27 04:08:53 smithi029 systemd[1]: ceph-4c587076-0439-11ef-bc93-c7b262605968@osd.0.service: Consumed 59min 49.913s CPU time. 2024-04-27T04:08:53.576 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:53 smithi029 systemd[1]: Stopping Ceph osd.1 for 4c587076-0439-11ef-bc93-c7b262605968... 2024-04-27T04:08:53.576 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:53.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:53.576 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:53.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:53.576 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:53.532+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:53.576 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:53.533+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:53.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:53.799+0000 7f49268c0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T04:08:53.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:53.799+0000 7f49268c0700 -1 osd.1 46 *** Got signal Terminated *** 2024-04-27T04:08:53.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:53 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:53.799+0000 7f49268c0700 -1 osd.1 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T04:08:54.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:54.093+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:54.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:54.093+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:54.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:54.093+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:54.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:54.094+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:54.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:54.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:54.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:54.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:54.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:54.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:54.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:54 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:54.524+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:55.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:55.138+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:55.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:55.138+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:55.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:55.138+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:55.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:55.138+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:55.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:55.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:55.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:55.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:55.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:55.497+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:55.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:55 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:55.498+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:56.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:56.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:56.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:56.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:56.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:56.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:56.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:56.118+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:56.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:56.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:56.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:56.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:56.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:56.485+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:56.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:56 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:56.486+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:57.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:57.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:57.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:57.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:57.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:57.118+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:57.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:57.118+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:57.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:57.478+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:57.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:57.478+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:57.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:57.478+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:57.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:57 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:57.479+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:58.469 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:58.156+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:58.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:58.156+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:58.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:58.156+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:58.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:58.156+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:58.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:58.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:58.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:58.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:58.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:58.524+0000 7f4921eb4700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.831657+0000 (oldest deadline 2024-04-27T04:03:07.831657+0000) 2024-04-27T04:08:58.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:58 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1[47316]: 2024-04-27T04:08:58.525+0000 7f4921eb4700 -1 osd.1 46 get_health_metrics reporting 225 slow ops, oldest is osd_op(client.24363.0:13 3.2 3:45d72e17:::1000000000a.00000000:head [write 0~1954 in=1954b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:59.470 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:59 smithi029 podman[85811]: 2024-04-27 04:08:59.411831459 +0000 UTC m=+5.741774904 container died 7123e1ad821935efdbc05ff600f365f3961e31d0f44af8e5ed564c8824c5cdc6 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.29.1, RELEASE=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-27T04:08:59.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:59.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:59.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:59.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:59.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:59.205+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:08:59.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:08:59 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:08:59.206+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:08:59.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:59 smithi029 podman[85811]: 2024-04-27 04:08:59.780421287 +0000 UTC m=+6.110364741 container cleanup 7123e1ad821935efdbc05ff600f365f3961e31d0f44af8e5ed564c8824c5cdc6 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1, CEPH_POINT_RELEASE=-18.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-27T04:08:59.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:08:59 smithi029 bash[85811]: ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1 2024-04-27T04:09:00.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:09:00 smithi029 podman[85828]: 2024-04-27 04:09:00.066574032 +0000 UTC m=+0.650333897 container remove 7123e1ad821935efdbc05ff600f365f3961e31d0f44af8e5ed564c8824c5cdc6 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1, ceph=True, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T04:09:00.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:00.223+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:09:00.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:00.223+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:09:00.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:00.223+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:09:00.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:00 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:00.223+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:09:00.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:09:00 smithi029 podman[85921]: 2024-04-27 04:09:00.317582007 +0000 UTC m=+0.018877942 image pull dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007 2024-04-27T04:09:00.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:09:00 smithi029 podman[85921]: 2024-04-27 04:09:00.419584323 +0000 UTC m=+0.120880258 container create b3c2df27992d748523289091dedb658b9a4655fc8ff691bbff78989399ed5544 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=HEAD, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308) 2024-04-27T04:09:00.664 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:09:00 smithi029 podman[85921]: 2024-04-27 04:09:00.665850039 +0000 UTC m=+0.367145975 container init b3c2df27992d748523289091dedb658b9a4655fc8ff691bbff78989399ed5544 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-27T04:09:00.961 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:09:00 smithi029 podman[85921]: 2024-04-27 04:09:00.669004316 +0000 UTC m=+0.370300257 container start b3c2df27992d748523289091dedb658b9a4655fc8ff691bbff78989399ed5544 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.0, io.buildah.version=1.29.1, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-27T04:09:00.961 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:09:00 smithi029 podman[85921]: 2024-04-27 04:09:00.766474396 +0000 UTC m=+0.467770339 container attach b3c2df27992d748523289091dedb658b9a4655fc8ff691bbff78989399ed5544 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.schema-version=1.0, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1) 2024-04-27T04:09:01.219 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:09:00 smithi029 podman[85921]: 2024-04-27 04:09:00.962593914 +0000 UTC m=+0.663889849 container died b3c2df27992d748523289091dedb658b9a4655fc8ff691bbff78989399ed5544 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_BRANCH=HEAD) 2024-04-27T04:09:01.598 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:01.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:09:01.598 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:01.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:09:01.598 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:01.255+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:09:01.598 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:01 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:01.255+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:09:01.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:01 smithi029 systemd[1]: Stopping Ceph osd.2 for 4c587076-0439-11ef-bc93-c7b262605968... 2024-04-27T04:09:01.969 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:09:01 smithi029 podman[85948]: 2024-04-27 04:09:01.599313739 +0000 UTC m=+0.626283676 container remove b3c2df27992d748523289091dedb658b9a4655fc8ff691bbff78989399ed5544 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.0, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20231212, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-04-27T04:09:01.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:09:01 smithi029 systemd[1]: ceph-4c587076-0439-11ef-bc93-c7b262605968@osd.1.service: Deactivated successfully. 2024-04-27T04:09:01.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:09:01 smithi029 systemd[1]: Stopped Ceph osd.1 for 4c587076-0439-11ef-bc93-c7b262605968. 2024-04-27T04:09:01.970 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 27 04:09:01 smithi029 systemd[1]: ceph-4c587076-0439-11ef-bc93-c7b262605968@osd.1.service: Consumed 1min 16.515s CPU time. 2024-04-27T04:09:02.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:02.077+0000 7f30ea8a0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T04:09:02.470 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:02.077+0000 7f30ea8a0700 -1 osd.2 46 *** Got signal Terminated *** 2024-04-27T04:09:02.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:02.077+0000 7f30ea8a0700 -1 osd.2 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T04:09:02.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:02.252+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:09:02.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:02.252+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:09:02.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:02.252+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:09:02.471 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:02 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:02.252+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:09:03.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:03.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:09:03.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:03.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:09:03.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:03.294+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:09:03.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:03 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:03.294+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:09:04.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:04.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:09:04.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:04.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:09:04.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:04.324+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:09:04.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:04 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:04.324+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:09:05.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:05.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:09:05.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:05.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:09:05.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:05.365+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:09:05.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:05 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:05.365+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:09:06.719 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:06.332+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6804 osd.3 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:09:06.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:06.332+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6812 osd.4 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:09:06.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:06.332+0000 7f30e5693700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.52:6820 osd.5 ever on either front or back, first ping sent 2024-04-27T04:02:47.734407+0000 (oldest deadline 2024-04-27T04:03:07.734407+0000) 2024-04-27T04:09:06.720 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:06 smithi029 ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2[52834]: 2024-04-27T04:09:06.332+0000 7f30e5693700 -1 osd.2 46 get_health_metrics reporting 103 slow ops, oldest is osd_op(client.24363.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-04-27T04:09:07.924 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:07 smithi029 podman[86028]: 2024-04-27 04:09:07.641598866 +0000 UTC m=+5.665505490 container died f8ecf2655a409e79442b26ed06f9344a4fd4950c07254c2fb68e65f6d22acc74 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-27T04:09:08.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:08 smithi029 podman[86028]: 2024-04-27 04:09:08.032839093 +0000 UTC m=+6.056745715 container cleanup f8ecf2655a409e79442b26ed06f9344a4fd4950c07254c2fb68e65f6d22acc74 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2, CEPH_POINT_RELEASE=-18.2.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.build-date=20231212, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-27T04:09:08.219 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:08 smithi029 bash[86028]: ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2 2024-04-27T04:09:08.616 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:08 smithi029 podman[86048]: 2024-04-27 04:09:08.277195581 +0000 UTC m=+0.626882519 container remove f8ecf2655a409e79442b26ed06f9344a4fd4950c07254c2fb68e65f6d22acc74 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2, maintainer=Guillaume Abrioux , ceph=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20231212) 2024-04-27T04:09:08.869 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:08 smithi029 podman[86139]: 2024-04-27 04:09:08.521225976 +0000 UTC m=+0.019075788 image pull dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007 2024-04-27T04:09:08.870 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:08 smithi029 podman[86139]: 2024-04-27 04:09:08.637244323 +0000 UTC m=+0.135094158 container create 8e37ab9d43fd146b4de1ba7d50e845ce55a9095be2db00019778bc50ef799984 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, ceph=True, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308) 2024-04-27T04:09:08.870 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:08 smithi029 podman[86139]: 2024-04-27 04:09:08.867953113 +0000 UTC m=+0.365802906 container init 8e37ab9d43fd146b4de1ba7d50e845ce55a9095be2db00019778bc50ef799984 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, RELEASE=HEAD) 2024-04-27T04:09:08.870 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:08 smithi029 podman[86139]: 2024-04-27 04:09:08.871146188 +0000 UTC m=+0.368995980 container start 8e37ab9d43fd146b4de1ba7d50e845ce55a9095be2db00019778bc50ef799984 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0) 2024-04-27T04:09:09.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:08 smithi029 podman[86139]: 2024-04-27 04:09:08.993648553 +0000 UTC m=+0.491498354 container attach 8e37ab9d43fd146b4de1ba7d50e845ce55a9095be2db00019778bc50ef799984 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T04:09:09.454 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:09 smithi029 podman[86139]: 2024-04-27 04:09:09.167796338 +0000 UTC m=+0.665646129 container died 8e37ab9d43fd146b4de1ba7d50e845ce55a9095be2db00019778bc50ef799984 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-27T04:09:09.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:09 smithi029 podman[86165]: 2024-04-27 04:09:09.718128854 +0000 UTC m=+0.540937589 container remove 8e37ab9d43fd146b4de1ba7d50e845ce55a9095be2db00019778bc50ef799984 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-4c587076-0439-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.29.1, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.0, ceph=True, org.label-schema.build-date=20231212, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-27T04:09:09.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:09 smithi029 systemd[1]: ceph-4c587076-0439-11ef-bc93-c7b262605968@osd.2.service: Deactivated successfully. 2024-04-27T04:09:09.969 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:09 smithi029 systemd[1]: Stopped Ceph osd.2 for 4c587076-0439-11ef-bc93-c7b262605968. 2024-04-27T04:09:09.970 INFO:journalctl@ceph.osd.2.smithi029.stdout:Apr 27 04:09:09 smithi029 systemd[1]: ceph-4c587076-0439-11ef-bc93-c7b262605968@osd.2.service: Consumed 1min 9.938s CPU time. 2024-04-27T04:09:14.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:09:14.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:09:17.890 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:09:26.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:09:26.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:09:29.950 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:09:41.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:09:41.952 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:09:45.022 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:09:48.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:09:48.023 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:09:48.098 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:09:48.098 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:09:48.099 INFO:tasks.cephadm:Teardown complete 2024-04-27T04:09:48.099 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/cephadm.py", line 1700, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/cephadm.py", line 2240, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/cephadm.py", line 2243, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/cephadm.py", line 1833, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/cephadm.py", line 226, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/cephadm.py", line 395, in _rm_cluster remote.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi052 2024-04-27T04:09:48.122 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-27T04:09:48.135 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-04-27T04:09:48.151 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-27T04:09:48.151 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-27T04:09:48.166 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:09:48.166 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:09:51.230 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:10:00.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:10:00.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:10:03.294 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:10:15.295 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:10:15.296 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:10:18.366 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:10:21.368 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:10:21.369 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:10:21.438 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:10:21.438 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:10:21.490 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi052 2024-04-27T04:10:21.511 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-27T04:10:21.522 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-27T04:10:21.523 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T04:10:21.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:10:21.532 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:10:21.546 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-04-27T04:10:21.551 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T04:10:21.551 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-04-27T04:10:21.552 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv01.front.sepia.ceph.com 3 6 377 64 +1298us[+1296us] +/- 59ms 2024-04-27T04:10:21.552 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 61 -1740us[-1740us] +/- 60ms 2024-04-27T04:10:21.552 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 61 +544us[ +544us] +/- 72ms 2024-04-27T04:10:21.552 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T04:10:24.606 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:10:33.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:10:33.608 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:10:36.670 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:10:48.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:10:48.672 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:10:51.742 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:10:54.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:10:54.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:10:54.814 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:10:54.814 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:10:54.815 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi052 2024-04-27T04:10:54.830 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-27T04:10:54.842 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-27T04:10:54.843 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-27T04:10:54.859 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-27T04:10:54.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:10:54.944 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:10:58.014 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:11:07.015 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:11:07.016 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:11:10.082 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:11:22.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:11:22.084 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:11:25.150 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:11:28.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:11:28.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:11:28.226 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:11:28.226 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:11:28.227 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi052 2024-04-27T04:11:28.236 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-27T04:11:28.248 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-27T04:11:28.259 INFO:teuthology.task.internal:Duration was 8502.706503 seconds 2024-04-27T04:11:28.259 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-27T04:11:28.270 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-27T04:11:28.271 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-27T04:11:28.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:11:28.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:11:28.320 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T04:11:31.362 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:11:40.363 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:11:40.364 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:11:43.422 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:11:55.423 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:11:55.424 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:11:58.494 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:12:01.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:12:01.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:12:01.566 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:12:01.567 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:12:01.567 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi052 2024-04-27T04:12:01.585 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-27T04:12:01.597 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-27T04:12:01.597 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-27T04:12:01.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:12:01.632 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:12:04.706 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:12:13.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:12:13.708 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:12:16.766 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:12:28.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:12:28.769 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:12:31.838 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:12:34.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:12:34.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:12:34.910 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:12:34.911 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:12:34.911 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi052 2024-04-27T04:12:34.930 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-27T04:12:34.943 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-27T04:12:34.950 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:12:34.951 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:12:34.976 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2024-04-27T04:12:38.014 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:12:47.015 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:12:47.016 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:12:50.078 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:13:02.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:13:02.080 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:13:05.150 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:13:08.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:13:08.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:13:08.226 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:13:08.226 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:13:08.227 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi052 2024-04-27T04:13:08.242 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-27T04:13:08.257 INFO:teuthology.task.internal:Transferring archived files... 2024-04-27T04:13:08.258 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675525/remote/smithi029 2024-04-27T04:13:08.259 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-27T04:13:08.299 DEBUG:teuthology.misc:Transferring archived files from smithi052:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675525/remote/smithi052 2024-04-27T04:13:08.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:13:08.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:13:11.358 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:13:20.359 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:13:20.360 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:13:23.426 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:13:35.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:13:35.428 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:13:38.494 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:13:41.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:13:41.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:13:41.566 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:13:41.566 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:13:41.567 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi052 2024-04-27T04:13:41.584 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-27T04:13:41.597 INFO:teuthology.task.internal:Not uploading archives. 2024-04-27T04:13:41.598 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-27T04:13:41.610 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-27T04:13:41.610 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-27T04:13:41.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:13:41.618 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:13:41.633 INFO:teuthology.orchestra.run.smithi029.stdout: 262213 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 27 04:09 /home/ubuntu/cephtest 2024-04-27T04:13:41.634 INFO:teuthology.orchestra.run.smithi029.stdout: 263988 212 -rwxr-xr-x 1 ubuntu ubuntu 215333 Apr 27 01:56 /home/ubuntu/cephtest/cephadm 2024-04-27T04:13:41.634 INFO:teuthology.orchestra.run.smithi029.stdout: 262214 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Apr 27 04:12 /home/ubuntu/cephtest/archive 2024-04-27T04:13:41.634 INFO:teuthology.orchestra.run.smithi029.stdout: 394511 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 27 04:10 /home/ubuntu/cephtest/archive/audit 2024-04-27T04:13:41.634 INFO:teuthology.orchestra.run.smithi029.stdout: 424481 176 -rw------- 1 ubuntu ubuntu 179389 Apr 27 04:10 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-04-27T04:13:41.634 INFO:teuthology.orchestra.run.smithi029.stdout: 262216 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 27 01:49 /home/ubuntu/cephtest/archive/syslog 2024-04-27T04:13:41.634 INFO:teuthology.orchestra.run.smithi029.stdout: 262218 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Apr 27 01:49 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T04:13:41.635 INFO:teuthology.orchestra.run.smithi029.stdout: 262217 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Apr 27 01:49 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T04:13:41.635 INFO:teuthology.orchestra.run.smithi029.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-27T04:13:44.670 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:13:53.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:13:53.672 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:13:56.734 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:14:08.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:14:08.736 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:14:11.806 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:14:14.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-27T04:14:14.808 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:14:14.882 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-27T04:14:14.882 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-27T04:14:14.883 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi052 2024-04-27T04:14:14.900 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-27T04:14:14.915 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-27T04:14:14.931 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-04-27T04:14:14.931 INFO:teuthology.nuke:Checking targets against current locks 2024-04-27T04:14:14.957 DEBUG:teuthology.nuke:shortname: smithi029 2024-04-27T04:14:14.958 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-27T04:14:14.981 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675525', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-27 01:43:27.748629', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC6Y1/iGgURJdjmMr99e4FyQN5Mawd9wp/wUJoS+Iq40L2ITjqAhz7XMQG8w6sgwgq/Zf7TfRoOcysBUwzMZryM='} 2024-04-27T04:14:14.985 DEBUG:teuthology.nuke:shortname: smithi052 2024-04-27T04:14:14.986 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-27T04:14:15.008 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675525', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-27 01:43:27.750281', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBG0HqsVnfhUZ9c4lxgbA0XF2R+ybv4CnxHRrldGfDt63LZaA33o6tWhAYl4Y8vLdCGG5gl7Y/pssZIwlNzx7TM='} 2024-04-27T04:14:15.053 INFO:teuthology.orchestra.console.smithi052:Power off 2024-04-27T04:14:15.054 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-27T04:14:15.078 DEBUG:teuthology.orchestra.console.smithi052:power off output: Chassis Power Control: Down/Off 2024-04-27T04:14:15.078 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T04:14:15.096 INFO:teuthology.orchestra.console.smithi029:Power off 2024-04-27T04:14:15.096 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-27T04:14:15.106 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is on 2024-04-27T04:14:15.119 DEBUG:teuthology.orchestra.console.smithi029:power off output: Chassis Power Control: Down/Off 2024-04-27T04:14:15.119 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T04:14:15.143 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-04-27T04:14:19.107 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T04:14:19.145 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T04:14:19.218 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is on 2024-04-27T04:14:19.256 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-04-27T04:14:23.222 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T04:14:23.257 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T04:14:23.336 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is off 2024-04-27T04:14:23.368 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off 2024-04-27T04:14:23.437 INFO:teuthology.orchestra.console.smithi052:Power off completed 2024-04-27T04:14:23.469 INFO:teuthology.orchestra.console.smithi029:Power off completed 2024-04-27T04:14:23.575 INFO:teuthology.run:Summary data: description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 8502.706502914429 failure_reason: 'Command failed on smithi029 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c587076-0439-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c ''ceph orch ps''"' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=3fadf09fe1454846857965df30295752 status: fail success: false 2024-04-27T04:14:23.575 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-27T04:14:23.665 INFO:teuthology.run:FAIL